Spark Engineer

We are seeking a skilled Spark/PySpark Engineer with a minimum of 4 years of experience in developing and managing big data solutions. The ideal candidate will have expertise in leveraging Apache Spark for large-scale data processing, along with proficiency in Python (PySpark) for building efficient and scalable data pipelines.

Responsibilities:

• Design, develop, and maintain scalable and efficient data pipelines using Apache Spark and PySpark.
• Collaborate with data architects and business teams to gather requirements and define data processing workflows.
• Implement and optimize distributed data processing systems for high performance and scalability.
• Write reusable, testable, and efficient code in PySpark for data ingestion, transformation, and processing.
• Work with structured and unstructured data from various sources such as HDFS, S3, and relational databases.
• Ensure data quality, integrity, and consistency across pipelines and workflows.
• Monitor, troubleshoot, and debug Spark jobs and data pipelines in production environments.
• Stay updated with the latest trends in big data technologies and apply best practices in development.

Requirements:

• Minimum 4 years of professional experience in big data engineering, with at least 2 years of hands-on experience in PySpark and Apache Spark.
• Proficiency in Python programming, with a strong understanding of functional programming and data manipulation libraries.
• Solid knowledge of distributed computing concepts and Spark internals, including RDDs, DataFrames, and Spark SQL.
• Experience in working with big data tools and frameworks such as Hadoop, Hive, Kafka, or Airflow.
• Strong understanding of relational and non-relational databases, including SQL and NoSQL.
• Familiarity with cloud platforms such as AWS, Azure, or GCP, including services like EMR, Databricks, or BigQuery.
• Knowledge of data modeling, schema design, and ETL processes.