Lead Data Engineer

Primary responsibilities:

• Design and develop end-to-end ETL pipelines, working from data source integration to visualization.
• Architect scalable data workflows and manage complex distributed systems to ensure efficient data processing.
• Collaborate cross-functionally in an Agile environment, adhering to Scrum principles to ensure iterative and incremental delivery.
• Mentor and lead a team of engineers, providing technical guidance and fostering a culture of continuous improvement.
• Partner with stakeholders across Product, Design, QA, and Operations teams to gather requirements, align priorities, and deliver high-quality solutions.
• Implement best practices for data security, governance, and performance optimization, including the use of technologies like Kerberos and Iceberg.
• Continuously explore and adopt new tools and technologies to enhance the pipeline’s efficiency and scalability.

Required Qualifications:

• Strong expertise in ETL development, including proficiency in Spark, Kafka, StarRocks, and Apache Polaris.
• Hands-on experience with Data Warehousing concepts and tools.
• Proficient in visualization tools like Power BI, with a deep understanding of creating actionable insights from data.
• Familiarity with data storage and security frameworks, including MinIO, Kerberos, and Iceberg.
• Experience working in an Agile model and a thorough understanding of Scrum principles and practices.
• 8+ years of experience in software development, with at least 3 years in a lead or architect role involving ETL and data pipeline development.
• Excellent communication and collaboration skills to interact with both technical and non-technical stakeholders.
• Strong analytical and problem-solving skills, with the ability to address complex challenges efficiently.