Azure Data Engineer

Job Description:

Role and Responsibilities

Technical Responsibilities:

• Data Pipelines: Operationalize and maintain data pipelines for investment entities, ensuring they meet business demands.
• ETL/ELT Pipelines: Deploy and optimize ETL/ELT pipelines for reliability, scalability, and security.
• Data Handling: Source, transform, and deliver structured and unstructured data for analytical use.
• Azure Resources: Provision and manage Azure resources, including security and cost management.
• Monitoring and Testing: Monitor pipelines, improve reliability, automate audits, and perform various tests.
• Technical Solutions: Translate requirements into solutions and participate in solution build, delivery, support, and troubleshooting.

Organizational Responsibilities:

• Agile Environment: Operate within an Agile development framework.
• Communication: Effectively communicate with engineers, product managers, analysts, and senior management.
• Leadership: Lead change initiatives and communicate impacts.
• Process Improvement: Identify and implement opportunities for process improvements.
• Data Management: Establish sound data management practices and help shape the development and production infrastructure.
• Project Management: Work on multiple projects simultaneously, ensuring timely delivery.

Qualifications:

• Experience: 3+ years with the Microsoft Azure data stack or equivalent, preferably in the financial industry.
• Education: Bachelor’s degree in a scientific or engineering field or equivalent experience.
• Certifications: Microsoft Data Engineer Associate certification is a plus.
• Industry Knowledge: Experience with best practices, operational procedures, and documentation.
• Relationship Building: Proven ability to build technical relationships and align expectations.
• Problem-Solving: Ability to trace data workflows and resolve issues independently.
• Teamwork: Collaborative mindset focused on team success.

Preferred Skills:

• Technical Experience: Experience with cloud and hybrid infrastructures, architecture designs, and technology management.
• Programming: Proficiency in Python (Pandas, Numpy) and SQL.
• Data Tools: Expertise in Databricks, Snowflake, or Azure Data Factory/Synapse.
• CI/CD: Experience with automating CI/CD pipelines using Azure DevOps.
• Containerization: Experience with container deployment to Kubernetes.
• Data Modeling: Experience with star schema reporting pipelines using Kimball or Data Vault v2.0 methodologies.
• Firewall and Networking: Understanding of firewall and networking concepts related to data platforms.