Senior Data Engineer

Arbisoft


Date: 1 week ago
City: Lahore
Contract type: Full time

Arbisoft is hiring a Senior Data Engineer! We are looking for individuals who can help us evolve, automate and scale our data pipelines and infrastructure. An ideal candidate should be quick to learn and adapt, care about analytical rigor and data quality, love writing code, and thrive in a collaborative environment.


Key Responsibilities:

  • Real-Time and Batch Data Consolidation: Design, build, and optimize data pipelines to support real-time and batch data processing across our systems.
  • Data Quality and Governance: Establish and enforce metadata standards, quality controls, and data governance practices to ensure high data reliability and discoverability.
  • Scalable, Cost-Effective Storage Solutions: Develop and maintain high-scale storage solutions with efficient columnar compression, supporting cost-effective and high-performance data access.
  • High-Throughput Computation Engine: Architect a robust, offline computation engine optimized for high-throughput and cost-efficient data processing.
  • Workflow Automation and Scaling: To enhance pipeline efficiency, and implement auto-scaling solutions, job prioritization, and support for complex workflows.
  • Collaborative Problem-Solving: Work closely with cross-functional teams to align data infrastructure solutions with business requirements and proactively address any issues, roadblocks, or optimization opportunities.


Knowledge & Skills :

  • Programming Languages: Python
  • Big Data & Stream Processing: Spark, PySpark, Kafka
  • Data Storage and ETL: S3, AWS Glue, Delta Lake, AWS Lake Formation, Airflow, Step Functions
  • Data Formats: Parquet, Avro, Iceberg
  • Advanced-Data Solutions: RDF, Knowledge Graphs, Embedding Stores, Vector Databases


Good To have:

  • 4+ years of overall experience.
  • Proven experience in data engineering with a focus on scalable, high-performance data pipelines
  • Strong understanding of data governance, quality control, and metadata management
  • Proficiency with AWS tools (S3, Glue, Lake Formation), Spark, and Kafka for data processing and storage
  • Skilled in handling various data formats (Parquet, Avro) and advanced data management solutions (Knowledge Graphs, Vector Databases)
  • Excellent problem-solving skills and a collaborative approach to teamwork.

How to apply

To apply for this job you need to authorize on our website. If you don't have an account yet, please register.

Post a resume

Similar jobs

Full-Stack Senior Software Engineer

FORMULATRIX, Lahore
3 days ago
FORMULATRIX supplies innovative software and robotic automation solutions to the life science industry including the leading pharmaceutical companies and academic research institutions around the world. We collaborate with researchers to simplify scientific workflows, such as sample preparation and analysis, by designing solutions without boundaries and bringing novel cutting-edge technology into the laboratory. We are committed to researchers, their labs, and...

Innovation & Commercialization Manager

PepsiCo Pakistan, Lahore
3 days ago
Job DescriptionApply nowJoin Our Talent CommunityJOB_DESCRIPTION.SHARE.HTMLCAROUSEL_PARAGRAPHJOB_DESCRIPTION.SHARE.HTMLOverviewThe primary purpose of this role is to enable accelerated, profitable, sustainable growth by driving the rapid prioritization, development & flawless execution, at scale, of a world class pipeline of human-centric innovation that is positive for the portfolio, people & planet. The role has a sizeable influence on the Business Unit innovation agenda, is at...

Access Control Specialist

Programmers Force, Lahore
4 days ago
Key Responsibilities{{:}} Application Access Control Management{{:}} Administer, monitor, and maintain different cloud application access control systems User Management{{:}} Create, modify, and deactivate access credentials for employees ensuring accurate records of all individuals granted access Security Compliance{{:}} Ensure compliance with security policies, procedures, and industry regulations regarding access control Internal and external Audits{{:}} Ensure internal and external audits and coordination with...