Data Engineer

Key Responsibilities

• Design, develop, and support batch and incremental data ingestion pipelines into the data lake.
• Develop and optimize complex SQL queries, data transformations, and ETL workflows.
• Build, maintain, and monitor Airflow DAGs for orchestration and workflow automation.
• Analyze and troubleshoot data quality, data reconciliation, and pipeline execution issues.
• Perform end-to-end data lineage analysis and trace data from source systems to downstream consumers.
• Debug data discrepancies by investigating source systems, transformation logic, and target datasets.
• Support data migration and modernization initiatives from legacy platforms to cloud-based data platforms.
• Work closely with business stakeholders, architects, and upstream/downstream teams to understand data requirements.
• Validate data completeness, accuracy, and consistency across multiple systems.
• Participate in production support activities and resolve data-related incidents within SLA timelines.
• Develop reusable data engineering components and recommend process improvements.

Required Skills

Strong Expertise

• SQL / PL/SQL
• Complex joins, CTEs, analytical functions
• Query optimization and performance tuning
• Data modeling and database concepts
• BigQuery
• Query development and optimization
• Partitioning and clustering concepts
• Data loading and transformation techniques
• ETL/Data Engineering
• Data ingestion patterns (Batch and Incremental)
• Data transformations and validations
• Data quality checks and reconciliation

Good to Have

• Apache Airflow
• Developing and troubleshooting DAGs
• Scheduling, monitoring, and dependency management
• Python
• Basic to intermediate scripting
• Data processing and automation
• Cloud Platforms (Preferably GCP)
• Cloud Storage
• BigQuery
• Composer/Airflow

Preferred Candidate Profile

• Strong analytical and problem-solving skills.
• Ability to independently investigate and resolve data issues.
• Good communication and stakeholder management skills.
• Experience in cloud data lake or warehouse environments.
• Exposure to customer, transactional, or enterprise-scale datasets is preferred.

Screening Focus Areas

• Advanced SQL/PLSQL
• ETL & Data Pipeline Design
• BigQuery Concepts
• Data Debugging & Root Cause Analysis
• Airflow DAG Development
• Python Basics
• Data Migration & Data Validation
• Source-to-Target Traceability and Data Lineage

Must Have

SQL/PLSQL, BigQuery/Cloud Data Warehouse, ETL, Data Analysis & Debugging

Good to Have

Airflow DAG Development, Python, GCP Services

Kafka, Pub/Sub optional