Senior Data Engineer

Title

Senior Data Engineer

Location

Pune

Exp

4+ yrs

Job Description

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and data processing workflows.
• Build and optimize ETL/ELT pipelines to extract, transform, and load data from multiple sources.
• Develop complex SQL queries and Python scripts for data extraction, transformation, and processing.
• Work with large datasets using Apache Spark and PySpark for distributed data processing.
• Design and maintain data warehouses, data lakes, and data models to support analytics requirements.
• Integrate data from structured and unstructured sources while ensuring data accuracy, consistency, and reliability.
• Implement data validation, quality checks, monitoring, and error-handling mechanisms.
• Optimize data pipelines and query performance to improve processing efficiency and scalability.
• Work with cloud-based data engineering platforms and services.
• Collaborate with Data Scientists, Data Analysts, BI Developers, and other stakeholders to understand data requirements.
• Implement workflow orchestration and scheduling using tools such as Apache Airflow.
• Participate in code reviews, testing, debugging, and production support activities.
• Maintain technical documentation related to data architecture, pipelines, and data integration processes.
• Follow best practices for data security, governance, version control, and CI/CD.

Required Skills & Qualifications

• 4+ years of hands-on experience in Data Engineering or related roles.
• Strong programming experience in Python and SQL.
• Hands-on experience in developing and maintaining ETL/ELT data pipelines.
• Experience with Apache Spark and PySpark for large-scale data processing.
• Strong understanding of data warehousing, data lakes, and data modeling concepts.
• Experience working with relational and non-relational databases.
• Knowledge of data integration, transformation, and pipeline optimization techniques.
• Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform (GCP).
• Understanding of data quality, data validation, and performance optimization.
• Experience with Git, CI/CD pipelines, and Agile development practices.
• Strong analytical, problem-solving, and communication skills.