Data Engineer

Title

Data Engineer

Location

Chennai

Exp

6+ Yrs

Job Description

Key Responsibilities

• Design, develop, and maintain scalable data pipelines and data processing workflows.
• Build and manage ETL/ELT processes for extracting, transforming, and loading data from multiple sources.
• Develop and optimize complex SQL queries for data extraction, transformation, and analysis.
• Work with large datasets and implement efficient data processing solutions using Python and relevant data engineering frameworks.
• Design and maintain data warehouses, data lakes, and data models to support analytics and reporting requirements.
• Work with Apache Spark and other distributed data processing technologies for handling large-scale datasets.
• Integrate data from multiple sources, ensuring data accuracy, consistency, quality, and reliability.
• Implement data validation, monitoring, and error-handling mechanisms across data pipelines.
• Optimize data workflows and query performance to improve processing efficiency and scalability.
• Collaborate with Data Analysts, Data Scientists, BI Developers, and business stakeholders to understand data requirements.
• Ensure data security, governance, and compliance with organizational standards.
• Troubleshoot data pipeline failures and resolve performance and data quality issues.
• Maintain technical documentation related to data architecture, pipelines, workflows, and integration processes.

Required Skills & Qualifications

• 6+ years of hands-on experience in Data Engineering or related roles.
• Strong programming skills in Python and SQL.
• Hands-on experience in developing and maintaining ETL/ELT pipelines.
• Good understanding of data warehousing concepts, data lakes, and data modeling.
• Experience working with Apache Spark or other distributed data processing frameworks.
• Strong knowledge of relational databases and data integration techniques.
• Experience in data pipeline optimization, performance tuning, and data quality management.
• Understanding of cloud-based data engineering solutions and architecture.
• Good problem-solving, analytical, and communication skills.
• Ability to work independently and collaborate with cross-functional teams.

Preferred Skills

• Experience with cloud platforms such as AWS, Azure, or Google Cloud Platform (GCP).
• Familiarity with tools such as Databricks, Snowflake, Azure Data Factory, or AWS Glue.
• Knowledge of Apache Airflow or other workflow orchestration tools.
• Experience with Kafka or other data streaming technologies.
• Understanding of CI/CD, Git, and DevOps practices for data engineering.
• Knowledge of data governance, security, and compliance practices.