Data Analyst / Data Engineer

Job Description

Data Engineer / Data Analyst

Experience

5–8 Years

Employment Type

Full-Time

Primary Skills

Azure Data Lake, Azure Synapse Analytics, PySpark

Job Overview

We are looking for an experienced Azure Data Engineer / Data Analyst with strong hands-on expertise in Microsoft Azure data services, PySpark, SQL and data pipeline development.

The ideal candidate will be responsible for designing, developing, deploying and maintaining scalable, reliable and high-performance data pipelines and ETL/ELT processes. The candidate should have experience working with Azure Data Lake Storage Gen2, Azure Synapse Analytics, Azure Data Factory and Azure Databricks.

The role requires strong knowledge of data engineering principles, data transformation, data quality, performance optimization and CI/CD practices.

Key Responsibilities

• Design, develop and maintain scalable ETL/ELT data pipelines using Azure Data Factory, Azure Synapse Analytics and Azure Databricks.
• Develop and optimize PySpark applications for large-scale data processing and transformation.
• Implement and manage data storage solutions using Azure Data Lake Storage Gen2 and Azure Blob Storage.
• Build reusable, modular and parameterized data pipelines to support multiple data sources.
• Perform data ingestion, cleansing, transformation, validation and reconciliation.
• Work with SQL Server, PostgreSQL and other relational databases to develop optimized SQL queries.
• Integrate data from REST APIs, databases and file-based sources such as CSV, JSON, XML and Parquet.
• Implement incremental data loading, CDC and data quality frameworks.
• Optimize data pipelines for performance, scalability, reliability and cost efficiency.
• Monitor, troubleshoot and resolve production pipeline failures.
• Collaborate with data architects, business analysts and stakeholders to understand data requirements.
• Manage source code, version control and CI/CD deployments using Azure DevOps and Git.
• Work with Power BI, PowerApps and Power Automate for data analytics and business process automation.
• Follow software engineering best practices, including coding standards, testing and documentation.

Mandatory Technical Skills

• Strong hands-on experience with Azure Data Lake Storage Gen2.
• Strong knowledge of Azure Synapse Analytics.
• Strong programming experience in PySpark.
• Experience in Azure Data Factory and Azure Databricks.
• Good understanding of SQL and relational databases.
• Experience in designing and developing ETL/ELT pipelines.
• Understanding of data warehousing, data modeling and data transformation concepts.

Required Qualifications

• Bachelor's degree in Computer Science, Engineering, Information Technology or a related field, or equivalent experience.
• 5–8 years of relevant experience in Data Engineering / Data Analytics.
• Strong experience in Microsoft Azure data services.
• Good understanding of software engineering principles and data pipeline architecture.
• Strong analytical, problem-solving and troubleshooting skills.
• Excellent communication and collaboration skills.

Preferred Qualifications

• Experience with Microsoft Fabric.
• Hands-on experience with Power BI, PowerApps and Power Automate.
• Knowledge of REST APIs and OAuth authentication.
• Experience with Azure DevOps pipelines and repositories.
• Understanding of Git workflows and CI/CD practices.
• Knowledge of Ansible and infrastructure automation.
• Programming experience in Python, C#, Java or JavaScript.
• Experience with Delta Lake, Medallion Architecture and incremental data processing.

Key Skills for Screening

Azure Data Lake | Azure Synapse | PySpark | Azure Data Factory | Azure Databricks | SQL | ETL | Azure DevOps | Microsoft Fabric

Ideal Candidate

An Azure Data Engineer with 5–8 years of experience and strong hands-on expertise in Data Lake, Synapse and PySpark, capable of independently developing and maintaining scalable Azure-based data pipelines.