Data Engineer with Azure

Job Title

Azure Data Engineer (Azure to AWS Migration) – Spark, Kafka & Databricks

Location

Dallas TX

Job Type

Contract

Work Arrangement

5 Days Onsite

Domain

Telecom

Interview

2-3 rounds, Final round in person required

Note

US Citizenship Required for this role

Job Summary

We are seeking an experienced Azure Data Engineer with strong expertise in Apache Spark, Apache Kafka, Azure Databricks , and cloud migration projects , specifically migrating data platforms from Microsoft Azure to AWS . The ideal candidate will have extensive experience building scalable data pipelines, streaming solutions, and modern data lake architectures while driving cloud transformation initiatives.

This role requires hands-on experience with Azure services, Databricks, Spark optimization, Kafka event streaming, and AWS data services to support enterprise-scale migration efforts.

Key Responsibilities

• Design, develop, and maintain scalable batch and real-time data pipelines.
• Lead migration of enterprise data platforms from Microsoft Azure to Amazon Web Services (AWS) .
• Develop and optimize ETL/ELT pipelines using Azure Databricks and Apache Spark .
• Build and maintain real-time streaming applications using Apache Kafka .
• Migrate data lakes, data warehouses, and analytics workloads from Azure to AWS.
• Optimize Spark jobs for performance, scalability, and cost efficiency.
• Work closely with solution architects to design cloud-native AWS data architectures.
• Implement CI/CD pipelines for data engineering deployments.
• Ensure data quality, governance, monitoring, and security throughout migration activities.
• Collaborate with business stakeholders, data scientists, and application teams to deliver reliable data solutions.
• Troubleshoot production issues and optimize distributed data processing environments.

Required Skills

Cloud Platforms

• Microsoft Azure
• Amazon Web Services (AWS)
• Azure to AWS Migration experience (Mandatory)

Mandatory Qualifications

• 8+ years of Data Engineering experience.
• 4+ years of hands-on experience with Azure Databricks .
• Strong expertise in Apache Spark and PySpark .
• Hands-on experience with Apache Kafka and streaming architectures.
• Proven experience in Azure to AWS cloud migration projects .
• Experience designing enterprise-scale data lake solutions.
• Strong SQL and Python programming skills.
• Experience with Delta Lake and modern Lakehouse architecture.
• Knowledge of CI/CD and Infrastructure as Code.
• Excellent communication and stakeholder management skills.

Preferred Qualifications

• AWS Certified Data Engineer – Associate or AWS Certified Solutions Architect.
• Microsoft Azure Data Engineer Associate (DP-203).
• Databricks Certified Data Engineer Associate/Professional.
• Experience with Snowflake, Iceberg, or Apache Hudi.
• Experience with Airflow or other orchestration tools.