Senior Engineer (India Office)

We are seeking a Senior Data Engineer with AI/GenAI expertise to join the Data Platform (Foundry) team. This role will focus on building and optimizing enterprise-scale data pipelines while also enabling metadata-driven, AI-driven use cases, including copilots, LLM integrations, and intelligent data workflows using Databricks, Azure, ADF, Azure SQL, Astronomer/Airflow or Control M

Detailed Description

• Design, build, and maintain data pipelines using Databricks, Spark, ADF, Advanced SQL/Python, and Azure services
• Develop and optimize bronze → silver → gold transformations
• Schedule and orchestrate via ADF, Airflow, Control M. Airflow experience would be a huge plus
• Enable AI/GenAI use cases across the platform, including:
• Copilot integrations
• LLM-based workflows
• Intelligent data enrichment pipelines
• Work with Databricks GenAI capabilities .
• Integrate external AI platforms (e.g., Anthropic, Azure OpenAI) into enterprise data workflows
• Ensure data reliability, performance, and operational stability
• Troubleshoot and resolve production issues with strong SLAs
• Support ingestion using tools such as:
• Fivetran / ADF / Auto Loader
• Implement data quality, monitoring, and reconciliation frameworks
• Ability to read/write code in SQL, Python, other
• Collaborate across:
• Data Engineering teams
• Data Platform engineering
• Business stakeholders
• Contribute to platform modernization and AI adoption strategy
• Follow governance best practices:
• Code quality
• Deployment controls
• Audit and compliance
• Required Skills & Experience
• 8–10+ years in Data Engineering
• Strong experience with:
• Databricks (Delta Lake, Spark)
• Azure Data Platform - ADF, Synapse
• SQL - Azure SQL, SQL Server, Teradata, DB2, Oracle
• Hands-on experience with AI/GenAI technologies, including:
• Building or integrating copilot-style experiences
• Working with LLMs (e.g., Anthropic Claude, OpenAI models)
• Strong programming skills:
• Python, SQL
• Orchestration using — Airflow or Astronomer, Control M. Preferably Airflow
• Experience building scalable ETL/ELT pipelines
• Familiarity with production support and incident management

#LI-KS1