DB2 Data Migration & Databricks ETL Engineer (AI Enabled)

Request ID

115059-1

Title

DB2 Data Migration & Databricks ETL Engineer (AI Enabled)

Location

Remote

Pay Rate

$55.00-58.00/hr on W2/C2C (All Inclusive)

Experience Requested - 07+ Years of Experience Mandatory

Duration

06+ Months with Possible Extension.

We are seeking an experienced DB2 Data Migration / ETL Engineer to support enterprise-scale data modernization initiatives involving migration of IBM DB2 workloads to the Databricks Lakehouse platform. The ideal candidate will possess deep expertise in DB2 databases, ETL/ELT development, Databricks, Spark, and cloud-based data engineering practices. Experience leveraging AI/GenAI tools for accelerating migration assessment, code conversion, validation, and documentation is highly desirable.

Key Responsibilities

• Analyze and assess IBM DB2 databases, schemas, metadata, and dependencies for migration to Databricks.
• Extract complete inventories of DB2 database objects including tables, views, aliases, stored procedures, functions, triggers, indexes, and sequences.
• Design, develop, and optimize scalable ETL/ELT pipelines using Databricks, PySpark, SQL, and cloud-native technologies.
• Modernize DB2 SQL queries, stored procedures, and legacy ETL workflows into performant Databricks solutions.
• Perform schema mapping, metadata analysis, lineage discovery, and dependency validation.
• Execute data migration, reconciliation, validation, and data-quality testing across source and target environments.
• Implement security, governance, and access controls using Databricks Unity Catalog aligned with DB2 security models.
• Optimize Spark jobs, data ingestion frameworks, and large-scale data processing workloads.
• Collaborate with business stakeholders, architects, cloud engineers, and enterprise data teams to deliver migration programs.
• Develop automation utilities for migration assessment, code conversion, testing, and validation.

Required Skills

Database Technologies

• IBM DB2 LUW / Mainframe DB2
• DB2 Catalog Queries
• DB2 SQL & Stored Procedures
• TSO/ISPF
• DBeaver
• IBM Data Studio

Data Engineering

• Databricks Lakehouse Platform
• Apache Spark
• PySpark
• SQL
• ETL/ELT Development
• Data Migration & Data Modernization
• Data Validation & Reconciliation
• Data Quality Frameworks

Cloud & DevOps

• CI/CD Pipelines
• Git/GitHub/GitLab
• Azure Databricks (Preferred)
• Workflow Automation

Governance & Security

• Databricks Unity Catalog
• Data Access Controls
• Sensitive Data Management
• Data Governance

Preferred Industry Experience

• Property & Casualty (P&C) Insurance
• Financial Services
• Banking
• Enterprise Data Transformation Programs

AI / GenAI Skills (Preferred)

• Experience using Generative AI tools such as GitHub Copilot, Databricks Assistant, OpenAI, Azure OpenAI, or similar AI coding assistants.
• Knowledge of prompt engineering for SQL conversion, code modernization, documentation generation, and dependency analysis.
• Experience using LLMs for migration artifact creation and metadata exploration.
• Knowledge of RAG architectures and GenAI-enabled knowledge repositories.
• Development of Python/PySpark automation tools for schema discovery, mapping, code conversion, and migration validation.
• AI-assisted data quality, anomaly detection, and reconciliation frameworks.

Qualifications

• Bachelor's degree in Computer Science, Information Technology, Engineering, or related discipline.
• 8+ years of experience in Data Engineering, ETL Development, or Database Migration projects.
• 3+ years of hands-on Databricks and Spark experience.
• Strong experience in IBM DB2 environments and large-scale data migration initiatives.

Nice-to-Have

• Databricks Certified Data Engineer Associate/Professional
• Azure Data Engineer Associate (DP-203)
• Experience with Lakebridge Analyzer
• Data Lineage & Metadata Management Tools
• AI/ML experience within Databricks ecosystem