Senior Data QA Engineer – ETL, Databricks & AI/ML

Senior Data QA Engineer – ETL, Databricks & AI/ML

Location

Louisville, KY / Dallas, TX

Experience

8–10 Years

Role Overview

Seeking a Senior Data QA / Automation Engineer with strong experience in ETL/Data Testing, Databricks, Spark/PySpark, Python, SQL, and AI/ML validation. The ideal candidate will be responsible for validating enterprise data pipelines, Databricks platforms, ML workflows, integrations, and automated quality processes.

Key Responsibilities

• Perform end-to-end ETL/ELT, data quality, and Source-to-Target validation.
• Validate data transformations, aggregations, business rules, lineage, schemas, metadata, and Delta Lake.
• Test CDC, incremental loads, batch and streaming pipelines.
• Validate Databricks Notebooks, Workflows, Jobs, Delta Live Tables, Unity Catalog, and Bronze/Silver/Gold data layers.
• Perform Spark/PySpark transformation and data quality testing.
• Validate AI/ML models, predictions, feature engineering, model inputs/outputs, and MLOps workflows.
• Test data drift, schema drift, model deployment, MLflow/Databricks ML/Azure ML integrations, and GenAI/RAG workflows where applicable.
• Perform API, integration, contract, and end-to-end data flow testing.
• Develop automated testing frameworks and reusable utilities using Python and SQL.
• Integrate automated tests and quality gates into CI/CD pipelines.
• Generate test reports, metrics, and quality validation results.

Required Skills

• 8–10 years of QA/Automation Testing experience.
• 6+ years of ETL/Data Testing experience.
• 3+ years of Databricks/Spark/PySpark testing.
• Strong Python and SQL programming skills.
• Hands-on AI/ML Model Testing or MLOps validation experience.
• Strong experience with data warehouses, data lakes, APIs, and cloud data platforms.
• Experience building and maintaining test automation frameworks.
• Strong understanding of data quality, reconciliation, validation, and integration testing.

Preferred

• Experience with MLflow, Azure ML, Delta Lake, Unity Catalog, CI/CD, and GenAI/RAG testing.
• Experience testing large-scale structured and semi-structured datasets.
• Strong understanding of enterprise data architecture and data pipelines.