Azure Data Engineer (ADF & Databricks)

Job Title

Azure Data Engineer (ADF & Databricks)

Work Location & Reporting Address

Houston, TX 77079

Vendor Rate

$50/hr

Contract duration 6 Months

Interview Process (Is face to face required?) Yes

Must Have Skills

Azure Data Factory Pipelines, data flows (mapping wrangling), triggers, integration runtimes, parameterization, error handling & retries.

Azure Databricks Databricks Notebooks, Jobs Workflows, PySpark Scala, Delta Lake, cluster configuration and Eligible to workimization.

SQL Expert level SQL (CTEs, window functions, query tuning), working with large datasets.

Nice to have skills

Agile

Infrastructure as Code exposure is a plus.

Detailed Job Description

Experience 6-8 years in data engineering with at least 3 years on Azure. Azure Data Factory Pipelines, data flows mapping wrangling, triggers, integration runtimes, parameterization, error handling retries. Azure Databricks Databricks Notebooks, Jobs Workflows, PySpark Scala, Delta Lake, cluster configuration and Eligible to workimization. Programming Advanced Python or Scala at least one, strong coding standards, modularization, and unit testing e.g., pytest. SQL Expert level SQL CTEs, window functions, query tuning, working with large datasets. Data Storage Formats ADLS Gen2, Delta Lake, Parquet, partitioning, ZOrdering, Eligible to workIMIZE VACUUM. Ops CICD Git, branching strategies, CICD for ADF Databricks environment promotion Infrastructure as Code exposure is a plus.

Minimum years of experience

8-10 years

Certifications Needed

No

Top 3 responsibilities you would expect the Subcon to shoulder and execute

Coding and Development

UI testing

Stakeholder management