Lead Data Engineer with Databricks & AWS

Role

Lead Data Engineer with Databricks & AWS

Location

St. Louis, MO

Worku type

3 days onsite per week

Skill Requirements

• Primary focus:
• 60% Databricks
• 40% ETL experience (any ETL tool/platform)
• AWS is preferred but not mandatory.
• Project involves migration from Teradata to Databricks.

JD

Key Responsibilities

• Design and architect scalable, secure, and high-performance AWS and Databricks solutions, leveraging AWS Networking, Kinesis, Elastic Cache, Redshift, EKS, Cognito, and PaaS services
• Develop and maintain robust ETL/ELT pipelines using PySpark and Python within Databricks, including notebooks, jobs, Delta Lake tables, and Unity Catalog for governance
• Implement medallion architecture (bronze, silver, gold layers) and optimize Spark jobs for performance, cost, scalability, and reliability, addressing partitioning, skew, caching, and adaptive query execution
• Provision and manage cloud infrastructure using Terraform for Databricks workspaces, clusters, jobs, storage (ADLS/S3), networking, IAM roles and permissions, and related resources on Azure and AWS
• Write efficient SQL queries for data transformation, validation, and analytics within Databricks, ensuring data quality, lineage, security, and compliance
• Implement and maintain CI/CD pipelines using Jenkins and GitHub Actions for automated testing and deployment of notebooks, jobs, Delta Live Tables, and Terraform configurations
• Integrate data from diverse sources (databases, APIs, streaming, files) into cloud storage and processing layers
• Optimize cloud costs through auto-scaling clusters, spot instances, job scheduling, and efficient resource usage
• Monitor pipeline performance, troubleshoot failures, and implement alerting and observability using Databricks tools, cloud monitoring services, or third-party solutions

Required Skills

• Advanced proficiency in AWS Networking and VPC design
• Expertise in AWS Kinesis for real-time data streaming and analytics
• Deep experience with AWS Elastic Cache for distributed caching
• Extensive experience with AWS Redshift for data warehousing
• Strong knowledge of AWS PaaS services including Lambda, S3, and Glue
• Hands-on experience with AWS EKS for container orchestration
• Proficiency in AWS Cognito for identity and access management
• Expertise in Databricks, including Delta Lake, Unity Catalog, and Delta Live Tables
• Strong Python and PySpark skills for distributed data processing
• Advanced SQL skills for complex querying and optimization
• Proficiency in Terraform for infrastructure automation and management
• Experience with CI/CD tools such as Jenkins and GitHub Actions

Preferred Skills

• Knowledge of Azure Data Lake, Data Factory, Synapse, and Key Vault
• Understanding of big data modeling (star/snowflake, dimensional) and lakehouse architecture
• Experience with performance tuning in Spark and Databricks environments
• Familiarity with data governance, lineage, and compliance in cloud environments

Desired Qualifications

• Bachelor's degree in Computer Science, Information Technology, or a closely related discipline
• AWS Certified Solutions Architect – Professional
• Databricks Certified Data Engineer Professional or Apache Spark certification