Senior Databricks AI Engineer

Job Description: Senior Databricks AI Engineer

• Location: Atlanta, GA
• Duration: 6 Months
• Experience Required: 4–6 Years
• Role: Senior Databricks AI Engineer

Position Summary

• Design, develop, and deploy scalable data and AI solutions using the Databricks Lakehouse Platform.
• Apply expertise in Data Engineering, Machine Learning, Generative AI, and cloud technologies.
• Build enterprise-grade AI applications and data platforms.
• Develop scalable, secure, and efficient data pipelines and AI solutions.

Key Responsibilities

Data Engineering

• Design and develop scalable data pipelines using Databricks, PySpark, Delta Lake, and Spark SQL.
• Build ELT/ETL solutions for structured and unstructured data.
• Implement Medallion Architecture, including Bronze, Silver, and Gold layers.
• Optimize Databricks jobs, clusters, and workflows for performance and cost efficiency.
• Manage data ingestion from multiple cloud and on-premises sources.
• Design, develop, and maintain reliable data pipelines.
• Support data modernization initiatives and enterprise data platform development.

AI & Machine Learning

• Develop and deploy Machine Learning models using Databricks ML and Python.
• Implement MLOps practices for model training, deployment, monitoring, and governance.
• Build predictive analytics solutions and recommendation engines.
• Create feature stores and reusable ML pipelines.
• Implement AI model deployment and lifecycle management.
• Monitor and optimize machine learning models in production environments.

Generative AI

• Design and develop Generative AI applications using:
• Large Language Models (LLMs).
• Retrieval-Augmented Generation (RAG).
• Vector Databases.
• AI Agents and Multi-Agent Frameworks.
• Integrate various AI technologies to build enterprise AI solutions.
• Develop agentic AI applications and workflows.
• Collaborate with stakeholders to deliver AI-driven business solutions.

Essential Skills

• Strong expertise in Databricks and Data Engineering.
• Proficiency in Machine Learning and AI model deployment.
• Experience with cloud platforms such as Microsoft Azure or AWS.
• Familiarity with data pipeline design, development, and optimization.
• Strong knowledge of Generative AI technologies.
• Experience with PySpark, Spark SQL, and Delta Lake.
• Understanding of ETL/ELT processes and Medallion Architecture.
• Knowledge of AI model lifecycle management and deployment.

Role-Specific Requirements

• Design and develop scalable data pipelines using Databricks.
• Implement machine learning models and MLOps practices.
• Optimize data workflows for performance and cost efficiency.
• Collaborate with stakeholders to deliver AI-driven solutions.
• Develop Generative AI applications and integrate AI technologies.

Essential and Desirable Qualifications

• Experience in healthcare or banking domains is beneficial.
• Familiarity with AI governance and agentic AI is a plus.
• Knowledge of MLOps practices is advantageous.
• Understanding of data modernization programs is helpful.
• Databricks or Machine Learning certification is preferred.

Technical Skills / Keywords

• Digital: Machine Learning
• Digital: Cloud Foundry
• Digital: Databricks
• Digital: PySpark
• Generative AI

Desirable Skills

• Advanced Databricks and Lakehouse Platform knowledge.
• Experience with enterprise AI application development.
• Familiarity with AI governance, model monitoring, and responsible AI practices.
• Experience building RAG pipelines, vector search solutions, and AI agents.
• Knowledge of cloud-based data and AI architecture.