Databricks Data & AI Analytics Architect

Role Description

We are seeking an experienced Data & AI Analytics Architect / Lead Engineer to modernize enterprise analytics platforms and drive adoption of AI-powered analytical solutions on Databricks. The role focuses on building scalable data ingestion frameworks, metadata-driven architectures, conversational analytic capabilities, and governed data products that improve business decision-making and reduce time to insight.

The organization operates multiple reporting and analytics solutions across Trading, Risk, Finance, and Operations using a mix of legacy data warehouse technologies and custom data pipelines.

Current challenges include

• Fragmented ingestion and transformation patterns resulting in higher maintenance effort.
• Ongoing migration of analytical workloads from legacy platforms to Databricks.
• Increasing demand for self-service analytics, conversational data access, and AI-assisted decision support.
• Need for improved data quality, lineage, governance, and auditability.
• Requirement to accelerate insight generation and improve analytical scalability, reliability, and performance.

Skills

• Strong expertise in Databricks, Spark, Delta Lake, Unity Catalog, SQL Warehouses, Databricks Workflows, Databricks Genie, and Genie One.
• Experience designing enterprise-scale data ingestion, ETL/ELT, metadata-driven, and watermark-based processing frameworks.
• Experience modernizing and migrating workloads from legacy data warehouses to Databricks.
• Strong SQL skills including query optimization, semantic modeling, and metadata-driven analytics.
• Experience building AI/LLM-powered analytics solutions, conversational interfaces, and agent-based architectures.
• Experience developing analytical assistants/copilots leveraging business metadata, governance rules, and semantic context.
• Ability to design conversational data exploration with drill-down analysis, context retention, and root-cause investigation.
• Experience with automated insight generation including KPI monitoring, trend analysis, anomaly detection, correlation analysis, and executive summaries.
• Good understanding of cloud data and analytics architectures covering ingestion, transformation, modeling, and serving layers.
• Knowledge of visualization and dashboarding concepts enabling business-friendly analytical experiences.
• Ability to work effectively across business and technology teams to improve data availability, reliability, and performance.
• Familiarity with Trading, Risk, Finance, Operations, or Supply Chain domains is highly desirable.