Principal Engineer - Databricks Data Engineer

Job Description

• Audit the current Databricks workspace state, including environments, clusters, Unity Catalog setup, access controls, and cost configuration.
• Implement and harden Unity Catalog, including namespace structure, access controls, cluster policies, and environment separation.
• Define workspace standards, naming conventions, access request workflows, and user/team onboarding processes.
• Establish data quality frameworks, including SLAs, certification criteria, and data ownership models.
• Build automation templates for routine workspace operations such as cluster lifecycle management, cost alerting, and utilization monitoring.
• Stand up observability dashboards covering platform health, cost, job success rates, and DQ SLA KPIs, and monitor core data asset lineage across workspaces.
• Create runbooks for recurring operational tasks that can be executed by a mid-level/junior engineer without ongoing support from the senior resource.
• Conduct knowledge-transfer sessions with the platform engineering team.
• Provide final documentation covering architecture decisions, operational standards, open items, and recommendations for the permanent hire profile.

Key Responsabilities.

• Evaluate and optimize Databricks environments, including workspaces, clusters, catalogs, permissions, and security configurations.
• Define and implement governance standards to ensure secure, scalable, and compliant platform operations.
• Lead Unity Catalog adoption, designing data structures, access models, and environment segregation strategies.
• Establish RBAC policies, workspace standards, naming conventions, and user access processes.
• Drive data governance initiatives, including data quality frameworks, ownership models, SLAs, lineage, and stewardship practices.
• Develop automation solutions and Infrastructure as Code (IaC) frameworks to streamline platform administration and reduce manual effort.
• Implement cost optimization, monitoring, and operational excellence practices across Databricks environments.
• Design observability and performance dashboards to track platform health, utilization, job success rates, and data quality KPIs.
• Create operational documentation, runbooks, and support procedures to improve platform reliability and team readiness.
• Partner with security, compliance, infrastructure, and data engineering teams to align platform capabilities with business needs.
• Provide training, mentoring, and knowledge transfer sessions to empower platform administrators and engineering teams.
• Recommend platform improvements and long-term strategies to enhance governance, scalability, and organizational maturity.

Must have skills

Azure Data Factory, Databricks, Unity Catalog

Good to have skills

Data platform implementation, Google Workspace Administration