
Senior Infrastructure Engineer – Terraform & Databricks
Apptad · San Francisco, CA, United States · Hybrid
Weak hiring record · median 66 days open
- First seen
- 18 November 2026
- Reposted
- Not since September 2026
- Last checked
- Today · still on the employer's board
All reasons
- 55% of this employer's openings are over 60 days old
In its favour
- Posted today
- Still listed on the employer's own board
Senior Infrastructure Engineer – Terraform & Databricks
San Francisco, CA
Need candidates who can work onsite from Day 1 (3 days a week : Tuesday, Wednesday and Thursday – 3 days work from office) – NO REMOTE
Highly preferred local candidates
Experience: 09+ years
About the role
Building an enterprise Databricks Lakehouse on AWS, and all of it is managed as code. In this role you'll design, build and run the Terraform/Terragrunt foundation behind our Databricks workspaces, Unity Catalog, compute and security controls.
You'll work closely with Cloud Infrastructure/SRE, Security Engineering and Data Engineering. Your goal is to take our platform from sandbox to production-ready multi-environment setup that is secure, well governed and cost-efficient.
What you will do
- Design, build and maintain reusable Terraform modules for Databricks account-level (MWS) and workspace-level resources. This covers workspaces, Unity Catalog (metastores, catalogs, schemas, volumes, grants), cluster policies, instance pools, SQL warehouses, jobs, DLT pipelines and service principals.
- Use Terragrunt to manage environments consistently across sandbox, nonprod, preprod and prod, with minimal configuration drift.
- Provision and secure the supporting AWS infrastructure: customer-managed VPCs, PrivateLink/VPC endpoints, security groups, NACLs, IAM roles and policies, S3, KMS customer-managed keys (CMK) and Secrets Manager.
- Lead security hardening. That means fixing findings from security architecture reviews and cloud security posture management (CSPM) tools such as Wiz, enforcing PrivateLink-only access, IP access lists, CMK encryption and secret/OAuth token rotation.
- Set up identify and access controls, including Okta SSO and SCIM sync, role-based (RBAC) and tag-based (ABAC) access in Unity Catalog, and least-privilege service principals.
- Build and own CI/CD for infrastructure. This includes GitHub Actions plan/apply workflows with approvals, pre-commit hooks, Terratest, and security scanning with Checkov or tfsec.
- Set up monitoring and alerting with CloudWatch, Grafana and Splunk log forwarding. That covers cluster and job health, token expiry and cost anomalies.
- Drive FinOps practices: cluster policy guardrails, spot usage, moving work-loads from all-purpose to job clusters, enforcing tagging, and analysing DBU usage.
- Enable integrations with Dagster, dbt Cloud and BI tools such as Tableau, Power BI and Looker via SQL warehouses.
- Write and maintain architecture docs, ADRs, runbooks and module references, and enforce naming and tagging standards.
- Databricks – Hands-on build data pipelines using Pyspark
- Used Agentic IDE to build data pipelines
- Understands Medallion architecture
- Understands ML flow – Model inferencing
- Have domain expertise or understanding of Credit decisioning or Risk or Servicing (collections)
Skills Required
- 7+ years in infrastructure, platform or DevOps engineering with atleast 3 years of hands-on Terraform in production. Terragrunt experience is strongly preferred.
- 2+ years running Databricks on AWS, including workspace provisioning, Unity Catalog, cluster policies and the Databricks Terraform provider.
- Deep AWS knowledge; VPC networking, PrivateLink, IAM, KMS, S3, Secrets Manager and CloudWatch.
- Experience with CI/CD for Infrastructure as code (GitHub Actions, Jenkins or similar) and with GitOps workflows.
- Strong grasp of cloud security and compliance: encryption, network isolation, least privilege and audit logging. Experience in regulated environments such as financial services is a plus.
- Scripting skills in Python and/or Bash. Go is a plus for Terratest.
- Comfortable with Git, Code reviews and writing clear technical documentation.
Nice to have
- Databricks certifications (Data Engineer Associate/Professional or Platform Administrator), or HashiCorp Terraform Associate.
- Experience with Dela Live Tables, Vector Search, Model Serving or MLops Infrastructure.
- Familiarilty with Dagster, Dbt, Spark and Delta Lake.
- Experience with Wiz, Checkov, tfsec or similar policy-as-code and CSPM tools.
- Background in FinOps or cloud cost optimization.
- Experience with SOC 2 or similar compliance frameworks
Skills
More jobs at Apptad
All 278- specialized full-stack developerApptad · Georgia · United Statestoday
- L3 Infrastructure & Virtualization EngineerApptad · Houston, TX, United Statestoday
- Windows Infrastructure Automation EngineerApptad · Remotetoday
- AWS Cloud ArchitectApptad · New York City, NY, United States · USD 90–95/daytoday
- Senior DevOps Engineer with TerraformApptad · Oaks, PA, United Statestoday
Similar roles
- Network Engineer IIIHygiena · Camarillo, CA, United States · USD 48–60/hrtoday
- Sr Director Special ProgramsOracle · Nashville, TN, United States · Austin, TX, United States · Denver, CO, United States +3 · USD 139,400–291,800/yrtoday
- AWS Cloud ArchitectApptad · New York City, NY, United States · USD 90–95/daytoday
- Senior DevOps Engineer with TerraformApptad · Oaks, PA, United Statestoday
- Software & DevOps EngineerLorven technologies · Chicago, IL, United States · USD 65/hrtoday
- VSU - IT Support Specialist – (815995)Lorven technologies · Petersburg, VA, United States · USD 30–40/hrtoday