Data Architect (Databricks, Azure) - Hybrid Onsite

Marathon TS is looking to hire an onsite Data Architect (Databricks, Azure) to support a Federal Agency in Washington DC. We are looking for a Data Architect to design the data foundation for a generative AI program at a regulatory client. You will own how business applications and a Databricks lakehouse exchange data, so that every AI workflow has clean inputs, stable output contracts and governed access to highly confidential information.

What you'll do

• Define the interface contracts between business applications and AI workflows: inputs, output schemas, versioning and error handling.
• Design ingestion of large document submissions (PDF, Excel and Word, with single submissions of up to hundreds of gigabytes) from Azure storage into Databricks, including metadata, document keys and text extraction.
• Design the Unity Catalog model for confidential data: catalogs, schemas, fine-grained access controls tied to user assignments, and lineage.
• Model AI outputs and reviewer feedback: the AI draft versus the human final, provenance fields (model and prompt version, evidence links, confidence) and the write-back tables used to refine prompts.
• Recommend the operational data store pattern (Azure SQL, PostgreSQL or Databricks Lakebase) and how it stays in sync with the lakehouse.
• Plan and integrate the historical data the AI workflows depend on.
• Keep compute cost-efficient through cluster policies, job design and consumption tracking.

What you bring

• 10+ years in data architecture, including 4+ years designing Databricks lakehouses (Delta Lake, Unity Catalog, Workflows or Lakeflow).
• Azure data and security: ADLS or Blob Storage, private endpoints, Entra ID and Key Vault.
• Experience designing data contracts between applications and a data platform, beyond warehouse or BI modeling.
• Governance for sensitive or regulated data: fine-grained access control, lineage and audit.
• Pipelines for unstructured documents at high volume.
• Databricks certification (Data Engineer Professional or architect level).

Nice to have

• Data patterns for generative AI: vector search and retrieval-augmented generation.
• Experience in financial regulation, audit or federal environments.

Working with us

• You will work inside a secure, private cloud environment with highly confidential data.
• You must be authorized to work in the United States and able to pass a background check.
• We are looking for people who can start within a few weeks.

#cjjobs

Marathon TS is committed to the development of a creative, diverse and inclusive work environment. In order to provide equal employment and advancement opportunities to all individuals, employment decisions at Marathon TS will be based on merit, qualifications, and abilities. Marathon TS does not discriminate against any person because of race, color, creed, religion, sex, national origin, disability, age or any other characteristic protected by law (referred to as "protected status ").