Senior Datbricks Architect

Company Name

VARITE India Private Limited

About The Client

Client is an asset-light North American freight and logistics company that provides time-definite, expedited surface transportation as a reliable, cost-effective alternative to shipping cargo by air

About The Job

• We are looking for a senior, hands-on Databricks architect who has also designed enterprise data architectures end to end.
• You will define the target lakehouse architecture, establish Unity Catalog as the backbone of data governance, and lead the migration of legacy warehouse workloads onto Databricks.
• This is a builder role: you will write code, review pipelines, and set the technical standards the team follows — not just produce diagrams.

Essential Job Functions

Architecture & Design

• Define and own the target lakehouse architecture on Azure Databricks, including medallion (bronze / silver / gold) layering, storage design on ADLS Gen2, and integration patterns with Azure Data Factory.
• Design canonical and dimensional data models for finance, pricing, and operational domains, and set modeling standards the whole team follows.
• Produce architecture decision records (ADRs) so that platform choices are documented, reviewable, and reproducible.
• Design ingestion patterns for diverse source systems including AS400 / DB2, Oracle, SQL Server, transportation management systems, and SaaS finance applications.

Data Governance with Unity Catalog

• Design and implement the Unity Catalog structure (metastores, catalogs, schemas) aligned to environments and business domains.
• Implement fine-grained access control, row- and column-level security, and data masking for sensitive data.
• Establish data ownership and stewardship, lineage, tagging, and audit practices using Unity Catalog capabilities.
• Partner with Finance, Information Security, and business data owners to translate governance policies into enforceable platform controls.
• Define data quality rules and expectations (e.g., Delta Live Tables expectations or equivalent) and embed them into pipelines.

Hands-On Engineering

• Build and review PySpark, Spark SQL, and Delta Lake code; lead by example on code quality, testing, and documentation.
• Design orchestration using Databricks Workflows and ADF with clear dependencies, retries, and failure handling.
• Tune performance and cost: cluster policies, job compute vs. all-purpose compute, Photon, partitioning, Z-ordering / liquid clustering, and table maintenance.
• Troubleshoot complex production issues and drive root-cause fixes rather than workarounds.

Migration & Modernization

• Plan and lead the migration of workloads from legacy warehouses and semantic layers (e.g., Redshift, SSAS) onto Databricks, including validation and cutover.
• Rationalize redundant pipelines and tools into a simpler, consistent platform.

Leadership & Collaboration

• Act as the technical authority for the data platform across US and India teams; run design reviews and mentor engineers.
• Work with DevOps to ensure every pipeline is deployed through CI/CD with environment parity across Dev, QA, and Production.
• Communicate architecture choices and trade-offs clearly to technical and non-technical stakeholders.

Qualifications

Required Skills & Experience

• 12–15 years in data engineering and data architecture, with 4+ years of hands-on Azure Databricks in production.
• Deep, practical experience implementing Unity Catalog: catalog design, access control, lineage, and governance policies.
• Expert in PySpark, Spark SQL, and Delta Lake, including performance tuning at scale.
• Strong data modeling skills (dimensional, data vault, or canonical) for enterprise reporting and analytics.
• Experience with Azure Data Factory, ADLS Gen2, Azure Key Vault, and Entra ID integration.
• Proven record of migrating workloads from legacy data warehouses to a lakehouse.
• Working knowledge of CI/CD for data (Azure DevOps or GitHub), Git branching, and Databricks Asset Bundles.
• Excellent written and verbal communication; comfortable presenting to senior leadership.

Preferred Skills

• Databricks Certified Data Engineer Professional or Databricks Solutions Architect certification.
• Experience in transportation, logistics, or supply chain data (shipments, freight billing, pricing, carrier data).
• Experience with finance data domains (GL, revenue, costing) and ERP systems such as NetSuite or Oracle.
• Experience integrating Power BI with Databricks SQL warehouses.
• Experience with Terraform for Databricks workspace and Unity Catalog configuration.

What Success Looks Like In The First 90 Days

• Has documented the current-state platform and delivered a reviewed target lakehouse architecture with ADRs.
• Has a Unity Catalog design approved and the first governed domain live with access controls and lineage.
• Has published modeling and coding standards and is actively reviewing team pull requests.
• Has a sequenced migration plan for legacy warehouse workloads with validation criteria.

Working Arrangement

• Based in Mysore, India (hybrid).
• The role requires a daily overlap of 3–4 hours with U.S. Central / Eastern business hours for stand-ups, design reviews, and release coordination.

Education

• Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, or equivalent practical experience.

How to Apply

Interested candidates are encouraged to respond/submit their updated resumes, and for additional job opportunities, please visit Jobs In India – VARITE.

Unlock Rewards

Refer Candidates and Earn.

If you're not available or interested in this opportunity, please pass this along to anyone in your network who might be a good fit and interested in our open positions. VARITE offers a Candidate Referral program, where you'll receive a one-time referral bonus based on the following scale if the preferred candidate completes a three-month assignment with VARITE.

Experience Level Bonus Referral

• 0-2 years - INR 5,000
• 2-6 years - INR 7,500
• 6+ years - INR 10,000

About VARITE

VARITE is a global staffing and IT consulting company providing technical consulting and team augmentation services to Fortune 500 Companies in USA, UK, CANADA and INDIA. VARITE is currently a primary and direct vendor to the leading corporations in the verticals of Networking, Cloud Infrastructure, Hardware and Software, Digital Marketing and Media Solutions, Clinical Diagnostics, Utilities, Gaming and Entertainment, and Financial Services.

Equal Opportunity Employer

VARITE is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate based on race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, marital status, veteran status, or disability status.