JobHabor
Menu

Lead Data Engineer / Architect

Sidram Technologies · Cary, NC, United States

Weak hiring record · median 370 days open

100100

Hiring signal

Actively hiring

Posted today · Never reposted · Checked today · How we score

First seen
17 November 2026
Reposted
Not since September 2026
Last checked
Today · still on the employer's board
All reasons
  • 89% of this employer's openings are over 60 days old

In its favour

  • Posted yesterday
  • Still listed on the employer's own board

Did you apply?

Lead Data Engineer / Architect at Sidram Technologies. We’ll keep it in your applied jobs, so you can see it later and be warned before applying to the same employer twice.

Role

Lead Data Engineer / Architect

Location

Cary, NC (On-site / Hybrid)

Experience :12–18 Years – Visa Independent candidates – No H1 transfer

Employment

Full-Time with Altimetrik-

Customer

MetLife

ABOUT THE ENGAGEMENT

This is a senior hands-on leadership role. You will own the end-to-end technical design of the data and AI layers, build the reference implementations your engineers work from, and ship production-grade Python, Scala, and PySpark code every week. If you have not written or reviewed production code in the past year, this is not the right fit.

MUST-HAVE SKILLS & EXPERIENCE

Programming & Data Engineering

Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-scale batch and streaming data pipelines using Delta Lake and Spark technologies.

Strong SQL and data modelling — dimensional and normalised; schema design and data contract definition.

Databricks expertise — Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.

Azure data stack — ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks, error handling), Azure Event Hubs.

AI & Machine Learning

3+ years designing and shipping

LLM-based systems in production

RAG pipelines, agentic / tool-calling workflows, structured output, chunking and embedding strategy, vector and hybrid retrieval, and prompt engineering.

Evaluation discipline — golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback loops; you measure AI quality, not assert it.

Hands-on experience with LangChain, LlamaIndex, or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).

Metadata-driven thinking — schema inference, data profiling, lineage, catalogs, and configuration-driven frameworks that onboard the next source without new code.

Architecture & Governance

12–18 years of total experience in data engineering, data platform delivery, or related disciplines.

Proven delivery of a medallion / lakehouse architecture at enterprise scale — not just familiarity with the concept.

Azure security and governance — Entra ID, managed identities, RBAC, POSIX ACLs on ADLS Gen2, Key Vault, private endpoints, and PII handling.

CI/CD and infrastructure as code — Azure DevOps, Terraform, Databricks Asset Bundles, and automated testing of data pipelines.

Clear technical writing and the ability to present and defend a design to both engineers and non-technical stakeholders.

STRONGLY PREFERRED

Knowledge graphs and ontologies

RDF/SPARQL, property graphs (Neo4j), or graph modelling over a lakehouse.

Text-to-SQL or semantic-layer-backed natural-language query systems at enterprise scale, including access control and ambiguity handling.

ML-based anomaly detection on time-series or transactional financial data.

Financial services or insurance domain knowledge

finance close, general ledger, subledger, reconciliation, or actuarial data.

LLMOps and MLOps

model versioning, prompt versioning, cost governance, and observability tooling.

Databricks Data Engineer Professional, Azure DP-203 / DP-700, or AZ-305 certification.

dbt, Great Expectations, or similar data-quality and transformation tooling.

Workday, Prism, or Accounting Center exposure.

Rachael

IT Services | Development | Staffing

Email

rachael@sidramtech.com

Skills

  • Governance
  • Metadata
  • Architecture
  • Data Pipelines
  • Microsoft Azure
  • Data Profiling
  • PySpark
  • Performance Tuning
  • Neo4j
  • Databricks
  • Accounting
  • Artificial intelligence
  • Data Streaming
  • OpenAI
  • Generative AI
  • Apache Spark
  • leadership
  • Conceptual Models
  • Python (Programming Language)
  • Data Modelling
  • Continuous Integration
  • Automation of Tests
  • Knowledge of Finance
  • Computer Programming
  • Data Lakes
  • Information Engineering
  • Financial Data Analysis
  • Access Controls
  • General Ledger Documents
  • Management of Software Versions
  • Prompt Engineering
  • Datadog
  • Role-Based Access Control
  • Azure Data Factory
  • Prism (Software)
  • Machine Learning Operations
  • Catalogues
  • Expediting
  • Graph Database
  • Engineering Design Process
  • Anomaly Detection
  • Time Series
  • Retrieval-Augmented Generation
  • Scala (Programming Language)
  • Key Vault
  • Azure Data Lake
  • Production Code
  • Actuarial Science
  • Llamaindex
  • LangGraph Framework
  • Posix
  • Aquatic Facility Operator
  • Great Expectations (Foster Youth College-readiness and Support Program)
  • Information Retrieval

More jobs at Sidram Technologies

All 2,898

Similar roles