
Lead Data Engineer / Architect
Sidram Technologies · Cary, NC, United States
Weak hiring record · median 370 days open
- First seen
- 17 November 2026
- Reposted
- Not since September 2026
- Last checked
- Today · still on the employer's board
All reasons
- 89% of this employer's openings are over 60 days old
In its favour
- Posted yesterday
- Still listed on the employer's own board
Role
Lead Data Engineer / Architect
Location
Cary, NC (On-site / Hybrid)
Experience :12–18 Years – Visa Independent candidates – No H1 transfer
Employment
Full-Time with Altimetrik-
Customer
MetLife
ABOUT THE ENGAGEMENT
This is a senior hands-on leadership role. You will own the end-to-end technical design of the data and AI layers, build the reference implementations your engineers work from, and ship production-grade Python, Scala, and PySpark code every week. If you have not written or reviewed production code in the past year, this is not the right fit.
MUST-HAVE SKILLS & EXPERIENCE
Programming & Data Engineering
Expert - level proficiency in Python, Scala, and PySpark, with a strong track record of designing and delivering production-ready, modular, and well-tested solutions; developing and troubleshooting Spark workloads; and optimizing large-scale batch and streaming data pipelines using Delta Lake and Spark technologies.
Strong SQL and data modelling — dimensional and normalised; schema design and data contract definition.
Databricks expertise — Delta Lake, Unity Catalog, Jobs & Workflows, cluster and pool management, performance tuning, Model Serving.
Azure data stack — ADLS Gen2 (zone design, ACLs, lifecycle), Azure Data Factory (parameterized / metadata-driven frameworks, error handling), Azure Event Hubs.
AI & Machine Learning
3+ years designing and shipping
LLM-based systems in production
RAG pipelines, agentic / tool-calling workflows, structured output, chunking and embedding strategy, vector and hybrid retrieval, and prompt engineering.
Evaluation discipline — golden datasets, regression suites, accuracy and hallucination tracking, human-in-the-loop feedback loops; you measure AI quality, not assert it.
Hands-on experience with LangChain, LlamaIndex, or LangGraph, plus at least one provider stack (Azure OpenAI, OpenAI, or Databricks Model Serving).
Metadata-driven thinking — schema inference, data profiling, lineage, catalogs, and configuration-driven frameworks that onboard the next source without new code.
Architecture & Governance
12–18 years of total experience in data engineering, data platform delivery, or related disciplines.
Proven delivery of a medallion / lakehouse architecture at enterprise scale — not just familiarity with the concept.
Azure security and governance — Entra ID, managed identities, RBAC, POSIX ACLs on ADLS Gen2, Key Vault, private endpoints, and PII handling.
CI/CD and infrastructure as code — Azure DevOps, Terraform, Databricks Asset Bundles, and automated testing of data pipelines.
Clear technical writing and the ability to present and defend a design to both engineers and non-technical stakeholders.
STRONGLY PREFERRED
Knowledge graphs and ontologies
RDF/SPARQL, property graphs (Neo4j), or graph modelling over a lakehouse.
Text-to-SQL or semantic-layer-backed natural-language query systems at enterprise scale, including access control and ambiguity handling.
ML-based anomaly detection on time-series or transactional financial data.
Financial services or insurance domain knowledge
finance close, general ledger, subledger, reconciliation, or actuarial data.
LLMOps and MLOps
model versioning, prompt versioning, cost governance, and observability tooling.
Databricks Data Engineer Professional, Azure DP-203 / DP-700, or AZ-305 certification.
dbt, Great Expectations, or similar data-quality and transformation tooling.
Workday, Prism, or Accounting Center exposure.
Rachael
IT Services | Development | Staffing
rachael@sidramtech.com
Skills
- Governance
- Metadata
- Architecture
- Data Pipelines
- Microsoft Azure
- Data Profiling
- PySpark
- Performance Tuning
- Neo4j
- Databricks
- Accounting
- Artificial intelligence
- Data Streaming
- OpenAI
- Generative AI
- Apache Spark
- leadership
- Conceptual Models
- Python (Programming Language)
- Data Modelling
- Continuous Integration
- Automation of Tests
- Knowledge of Finance
- Computer Programming
- Data Lakes
- Information Engineering
- Financial Data Analysis
- Access Controls
- General Ledger Documents
- Management of Software Versions
- Prompt Engineering
- Datadog
- Role-Based Access Control
- Azure Data Factory
- Prism (Software)
- Machine Learning Operations
- Catalogues
- Expediting
- Graph Database
- Engineering Design Process
- Anomaly Detection
- Time Series
- Retrieval-Augmented Generation
- Scala (Programming Language)
- Key Vault
- Azure Data Lake
- Production Code
- Actuarial Science
- Llamaindex
- LangGraph Framework
- Posix
- Aquatic Facility Operator
- Great Expectations (Foster Youth College-readiness and Support Program)
- Information Retrieval
More jobs at Sidram Technologies
All 2,898- Data AnalystSidram Technologies · Waukegan, IL, United Statestoday
- CYBERSECURITY ENGINEER - IDENTITY & ACCESS MANAGEMENT (IAM)Sidram Technologies · Charlotte, NC, United Statestoday
- TPL - Engineering - Specialized - Quality/Test Engineer 4Sidram Technologies · Pontiac, IL, United Statestoday
- Palantir AI EngineerSidram Technologies · Connecticut · United Statestoday
- Order Management Cloud EngineerSidram Technologies · Durham, NC, United Statestoday
Similar roles
- Data Architect / Data EngineerSky Solutions · Tysons Corner, VA · USD 81,462–170,703/yrtoday
- Director of Data Platform & EngineeringGiftHEALTH · Columbus, OH · USD 164,000–225,000today
- Software Engineer (AI Data Engineering)SpaceX · Hawthorne, CA · USD 125,000–160,000/yrtoday
- Sr. Software Engineer (AI Data Engineering)SpaceX · Hawthorne, CA · USD 165,000–265,000/yrtoday
- Embedded Software Engineer, Data Center Engineering (Starmind)SpaceX · Bastrop, TXtoday
- Computer Engineer II - Naval Surface Warfare CenterTLN Worldwide Enterprises · Port Hueneme, CAtoday