
AI Engineer Lead
Spark Tek · Owings MIlls, MD
Weak hiring record · median 103 days open
- First seen
- 18 November 2026
- Reposted
- Not since September 2026
All reasons
- 78% of this employer's openings are over 60 days old
In its favour
- Posted yesterday
- Still listed on the employer's own board
Detailed JD Key Responsibilities
Technical Leadership & Stakeholder Management
Set the technical direction and roadmap for ML sustenance across a multi-service portfolio. Act as the primary onsite technical point of contact, leading and mentoring a globally distributed (follow-the-sun) team of engineers. Drive design reviews, agree on service-level objectives (SLOs), and partner with downstream client teams to onboard new AI agents and standardize model evaluation processes.
ML/GenAI Architecture & Pipelines
Design, build, and sustain automated continuous training (CT), data ingestion, and feature engineering pipelines (utilizing concepts similar to Kubeflow, MLflow, or Apache Airflow). Ingest datasets from distributed SQL or messaging systems (like Apache Kafka). Establish robust versioning and governance for models, prompts, and evaluation configurations, setting up experiment tracking and hyperparameter tuning to optimize performance.
Agentic Evaluation Infrastructure & Quality
Configure isolated/hermetic agent environments (using Kubernetes/Docker principles), integrating tools for synthetic data generation and simulated execution traces. Build and sustain evaluation pipelines, golden datasets, and quality dashboards to continuously monitor model output. Set up configurable alerting based on specific metric thresholds (e.g., Accuracy, AUC, Trajectory Exact Match, ROUGE) to proactively identify semantic drift or hallucinations, and audit automated evaluators (LLM-as-a-judge).
Deployment, Serving, Reliability & Observability
Build and maintain CI/CD workflows for safe, automated deployment and rollback of containerized ML models. Set up comprehensive observability (metrics, logs, traces) to monitor production health and low-latency serving. Participate in operational support to triage incidents, mitigate production issues, manage traffic rollouts, and drive root-cause analysis to resolve capacity bottlenecks and lower MTTR.
Privacy & Security Implementation
Enforce strict Data Isolation Models for sensitive data. Implement strict Identity and Access Management (IAM), Role-Based Access Control (RBAC), and adhere to rigorous data retention and deletion policies (aligned with Zero Trust and strict compliance standards) across all tenant onboardings and platform versions.
Operational Automation & Cost Optimization
Develop AI-assisted operational automation (e.g., monitoring agents, auto-triage, and documentation-sync hooks) to scale the team's impact. Optimize compute and token costs across serving and evaluation workloads to ensure cost-efficient scaling of AI capabilities.
Minimum Qualifications
- 8+ years of experience in software engineering, ML engineering, or MLOps/DevOps, focusing on the ML lifecycle (continuous training, model deployment, and continuous monitoring), including 2+ years in a technical lead role.
- Strong experience with Google Cloud Platform (GCP) MLOps services (Vertex AI, Kubeflow) or equivalent open-source ecosystems (Apache Airflow, MLflow).
- Proficiency in programming languages such as Python or Go, and experience with large-scale data processing and distributed systems.
- Experience working with large-scale production deployment, container orchestration (e.g., Kubernetes, Docker), and CI/CD pipelines (e.g., Jenkins, GitLab CI).
Experience with production operations for distributed systems
monitoring and alerting, SLO definition, on-call incident triage, and root-cause analysis.
- Proven ability to design and implement end-to-end ML pipelines for data management, model training, evaluation, and deployment.
- Demonstrated ability to lead and mentor engineers across time zones, communicate clearly with technical and non-technical stakeholders, and manage multiple partner priorities.
Preferred Qualifications
- Familiarity with Large Language Models (LLMs) and open-source Agentic AI frameworks (e.g., LangChain, AutoGen, LlamaIndex, or internal equivalent like Agent Development Kit).
- Knowledge of standard ML evaluation metrics for Generative and Classification models (e.g., BLEURT, ROUGE, Precision/Recall, AUC, Trajectory Exact Match).
- Experience managing highly sensitive data (e.g., PII/PHI) and implementing strict access control and encryption mechanisms.
- Strong analytical and debugging skills, with the ability to troubleshoot complex, distributed ML systems and infrastructure dependencies.
- Experience monitoring model quality in production, including data/concept drift detection, A/B testing, and building quality dashboards (e.g., Grafana, Datadog)
- Experience running knowledge-transfer and service-transition programs, or working in a sustenance, support-engineering, or SRE organization.
Skills
- Machine Learning
- Generative AI
- Kubeflow
- MLflow
- Airflow
- SQL
- Kafka
- Kubernetes
- Docker
- LLM
- IAM
- RBAC
- Zero Trust
- MLOps
- GCP
- Vertex AI
- Python
- Go
- Jenkins
- GitLab CI
- LangChain
- AutoGen
- LlamaIndex
- Grafana
- Datadog
More jobs at Spark Tek
All 138- AI Product Owner / LeadSpark Tek · Owings MIlls, MD2d ago
- Job Title/Role Java Full Stack + AI ArchitectSpark Tek · Owings MIlls, MD2d ago
- Integration Developer (26-02391) -Spark Tek · Remote13d ago
- Oracle MySQL Database Architect (26-02390) -Spark Tek · Remote13d ago
- Java Developer (26-02387) -Spark Tek · Location not stated13d ago
Similar roles
- Software Engineer Fire Control SpecialistContract Professionals · Sterling Heights, MI · USD 50–56/hrtoday
- Senior AI Frontend EngineerTeamViewer Germany · Austin, United Statestoday
- Staff Software Engineer, Affirm BankAffirm · US · USD 230,000–290,000/yrtoday
- API Back End DeveloperThe Computer Merchant · Boston, MA · USD 60–66/hrtoday
- Sr. Staff Software Development EngineerZscaler · Santa Clara, California, USA · USD 180,000–225,000/yrtoday
- Lead Software Engineer - AWS | Full Stack Java DevelopmentChase- · OH, United Statestoday