
Automation Engineer - AI (26-09847) -
ClifyX · Location not stated
Strong hiring record · median 17 days open
- First seen
- 19 November 2026
- Reposted
- Not since September 2026
All reasons
- No location stated
In its favour
- Posted today
- Still listed on the employer's own board
Key Responsibilities
Design & Build AI Eval Pipelines
Architect automated evaluation frameworks measuring faithfulness, answer completeness, hallucination rates, and task completion across complex conversational and agentic workflows.
LLM-as-a-Judge & Heuristic Scoring
Develop, calibrate, and maintain rubric-based LLM-as-a-Judge evaluators.
Adversarial & Safety Testing (Guardrail Validations)
Design systematic stress tests for prompt injection, jailbreaks, PII leakage, guardrail boundaries, and adversarial edge cases.
Partner with ML/Platform Engineers
Inspect distributed traces, logs, and telemetry to pinpoint whether failures stem from prompts, models, RAG retrieval, or downstream connectors.
Synthetic Test Data Generation
Build programmatic pipelines to generate diverse persona simulations, multi-turn edge cases, and high-quality golden evaluation datasets.
CI/CD Integration
Integrate automated regression suites into build/release pipelines to gate prompt changes, model bumps, and backend API updates.
More jobs at ClifyX
All 204- Omniverse Platform EngineerClifyX · Irving, TXtoday
- Application Support Engineer-hands-on experience with WebSphere Commerce Suite (WCS) 8.xClifyX · Springfield, MOtoday
- Cybersecurity Segment HeadClifyX · Edison, NJtoday
- Network Project Engineer (26-09849) -ClifyX · Location not statedtoday
- Data Tech Lead – Data Architecture & Modelling (26-09848) -ClifyX · Location not statedyesterday
Similar roles
- QA Automation & Pipeline Engineer (CI/CD Engineering)Aquablu · Amsterdam, Noord-Holland, Netherlands · EUR 65,000–80,000/yrUpdated today
- Mobile Test Automation Engineer(Appium)Artech · San Antonio, TX · USD 37–38/hrtoday
- Senior Automation EngineerDoncasters · Oxford, AL, United Statestoday
- Senior Staff Agentic AI & Automation EngineerCircana · Bangalore, KA, Indiatoday
- Sr. Automation Engineer - IT Asset ManagementDatabricks · Bellevue, Washington · Mountain View, California · San Francisco, California +1 · USD 143,400–197,100/yrtoday
- Automation EngineerEarnix · Ramat Gan, Tel Aviv District, Israeltoday