
Senior AI Agent Evaluation Engineer
Aditi Consulting · Bangalore, KA
Strong hiring record · median 19 days open
- First seen
- 19 November 2026
- Reposted
- Not since September 2026
All reasons
In its favour
- Posted today
- Still listed on the employer's own board
Summary
The Senior AI Agent Evaluation Engineer is a hands-on technical role responsible for defining, developing, and executing robust evaluation frameworks and test strategies for AI agents built using the Google Agent Development Kit (ADK). This role bridges AI development and reliable deployment, ensuring our agents are safe, ethical, effective, and meet high-quality performance standards. A significant focus of the role is on Responsible AI and Safety Evals, including adversarial testing, red-teaming, bias detection, and toxicity measurement. The engineer will design scalable evaluation pipelines, build synthetic test environments, and work closely with AI/ML Engineers, Researchers, and policy teams to ensure agents are production-ready and compliant with regulatory and ethical standards. The role is 70% Automation and 30% Manual Testing, and sits within the AI team as a key quality authority.
Job Responsibilities
- Design, implement, and maintain scalable, repeatable evaluation datasets, metrics, and benchmarks to measure agent performance, robustness, safety, and alignment.
- Build synthetic testing environments and simulation strategies to stress-test agents under diverse real-world conditions.
- Develop and execute adversarial testing, jailbreaking, and red-teaming methodologies to identify potential harm, bias, toxicity, and unauthorized behavior in agent responses.
- Implement and measure adherence to established ethical guidelines, safety policies, and content-filtering mechanisms across all agent deployments.
- Partner with policy and legal teams to ensure agent evaluations cover regulatory compliance and fairness objectives.
- Define comprehensive end-to-end QA strategies for functional, integration, regression, and user acceptance testing (UAT) tailored for conversational and goal-oriented AI agents.
- Develop and maintain detailed test artifacts covering agent features, tool use, memory, and reasoning capabilities.
- Identify, document, prioritize, and track defects, performance regressions, and alignment failures using Jira, collaborating with AI/ML Engineers and Researchers to analyze root causes and validate fixes.
- Integrate evaluation pipelines into CI/CD processes to enable continuous quality assurance and fast iteration cycles.
- Analyze and interpret evaluation results, providing clear, actionable quality and safety reports to stakeholders and development teams, with a specific focus on risk metrics and mitigation recommendations.
Required Skills
- Direct experience designing and executing safety evaluations including red-teaming, adversarial testing, jailbreaking, bias detection, and toxicity/harmful content measurement for generative AI models.
- Proven track record building and running general evaluations for LLM-powered applications, including designing evaluation datasets, metrics, and scoring rubrics.
- Direct or strong conceptual understanding of the Google Agent Development Kit (ADK) and its components.
- Experience designing and executing prompt injection attacks and defenses to test agent robustness.
- Hands-on experience with evaluation frameworks and libraries such as LangSmith, DeepEval, Ragas, Giskard, or Hugging Face evaluation tooling.
- Strong proficiency in Python for script development, data processing, test automation, and evaluation pipeline development; PyTest is mandatory.
- Familiarity with Google Cloud Platform (GCP) services relevant to AI/ML and experience integrating testing into MLOps workflows.
- Experience with CI/CD pipelines and Git-based workflows for automated evaluation and regression testing.
- Excellent analytical and problem-solving skills, with the ability to decompose complex agent behaviors into testable scenarios and measurable outcomes.
- Strong communication skills to articulate complex agent behaviors, risks, and safety findings clearly to both technical and non-technical audiences.
- Experience working in agile environments, collaborating closely with AI/ML Engineers, Researchers, product owners, and distributed stakeholders.
Education & Qualifications
- Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience.
- 8-10 years of overall Software QA experience, with at least 2-3 years focused on testing or evaluating AI/ML systems, conversational agents, or Large Language Models (LLMs).
- Fundamental understanding of how LLMs work, including concepts like RAG, prompting strategies, fine-tuning, and the technical challenges related to alignment.
- Proficiency with data analysis and visualization tools (e.g., Pandas, SQL) for analyzing evaluation and safety results at scale.
- Familiarity with AI ethics principles and industry standards for responsible AI development.
- Experience testing complex APIs, especially for tool-integration and function-calling scenarios in agentic workflows.
- Experience with GIGO testing tools or frameworks that specialize in testing model responses to highly ambiguous, misleading, or malformed inputs.
- Awareness of data poisoning vectors and testing strategies for training and retrieval pipelines.
- Certifications in AI/ML, cloud platforms (GCP), or software testing methodologies.
#AditiConsulting
#26-07406
Skills
- Jira
- Generative AI
- LLM
- LangSmith
- Hugging Face
- Python
- pytest
- GCP
- MLOps
- Git
- Retrieval-Augmented Generation
- Pandas
- SQL
More jobs at Aditi Consulting
All 109- Senior Test Automation EngineerAditi Consulting · Bangalore, KAtoday
- Applied AI EngineerAditi Consulting · Bangalore, KAtoday
- User Experience (UX) DesignerAditi Consulting · Austin, TX · USD 95–101/hrtoday
- Full Stack DeveloperAditi Consulting · Richfield, MN · USD 75/hrtoday
- QA LeadAditi Consulting · Brownsburg, IN · USD 65–70/hrtoday
Similar roles
- AI Assurance / AI Evaluation EngineerClifyX · MUMBAI5d ago
- Principal AI Engineer - Agentic Systems and Evaluation PlatformAmgen · India - Hyderabad6d ago
- Hardware Engineering Expert - AI Training & EvaluationJobgether · India · USD 60–120/hrtoday
- Machine Learning EngineerAmgen · India - Hyderabad16d ago
- Sr AI/ML Engineer, Applied AIThermo Fisher Scientific · Bangalore, Karnātaka, IndiaUpdated 2d ago
- Software Engineer IKaseya Careers · Pune, India6d ago