
Senior AI Agent Evaluation Engineer
Varite India Private · Bangalore, KA
Strong hiring record · median 19 days open
- First seen
- 19 November 2026
- Reposted
- Not since September 2026
All reasons
In its favour
- Posted today
- Still listed on the employer's own board
Company Name
VARITE India Private Limited
About The Client
A data management and cloud services company headquartered in the United States specializes in simplifying backup, recovery, and data protection processes. The firm offers a cloud-based platform integrating backup, instant recovery, archival, search, and analytics functions. Known for its innovative data management approach, the company caters to organizations seeking efficient and scalable solutions.
About The Job
- The Senior AI Agent Evaluation Engineer is a hands-on technical role responsible for defining, developing, and executing robust evaluation frameworks and test strategies for AI agents built using the Google Agent Development Kit (ADK).
- This role bridges AI development and reliable deployment ensuring our agents are safe, ethical, effective, and meet high-quality performance standards.
- A significant focus of the role is on Responsible AI and Safety Evals, including adversarial testing, red-teaming, bias detection, and toxicity measurement.
- The engineer will design scalable evaluation pipelines, build synthetic test environments, and work closely with AI/ML Engineers, Researchers, and policy teams to ensure agents are production-ready and compliant with regulatory and ethical standards.
- The role is 70% Automation and 30% Manual Testing, and sits within the AI team as a key quality authority.
Essential Job Functions
- Design, implement, and maintain scalable, repeatable evaluation datasets, metrics, and benchmarks to measure agent performance, robustness, safety, and alignment (e.g., faithfulness, hallucination, prompt injection resistance).
- Build synthetic testing environments and simulation strategies to stress-test agents under diverse real-world conditions, specifically for agents utilizing Google ADK and related Google AI/ML services (e.g., Vertex AI, Gemini models).
- Develop and execute adversarial testing, jailbreaking, and red-teaming methodologies to identify potential harm, bias, toxicity, and unauthorized behavior in agent responses.
- Implement and measure adherence to established ethical guidelines, safety policies, and content-filtering mechanisms across all agent deployments.
- Partner with policy and legal teams to ensure agent evaluations cover regulatory compliance and fairness objectives.
- Define comprehensive end-to-end QA strategies functional, integration, regression, and user acceptance testing (UAT) tailored for conversational and goal-oriented AI agents.
- Develop and maintain detailed test artifacts (test plans, test cases, test scenarios) covering agent features, tool use, memory, and reasoning capabilities.
- Identify, document, prioritize, and track defects, performance regressions, and alignment failures using Jira, collaborating with AI/ML Engineers and Researchers to analyze root causes and validate fixes.
- Integrate evaluation pipelines into CI/CD processes to enable continuous quality assurance and fast iteration cycles.
- Analyze and interpret evaluation results, providing clear, actionable quality and safety reports to stakeholders and development teams, with specific focus on risk metrics and mitigation recommendations.
Qualifications
Required Qualifications
Education & Experience
- Bachelor's or Master's degree in Computer Science, Information Systems, Engineering, or equivalent practical experience.
- 8–10+ years of overall Software QA experience, with at least 2–3 years focused on testing or evaluating AI/ML systems, conversational agents, or Large Language Models (LLMs).
Technical Skills AI Agent Evaluation & Safety
- Direct experience designing and executing safety evaluations red-teaming, adversarial testing, jailbreaking, bias detection, and toxicity/harmful content measurement for generative AI models.
- Proven track record building and running general evaluations (Evals) for LLM-powered applications, including designing evaluation datasets, metrics, and scoring rubrics.
- Direct or strong conceptual understanding of the Google Agent Development Kit (ADK) and its components, including agent orchestration, tool use, and session/memory management.
- Experience designing and executing prompt injection attacks and defenses to test agent robustness.
- Hands-on experience with evaluation frameworks and libraries such as LangSmith, DeepEval, Ragas, Giskard, or Hugging Face evaluation tooling.
Technical Skills Automation & Tooling
- Strong proficiency in Python for script development, data processing, test automation, and evaluation pipeline development; PyTest is mandatory.
- Familiarity with Google Cloud Platform (GCP) services relevant to AI/ML (e.g., Vertex AI) and experience integrating testing into MLOps workflows.
- Experience with CI/CD pipelines and Git-based workflows for automated evaluation and regression testing.
Soft Skills
- Excellent analytical and problem-solving skills, with the ability to decompose complex agent behaviors into testable scenarios and measurable outcomes.
- Strong communication skills to articulate complex agent behaviors, risks, and safety findings clearly to both technical and non-technical audiences.
- Experience working in agile environments, collaborating closely with AI/ML Engineers, Researchers, product owners, and distributed stakeholders.
Preferred Qualifications
- Fundamental understanding of how LLMs work, including concepts like RAG, prompting strategies, fine-tuning, and the technical challenges related to alignment.
- Proficiency with data analysis and visualization tools (e.g., Pandas, SQL) for analyzing evaluation and safety results at scale.
- Familiarity with AI ethics principles and industry standards for responsible AI development.
- Experience testing complex APIs, especially for tool-integration and function-calling scenarios in agentic workflows.
- Experience with GIGO (Garbage-In/Garbage-Out) testing tools or frameworks that specialize in testing model responses to highly ambiguous, misleading, or malformed inputs.
- Awareness of data poisoning vectors and testing strategies for training and retrieval pipelines.
- Certifications in AI/ML, cloud platforms (GCP), or software testing methodologies.
How to Apply
Interested candidates are encouraged to respond/submit their updated resumes, and for additional job opportunities, please visit Jobs In India – VARITE.
Unlock Rewards
Refer Candidates and Earn.
If you're not available or interested in this opportunity, please pass this along to anyone in your network who might be a good fit and interested in our open positions. VARITE offers a Candidate Referral program, where you'll receive a one-time referral bonus based on the following scale if the preferred candidate completes a three-month assignment with VARITE.
Experience Level Bonus Referral
- 0-2 years – INR 5,000
- 2-6 years – INR 7,500
- 6+ years – INR 10,000
About VARITE
VARITE is a global staffing and IT consulting company providing technical consulting and team augmentation services to Fortune 500 Companies in USA, UK, CANADA and INDIA. VARITE is currently a primary and direct vendor to the leading corporations in the verticals of Networking, Cloud Infrastructure, Hardware and Software, Digital Marketing and Media Solutions, Clinical Diagnostics, Utilities, Gaming and Entertainment, and Financial Services.
Equal Opportunity Employer
VARITE is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate based on race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, marital status, veteran status, or disability status.
Skills
- Google ADK
- Vertex AI
- Gemini
- Jira
- LLM
- Generative AI
- LangSmith
- Hugging Face
- Python
- pytest
- GCP
- MLOps
- Git
- Retrieval-Augmented Generation
- Pandas
- SQL
More jobs at Varite India Private
All 193- Senior Test Automation EngineerVarite India Private · Bangalore, KAtoday
- Agentic AI EngineerVarite India Private · Bangalore, KAtoday
- Applied AI EngineerVarite India Private · Bangalore, KAtoday
- SOFTWARE ENGINEERVarite India Private · Bengaluru, KAtoday
- . Network Security EngineerVarite India Private · Bengaluru Millenia, KAtoday
Similar roles
- AI Assurance / AI Evaluation EngineerClifyX · MUMBAI5d ago
- Senior AI Agent Evaluation EngineerAditi Consulting · Bangalore, KAtoday
- Principal AI Engineer - Agentic Systems and Evaluation PlatformAmgen · India - Hyderabad6d ago
- Hardware Engineering Expert - AI Training & EvaluationJobgether · India · USD 60–120/hrtoday
- Machine Learning EngineerAmgen · India - Hyderabad16d ago
- Sr AI/ML Engineer, Applied AIThermo Fisher Scientific · Bangalore, Karnātaka, IndiaUpdated 2d ago