JobHabor
Menu

Production Eval & Observability Engineer

Varite India Private · Noida, UP

Strong hiring record · median 12 days open

100100

Hiring signal

Actively hiring

Posted today · Never reposted · How we score

First seen
18 November 2026
Reposted
Not since September 2026
All reasons

In its favour

  • Posted today
  • Still listed on the employer's own board

Did you apply?

Production Eval & Observability Engineer at Varite India Private. We’ll keep it in your applied jobs, so you can see it later and be warned before applying to the same employer twice.

Company Name

VARITE India Private Limited

About The Client

An Indian multinational information technology (IT) consulting company headquartered in Noida, The company has offices in 52 countries and over 225,944 employees. The Client is a global IT services and consulting company that offers a wide range of services and products across various industries like IT Infrastructure Services, Cybersecurity Services, Cloud Services, Big Data and Analytics, Internet of Things (IoT) Solutions, Semiconductor Services and Enterprise Software Products.

About The Job

  • Exp: 6-9 Years
  • We are looking for a Production Eval & Observability Engineer to build and enhance production evaluation and monitoring capabilities for GenAI applications.
  • The role involves designing observability pipelines, implementing LLM evaluation frameworks, detecting model drift/regressions, and developing monitoring dashboards for production AI systems.

Essential Job Functions

  • Build a production sampling sidecar to consume OpenTelemetry (OTel) traces from AI agents.
  • Implement asynchronous LLM-judge scoring for production interactions.
  • Develop drift and regression detection mechanisms against predefined golden baselines.
  • Integrate user feedback and negative signals into evaluation datasets.
  • Build dashboards for usage tracking, pass-rate monitoring, drift alerts, and cost visibility.
  • Implement data-classification-aware logging and monitoring controls.

Qualifications

Required Skills

  • 6+ years of experience in software engineering with a strong focus on Observability and Monitoring.
  • Hands-on experience with OpenTelemetry (OTel) collectors and telemetry pipelines.
  • Expertise in monitoring tools such as Splunk, New Relic, Grafana, or CloudWatch.
  • Strong programming skills in Python.
  • Experience with AWS Lambda, ECS, and cloud-native architectures.
  • Understanding of asynchronous processing and sidecar architecture patterns.
  • Experience with logging, telemetry, and data-classification-aware monitoring.

Nice to Have

  • Experience with LLM Evaluation, LLM-as-a-Judge, and drift/regression detection frameworks.
  • Familiarity with Amazon Bedrock Guardrails or similar AI safety/content moderation solutions.
  • Exposure to regulated environments such as Financial Services or other compliance-driven industries.
  • Understanding of GenAI observability, model monitoring, and evaluation pipelines.

Technology Stack

  • OpenTelemetry (OTel)
  • Splunk / New Relic / Grafana / CloudWatch
  • Python
  • AWS Lambda
  • AWS ECS
  • Amazon Bedrock Guardrails
  • Observability & Monitoring Pipelines
  • LLM Evaluation Frameworks

How to Apply

Interested candidates are encouraged to respond/submit their updated resumes, and for additional job opportunities, please visit Jobs In India – VARITE.

Unlock Rewards

Refer Candidates and Earn.

If you're not available or interested in this opportunity, please pass this along to anyone in your network who might be a good fit and interested in our open positions. VARITE offers a Candidate Referral program, where you'll receive a one-time referral bonus based on the following scale if the referred candidate completes a three-month assignment with VARITE.

Experience Level Bonus Referral

0-2 years

INR 5,000

2-6 years

INR 7,500

6+ years

INR 10,000

About VARITE

VARITE is a global staffing and IT consulting company providing technical consulting and team augmentation services to Fortune 500 Companies in USA, UK, CANADA and INDIA. VARITE is currently a primary and direct vendor to the leading corporations in the verticals of Networking, Cloud Infrastructure, Hardware and Software, Digital Marketing and Media Solutions, Clinical Diagnostics, Utilities, Gaming and Entertainment, and Financial Services.

Equal Opportunity Employer

VARITE is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, marital status, veteran status, or disability status.

Skills

More jobs at Varite India Private

All 74

Similar roles