
Production Eval & Observability Engineer
Varite India Private · Noida, UP
Strong hiring record · median 12 days open
- First seen
- 18 November 2026
- Reposted
- Not since September 2026
All reasons
In its favour
- Posted today
- Still listed on the employer's own board
Company Name
VARITE India Private Limited
About The Client
An Indian multinational information technology (IT) consulting company headquartered in Noida, The company has offices in 52 countries and over 225,944 employees. The Client is a global IT services and consulting company that offers a wide range of services and products across various industries like IT Infrastructure Services, Cybersecurity Services, Cloud Services, Big Data and Analytics, Internet of Things (IoT) Solutions, Semiconductor Services and Enterprise Software Products.
About The Job
- Exp: 6-9 Years
- We are looking for a Production Eval & Observability Engineer to build and enhance production evaluation and monitoring capabilities for GenAI applications.
- The role involves designing observability pipelines, implementing LLM evaluation frameworks, detecting model drift/regressions, and developing monitoring dashboards for production AI systems.
Essential Job Functions
- Build a production sampling sidecar to consume OpenTelemetry (OTel) traces from AI agents.
- Implement asynchronous LLM-judge scoring for production interactions.
- Develop drift and regression detection mechanisms against predefined golden baselines.
- Integrate user feedback and negative signals into evaluation datasets.
- Build dashboards for usage tracking, pass-rate monitoring, drift alerts, and cost visibility.
- Implement data-classification-aware logging and monitoring controls.
Qualifications
Required Skills
- 6+ years of experience in software engineering with a strong focus on Observability and Monitoring.
- Hands-on experience with OpenTelemetry (OTel) collectors and telemetry pipelines.
- Expertise in monitoring tools such as Splunk, New Relic, Grafana, or CloudWatch.
- Strong programming skills in Python.
- Experience with AWS Lambda, ECS, and cloud-native architectures.
- Understanding of asynchronous processing and sidecar architecture patterns.
- Experience with logging, telemetry, and data-classification-aware monitoring.
Nice to Have
- Experience with LLM Evaluation, LLM-as-a-Judge, and drift/regression detection frameworks.
- Familiarity with Amazon Bedrock Guardrails or similar AI safety/content moderation solutions.
- Exposure to regulated environments such as Financial Services or other compliance-driven industries.
- Understanding of GenAI observability, model monitoring, and evaluation pipelines.
Technology Stack
- OpenTelemetry (OTel)
- Splunk / New Relic / Grafana / CloudWatch
- Python
- AWS Lambda
- AWS ECS
- Amazon Bedrock Guardrails
- Observability & Monitoring Pipelines
- LLM Evaluation Frameworks
How to Apply
Interested candidates are encouraged to respond/submit their updated resumes, and for additional job opportunities, please visit Jobs In India – VARITE.
Unlock Rewards
Refer Candidates and Earn.
If you're not available or interested in this opportunity, please pass this along to anyone in your network who might be a good fit and interested in our open positions. VARITE offers a Candidate Referral program, where you'll receive a one-time referral bonus based on the following scale if the referred candidate completes a three-month assignment with VARITE.
Experience Level Bonus Referral
0-2 years
INR 5,000
2-6 years
INR 7,500
6+ years
INR 10,000
About VARITE
VARITE is a global staffing and IT consulting company providing technical consulting and team augmentation services to Fortune 500 Companies in USA, UK, CANADA and INDIA. VARITE is currently a primary and direct vendor to the leading corporations in the verticals of Networking, Cloud Infrastructure, Hardware and Software, Digital Marketing and Media Solutions, Clinical Diagnostics, Utilities, Gaming and Entertainment, and Financial Services.
Equal Opportunity Employer
VARITE is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. We do not discriminate on the basis of race, color, religion, sex, sexual orientation, gender identity or expression, national origin, age, marital status, veteran status, or disability status.
Skills
- Generative AI
- LLM
- OpenTelemetry
- Splunk
- New Relic
- Grafana
- AWS CloudWatch
- Python
- AWS Lambda
- ECS
- Bedrock
More jobs at Varite India Private
All 74- Full Stack Technical LeadVarite India Private · PAN India, KAtoday
- Applications PackagingVarite India Private · Noida, UPtoday
- Java Backend DeveloperVarite India Private · Hyderabad, TStoday
- Informatica CDGC & Custom API DevelopmentVarite India Private · Remotetoday
- Network Engineer Senior ManagerVarite India Private · Bangalore, KAtoday
Similar roles
- Observability EngineerPoint72 · Bengaluru, India8d ago
- Observability EngineerNasdaq · India - Bangalore - Karnataka20d ago
- Senior Observability EngineerMX Build Technologies India Private · Chennai, Tamil Nadu, Indiatoday
- Sr. Software Engineer - ObservabilityDatabricks · Bengaluru, Indiatoday
- GNS Observability Senior EngineerTjx · Hyderabad, TS 5000816d ago
- Senior Observability EngineerEnsono · Bengaluru, Indiatoday