Latest AI JobsBrowse Jobs
Sarvam AI logo

Data Scientist - Evaluations, Chanakya

Sarvam AIBengaluruEngineering
Apply on company site
Salary
Not listed
Job type
Full-time
Posted
17 Apr 2026
Status
Open · checked 7d ago

About this role

Sarvam AI is seeking a Data Scientist for Evaluations in Bengaluru to anchor the evaluation function for their AI models and systems. This full-time engineering role suits professionals with several years of data science experience who want to build domain-specific evaluation frameworks for high-stakes AI applications in India.

Role Overview

This position requires designing, building, and maintaining evaluation systems that measure model quality within operational contexts. Rather than running standard benchmarks, the chosen professional will create custom evaluation harnesses for critical use cases where accuracy matters significantly. The individual will collaborate directly with product managers, deployment teams, and MLOps engineers to determine whether models meet deployment standards and maintain quality after release.

Key Responsibilities

Create evaluation frameworks for AI outputs across specific domains including document comprehension, geospatial reasoning, command summarisation, and enterprise workflow automation.
Partner with domain specialists and clients to define quality metrics and convert operational needs into measurable signals.
Execute structured evaluation cycles before and after deployment, establishing dashboards to monitor production model quality.
Detect failure cases, edge cases, and distribution shifts by actively searching for errors rather than confirming correct outputs.
Work alongside the MLOps Engineer to implement automated, versioned, and reproducible evaluation pipelines triggered by deployment events.
Create and oversee domain-specific datasets for benchmarking, fine-tuning, and evaluation, incorporating human annotation workflows when necessary.
Share internal quality reports and findings that influence engineering and product development roadmaps.

Requirements

Possess between three to six years of professional background in machine learning research, data science, or applied artificial intelligence, with at least two years focusing on production-level large language models.
Bring strong foundations in probability and statistics to ensure evaluation validity.
Demonstrate practical experience building custom evaluation frameworks from the ground up, including red-teaming methodologies, inter-rater reliability, and custom metrics.
Maintain strong proficiency in Python alongside familiarity with NumPy, pandas, HuggingFace datasets, LangSmith, RAGAS, EleutherAI Eval Harness, or similar tools.
Show prior application-level experience with model fine-tuning, prompt engineering, or reinforcement learning from human feedback.
Handle complex unstructured data types such as field reports, transcripts, PDFs, and doctrine documents.

Nice to Have

Previous professional exposure within high-stakes sectors like legal, defence, finance, or healthcare.
Background in adversarial evaluation or red-teaming.

Skills and Tools

Python
MLOps
Pandas
NumPy
HuggingFace datasets
RAGAS
LangSmith
EleutherAI Eval Harness
Prompt engineering
Model fine-tuning
RLHF
Statistics and probability

About the Company

Sarvam AI builds a comprehensive sovereign AI platform for India spanning infrastructure, research, models, and applications. Backed by prominent venture capital firms including Khosla Ventures, Peak XV, and Lightspeed, the enterprise collaborates with major public institutions and leading brands such as Tata Capital, SBI Life, CRED, IDFC, and LIC to make artificial intelligence functional for India.

How to Apply

Interested candidates should visit the official Sarvam AI careers page on sarvam.ai to submit their application.

Skills and tech stack

PythonMLOps

Questions candidates ask

Where is the Data Scientist - Evaluations role located?

The position is based in Bengaluru, India, on-site at Sarvam AI.

What is the primary focus of this data science position?

The role anchors the evaluations function by designing, building, and maintaining custom evaluation frameworks that measure AI model quality in operational contexts.

What experience level is required for applicants?

Applicants should have three to six years of experience in ML research, data science, or applied AI, with at least two years working specifically with production-level large language models.

Which tools and libraries are relevant for this position?

Key tools include Python, pandas, NumPy, HuggingFace datasets, RAGAS, LangSmith, and the EleutherAI Eval Harness.

How can I apply for this job at Sarvam AI?

Candidates can apply directly through the official Sarvam AI website at sarvam.ai.

Ready to apply?

The application is on the official Sarvam AI careers page.

Apply on company site