Latest AI JobsBrowse Jobs
Scale AI logo

Machine Learning Research Scientist, Evaluations

Scale AISan Francisco, CA; Seattle, WA; New York, NYResearch
Apply on company site
Salary
Not listed
Job type
Full-time
Posted
26 Aug 2026
Status
Open · checked 9d ago

About this role

Scale AI is seeking a Machine Learning Research Scientist for the evaluations pod within the GenAI Research Organization based in San Francisco, Seattle, or New York. This position suits researchers and engineers focused on advancing generative AI through rigorous evaluation and failure mode analysis of large language models.

Role Overview

The Machine Learning Research Scientist, Evaluations position sits in the GenAI Research Organization at Scale AI. The professional will join the evaluation pod to build benchmarks and diagnose model failure problems across both text and multimodal modalities. Responsibilities include working alongside top foundation model labs to turn failure analysis into strategic and technical input for future generative AI models, as well as defining best practices for evaluation-driven artificial intelligence development.

Key Responsibilities

Analyze and characterize model behavior to diagnose failure modes in frontier large language models and agents, covering root cause analysis of reasoning errors, capability gaps, alignment issues, and robustness.
Design and create evaluation methods and benchmarks to measure large language model capabilities in text and multimodal modalities.
Apply post-training knowledge, including supervised fine-tuning, reinforcement learning from human feedback, and reward modeling, to connect observed model failures with specific training interventions and data.
Publish investigation findings in top-tier artificial intelligence conferences.

Nice to Have

A Master's or Ph.D. degree in Artificial Intelligence, Computer Science, Machine Learning, or a related discipline.
Strong background in deep learning, large-scale model fine-tuning, and reinforcement learning.
Prior practical knowledge of evaluation development or benchmark creation, alongside post-training procedures like instruction tuning, preference modeling, or reinforcement learning from human feedback.
Demonstrated publication record in major journals or artificial intelligence conferences such as ICML, NeurIPS, EMNLP, ICLR, CVPR, and ACL.
Background working in a customer-facing capacity.

Skills and Tools

Large Language Models (LLM)
Supervised Fine-Tuning (SFT)
Reinforcement Learning from Human Feedback (RLHF)
Reward Modeling

Salary and Benefits

The base salary range for this full-time role in San Francisco, New York, and Seattle is $165,600 to $207,000 USD, with the final offer determined by interview performance, location, skills, experience, and relevant training. Eligible team members receive board-approved equity compensation and benefits including comprehensive vision, medical, and dental coverage, generous paid time off, retirement plans, a learning and development stipend, and potentially a commuter stipend.

About the Company

Scale AI builds reliable artificial intelligence systems for critical decisions, offering high-quality data and full-stack technology that power leading models. The organization helps enterprises and governments deploy, build, and oversee impactful AI applications, partnering with prominent industry leaders and agencies such as Meta, Mayo Clinic, Ernst & Young, Time Inc., the Government of Qatar, and U.S. government entities including the Air Force and Army. Scale AI operates as an inclusive, equal opportunity workplace.

How to Apply

To submit an application for this position, please visit the official Scale AI careers page.

Skills and tech stack

LLM

Questions candidates ask

What locations are eligible for this position?

This role is located in San Francisco, CA, Seattle, WA, and New York, NY.

What is the salary range for this job?

The base salary range for this position in San Francisco, New York, and Seattle is $165,600 to $207,000 USD.

What benefits are provided by the company?

Eligible employees receive comprehensive health, dental, and vision coverage, retirement benefits, a learning and development stipend, generous paid time off, equity-based compensation, and potentially a commuter stipend.

What department does this role belong to?

This role belongs to the Research department, specifically within the evaluation pod of the GenAI Research Organization.

What are the primary responsibilities of the role?

Responsibilities include analyzing and diagnosing failure modes in frontier LLMs and agents, designing text and multimodal benchmarks, applying post-training expertise to connect failures to training data, and publishing research findings in top-tier conferences.

Ready to apply?

The application is on the official Scale AI careers page.

Apply on company site