Latest AI JobsBrowse Jobs
Speak logo

Assessment Design Lead

SpeakUS - RemoteLearning & Curriculum
Apply on company site
Salary
$100K - $140K USD
Job type
Full-time
Work mode
Remote
Posted
10 Aug 2026
Status
Open · checked 7d ago

About this role

The Assessment Design Lead at Speak is a full-time remote position for a qualified professional with at least four years of experience in psychometric design. This role suits an expert in language assessment who is eager to define measurable learning standards and drive efficacy at a leading AI education company.

Role Overview

Speak is seeking an Assessment Design Lead to own how the platform measures learning efficacy and speaking proficiency. Operating within the dedicated Proficiency team and reporting to the Head of Learning Design and Curriculum, this role defines what the platform measures, why, and how across curriculum mastery, proficiency tests, and placement tests. The successful candidate ensures that unproctored, remote assessments remain fair, stable, and resistant to gaming while collaborating tightly with machine learning engineers and product managers.

Key Responsibilities

Own assessment design by establishing what Speak measures and how often across curriculum mastery assessments, proficiency tests, and placement tests, starting immediately with proficiency testing.
Define constructs and build the rubric and blueprint layer, translating objectives like fluency or pronunciation into scoreable rubrics and item specs for item writers and machine learning engineers.
Own validity and the quality bar by signing off on content validity, operationalizing mastery scores, auditing items and rubrics for bias across learner subgroups, and preventing unfair penalization of specific accents or native languages.
Design and execute the validity evidence plan, including concurrent and criterion studies benchmarking Speak assessments against external standards like CEFR and expert human ratings.
Partner closely with product managers and machine learning engineers on automated scoring, calibration, and feedback generation to ensure the scoring pipeline aligns with the defined constructs.

Requirements

4+ years of psychometric or assessment design experience creating rubrics, blueprints, and item specs for a shipped language assessment product.
Deep familiarity with language proficiency frameworks such as CEFR, ACTFL, IELTS, or TOEFL band descriptors.
Ability to identify differential item functioning to ensure fairness across various learner populations and native languages.
Strong capability to translate qualitative constructs into concrete, technical requirements for machine learning pipelines without losing nuance.
Quantitative rigor with comfort running or interpreting statistics such as inter-rater reliability, classical test theory, and basic item response theory concepts.
Direct AI fluency and judgment to utilize AI tools in personal workflows while evaluating whether generated outputs are precise enough to ship.
Comfort operating with high autonomy in a 0-to-1 environment, handling ambiguity, and managing tradeoffs effectively.

Nice to Have

Background in speech or pronunciation science to directly own pronunciation frameworks and error taxonomies.
Familiarity with adaptive testing or IRT-adjacent concepts.
Previous experience working within a large-scale language testing organization or high-rigor assessment environment.
Experience thriving in an EdTech startup environment or newly forming team.
Advanced degree such as a Master's or PhD in psychometrics, measurement, applied linguistics, second language acquisition, or a related quantitative field.
Track record of authoring technical or validity reports and publishing assessment research.

Skills and Tools

Psychometric and assessment design
Rubric, blueprint, and item specification creation
Language proficiency frameworks (CEFR, ACTFL, IELTS, TOEFL)
Quantitative statistics (inter-rater reliability, classical test theory, IRT concepts)
Bias auditing and fairness across learner subgroups
AI tools for workflow efficiency and drafting

Salary and Benefits

Annual salary range of $100,000 to $140,000 in USD.
Opportunity to join a fast-growing, well-funded AI company with a distributed team across global locations.

About the Company

Speak is on a mission to reinvent language learning by building a conversation-first, AI-powered tutor in your pocket. Founded in 2019 and launched first in South Korea, Speak is a leading AI language learning application serving users across multiple markets and languages. Backed by prominent venture investors including OpenAI, Accel, Founders Fund, and Khosla Ventures, the company operates with a distributed team across San Francisco, Seoul, Tokyo, Taipei, and Ljubljana.

How to Apply

To apply for this role, please visit the official company careers page to submit your application.

Questions candidates ask

What is the primary focus of the Assessment Design Lead?

The role owns how Speak measures learning efficacy and speaking proficiency, including curriculum mastery assessments, proficiency tests, and placement tests, with an immediate focus on the Proficiency Test.

Is this position remote?

Yes, this is a full-time remote position based in the US.

What is the salary range for this role?

The annual salary is between $100,000 and $140,000 USD.

What level of experience is required?

Candidates must have 4+ years of experience designing rubrics, blueprints, and item specs for a shipped language assessment product or equivalent psychometric measurement work.

Who will I report to?

You will report to the Head of Learning Design and Curriculum and work within the Proficiency team.

Ready to apply?

The application is on the official Speak careers page.

Apply on company site