Latest AI JobsBrowse Jobs
F

Research Engineer - Benchmarks

firecrawlSan Francisco HQ; Toronto HubEngineering Team
Apply on company site
Salary
$250K – $290K
Job type
Full-time
Work mode
Remote
Experience
Senior · 4+ yrs
Posted
29 Sept 2026
Status
Open · checked 7d ago

About this role

This role suits an independent research engineer with strong Python skills and experience shipping public evaluations who wants to measure the market for AI data providers at Firecrawl in San Francisco or Toronto. The position involves designing end-to-end automated benchmarks and public leaderboards that feed directly into the Alexandria repository system.

Role Overview

Firecrawl seeks a Research Engineer - Benchmarks to build independent, rigorous, and automated public evaluations for data providers across company records, contact finding, and job listings. The position operates out of the San Francisco HQ or Toronto Hub on a hybrid or on-site schedule. The person in this role will own the evaluation methodology from dataset creation and scoring to weekly public releases and integration with Alexandria's provider selection loop.

Key Responsibilities

Design and execute head-to-head benchmarks comparing data providers against verified ground truth for company details, person discovery, and job listings.
Build and maintain test datasets, scoring frameworks, and automated harnesses while preventing data leakage or staleness.
Own the weekly benchmark release process, ensuring every run is versioned, reproducible, and defensible against vendor scrutiny.
Measure metrics that buyers care about, including accuracy, freshness, coverage, speed, and cost.
Collaborate with marketing to publish public leaderboard pages.
Integrate benchmark outputs directly into Alexandria to optimize provider selection routines.
Expand the categories of evaluated tools inside and outside Alexandria.

Requirements

Minimum of 4 years of experience in machine learning, research engineering, or data engineering, specifically involving shipped evaluations or benchmarks.
Proven background from a frontier lab, data organization, third-party benchmark entity, or public benchmark initiative.
Strong proficiency in Python and API integrations, with the ability to handle messy vendor APIs, inconsistent schemas, and rate limits.
Demonstrated expertise in building test sets and scoring mechanisms covering sampling, labeling, inter-rater agreement, and appropriate use of LLM judges.
Heavy utilization of AI tools with a self-improving workflow and the ability to ship independently without direct instructions.
Clear written communication skills tailored for engineers, marketers, and external vendors.
Authorization to work in the United States or Canada.

Skills and Tools

Python
API Integrations
LLM Judges
Automated Benchmark Harnesses
Dataset Sampling and Labeling Frameworks

Salary and Benefits

Annual base salary ranges from $250,000 to $290,000 USD for San Francisco or $210,000 to $224,000 CAD for Toronto.
Competitive equity package.
Mandatory 15 days of PTO plus flexible additional time off, 12 weeks of fully paid parental leave, and a 3-month paid sabbatical after 4 years.
Monthly wellness stipend of $100 USD and up to $1,000 USD annually for professional learning and development.
Comprehensive health, dental, and vision insurance options tailored to US or Canadian employment locations, alongside life and disability insurance, mental health support via Talkspace, fertility and family-building coverage through Carrot, and retirement plans (401(k) or Group RRSP).
Local perks including team offsites, snacks, drinks, team lunches, and e-bike transportation loans at the San Francisco headquarters.

About the Company

Firecrawl provides an API service that converts web URLs into clean, LLM-ready markdown and structured data for AI agents. Backed by a Series B led by Smash Capital, the company develops Alexandria, a large repository of knowledge designed for AI infrastructure. The organization operates with a small, high-ownership team scaling rapidly past initial milestones and GitHub repository rankings.

How to Apply

Interested candidates should apply on the official company careers page.

Skills and tech stack

PythonLLMLLMs

Questions candidates ask

What are the location and work style requirements for this role?

The role is based out of the San Francisco HQ (on-site 5 days a week) or the Toronto Hub (hybrid, on-site 3+ days a week).

What is the salary range for the Research Engineer position?

The salary ranges from $250,000 to $290,000 USD per year in San Francisco, and from $210,000 to $224,000 CAD per year in Toronto, supplemented by competitive equity.

Does Firecrawl offer visa sponsorship for this job?

US visa sponsorship is not available. For the Toronto Hub, visa sponsorship will be considered on a case-by-case basis.

How many years of experience are required?

Candidates must have 4+ years of experience in machine learning, research engineering, or data engineering, with demonstrated evaluation or benchmark work.

What technologies and languages are required?

The role requires strong proficiency in Python and API integrations, along with experience building scoring methods and test sets.

Ready to apply?

The application is on the official firecrawl careers page.

Apply on company site