Research Engineer, Web Crawling

Thinking MachinesSan FranciscoResearch Infrastructure (ML Infrastructure and Training Stack)

Apply on company site

Opens the official Thinking Machines careers page

Salary

$350K-$475K USD

Job type

Full-time

Experience

Senior · 8+ yrs

Posted

24 Sept 2026

Status

Open · checked 7d ago

About this role

Thinking Machines is seeking a Research Engineer for Web Crawling in San Francisco to build and manage internet-scale data collection systems. This full-time position is tailored for experienced engineers with a strong background in distributed web scraping and data pipeline development.

Role Overview

Thinking Machines is hiring a Software Engineer to design, operate, and maintain web-crawling systems operating at internet scale. The responsibilities encompass distributed collection, filtering, deduplication, and determining data retention strategies for frontier model training.

Key Responsibilities

Design and scale the web crawler and ingestion infrastructure utilized for sourcing pretraining data for Inkling

Build data pipelines capable of large-scale extraction, deduplication, and quality filtering

Develop specialized crawlers targeting high-value or difficult-to-reach data repositories

Collaborate with the pretraining team to analyze how adjustments in crawled data influence model performance

Enhance reliability and efficiency across petabyte-scale crawling and ingestion infrastructure

Establish technical direction for the crawling domain and onboard incoming engineers

Requirements

Minimum of 8 years of professional experience designing, building, and scaling web crawlers, scrapers, or large distributed data-acquisition architectures

Demonstrated history of owning internet-scale crawler or data-acquisition infrastructure

Advanced software engineering proficiency in programming languages such as Python, Go, or Rust alongside practical distributed systems experience

Familiarity with practical and legal aspects of large-scale web data gathering, including rate limiting, licensing, and robots.txt protocols

Nice to Have

Background applying machine learning to crawl selection, extraction, or data quality classification at internet scale

Experience establishing technical direction for crawling, search infrastructure, or data acquisition teams

Experience designing architectures intended for petabyte-scale processing and storage

History of contributing to open-source data infrastructure, scraping, or crawling utilities

Previous employment at a search engine dealing with crawling and indexing or within a frontier AI laboratory data acquisition unit

Skills and Tools

Python

Rust

Distributed Systems

Web Crawling and Scraping Frameworks

Data Pipelines

Salary and Benefits

Expected annual salary range spans from $350,000 to $475,000 USD, determined by background, skills, and experience

Comprehensive health, dental, and vision insurance coverage

Unlimited paid time off (PTO)

Paid parental leave

Relocation assistance provided as needed

About the Company

Thinking Machines builds artificial intelligence designed to extend human will and judgment. The organization trains frontier models via Inkling, builds Tinker to empower custom model creation, and designs interfaces to improve human-AI communication.

How to Apply

Interested candidates should submit their application directly on the official Thinking Machines careers page.

Skills and tech stack

Python Go Rust Machine Learning Distributed Systems

Questions candidates ask

What is the location of the Research Engineer, Web Crawling role?

The position is based in San Francisco, California.

What is the salary range for this position?

The expected annual salary ranges from $350,000 to $475,000 USD, depending on background, skills, and experience.

How many years of experience are required?

Candidates must have 8 or more years of experience designing, building, and scaling web crawlers, scrapers, or large-scale distributed data-acquisition systems.

Does Thinking Machines offer visa sponsorship?

Yes, visa sponsorship is provided for qualified candidates.

What benefits are included with employment?

Benefits include health, dental, and vision coverage, unlimited PTO, paid parental leave, and relocation support.

Ready to apply?

The application is on the official Thinking Machines careers page.

Apply on company site