About this role
Engineers who thrive on building rigorous evaluation systems for high-stakes AI applications will find a great fit here. This position exists to own the reliability and quality bar for an AI copilot embedded within a modern trading platform used by sophisticated investors. The team builds cloud-based trading infrastructure that provides high-performance access to liquidity venues, advanced charting, and institutional workflows.
Working in this capacity, the person will design and build evaluation frameworks that measure correctness, safety, latency, and regression risk across market analysis and trading workflows. Responsibilities include developing curated golden sets, stress cases, and automated quality gates that block releases when key metrics degrade.
What you will do
- Design and build evaluation systems that measure correctness, safety, latency, and regression risk across market analysis, portfolio reasoning, and trading workflows.
- Develop and maintain benchmarks including curated golden sets, scenario suites, adversarial cases, and refreshed market test corpora.
- Build automated quality gates and regression workflows that block releases when key metrics degrade.
- Partner with engineering and product to define safe tool and action contracts, ensuring predictable assistant behavior.
- Own model improvement loops tied to evaluations, including data collection strategies, error taxonomy, and prompt changes.
- Design and operate monitoring and incident response for AI systems, covering telemetry, alerting, and root cause analysis.
- Develop a deep understanding of trading concepts such as margin, shorting, and execution to express them accurately to users.
What you need
- At least eight years of experience shipping production software with strong proficiency in any programming language.
- Strong knowledge of computer science fundamentals, testing methodology, and systems design.
- Experience building evaluation frameworks, test harnesses, and benchmark suites for complex systems like LLMs or search.
- Experience running model improvement cycles including dataset curation, labeling, offline experimentation, and deploying changes.
- Ability to define metrics, build measurement pipelines, and drive engineering and product decisions from data.
- Comfort working across the stack by debugging model failures and partnering with frontend and product teams.
- High degree of self-motivation and willingness to jump into unfamiliar areas to solve problems.
Nice to have
- Experience with fine-tuning, preference optimization, distillation, or prompt techniques for improving tool-use reliability
- Experience creating domain-specific benchmarks and adversarial suites for high-stakes applications
- Deep experience with trading across asset classes and margin types
- Experience with Rust and performance-sensitive services
- Experience designing incident response and SLOs for machine learning and AI systems
Benefits
- Competitive compensation packages
- Company equity
- 401k matching
- Gender neutral parental leave
- Full medical, dental and vision insurance
Skills and tech stack
Questions candidates ask
What is the remote work policy for this position?
The job location is listed as Remote and includes the tag for remote work compatibility.
What is the salary range for this role?
The base salary range is $200,000 to $500,000, which represents starting base salaries based on job-related factors such as experience, skills, and location.
What tools and technologies are used on the team?
The tech stack includes Rust, TypeScript, Postgres, React for web, React Native for mobile, observability tooling, and LLM APIs with model serving.
What does the team actually build?
The team builds a new cloud-based trading platform providing high-performance traders access to liquidity venues across multiple asset classes, charting capabilities, and sophisticated order handling.