Latest AI JobsBrowse Jobs
Flodesk logo

Senior Applied AI Engineer, Agent Quality & Evaluations

FlodeskRemoteProduct
Apply on company site
Salary
$150K - $250K
Job type
Full-time
Work mode
Remote
Posted
23 Sept 2026
Status
Open · checked 7d ago

About this role

Flodesk is hiring a remote Senior Applied AI Engineer for Agent Quality & Evaluations to own prompt management, agent behavior, and measurement systems. This role suits an experienced engineer with a strong background in production language models and testing frameworks who wants to shape AI-driven marketing tools for entrepreneurs.

Role Overview

Flodesk is scaling its artificial intelligence capabilities to help small business owners build complete marketing campaigns automatically. The Senior Applied AI Engineer, Agent Quality & Evaluations will lead the quality, testing, and continuous improvement loop for member-facing AI features. Reporting directly to the Head of Product for AI Systems and Core Experience, the professional in this position will handle prompt engineering, agent loops, quality standards, and evaluation systems across various product surfaces like email creation, workflows, segmentation, and recommendations.

Key Responsibilities

Oversee member-facing AI behavior spanning prompt-to-create functions, agentic editing, and upcoming features like analytics, scheduling, and recommendations.
Author, test, version, and document prompts alongside agent instructions with detailed change logs.
Architect how the application utilizes context, user data, tools, and product state while building necessary prototypes and agent loops.
Establish and operate the organization's AI evaluation frameworks, covering benchmark datasets, scoring rubrics, regression suites, and human review processes.
Construct evaluation harnesses, monitoring tools, and performance dashboards for scalable quality measurement.
Identify and resolve failures across prompts, underlying models, context, and orchestration layers.
Establish quality baselines and release gates for prompt, model, and agent updates.

Requirements

Demonstrated history of developing or improving production-grade LLM or agentic software products.
Proficiency in programming languages such as Python or TypeScript for building internal tools and evaluation harnesses.
Clear understanding of the interactions between prompts, context, orchestration, tools, and product state.
Direct background in LLM evaluation, observability design, and combining automated tests with human judgment.
Strong product intuition capable of turning qualitative user feedback into measurable system improvements.
Excellent communication skills across technical and non-technical teams regarding quality, cost, and latency tradeoffs.

Nice to Have

Experience building content generation, creative tools, or marketing software.
Background creating multi-turn agents that leverage tools and maintain user intent over multiple edits.
Familiarity with observability platforms such as Langfuse or Braintrust.
Previous work on products tailored for creators, marketers, or small businesses.

Skills and Tools

Python
TypeScript
LLMs and agent orchestration
Evaluation and observability platforms (Langfuse, Braintrust)
Prompt engineering and versioning
System thinking and prototyping

Salary and Benefits

Base salary range of $150,000 to $250,000, determined by location and experience.
Tier 1 city resident range: $165,000 - $250,000.
Other US-based locations range: $150,000 - $235,000.
Fully paid individual health insurance coverage.
22 weeks of paid maternity leave for birthing parents and 16 weeks of paid parental leave for non-birthing parents.
Unlimited flexible time off.
401(k) matching program for US-based employees.
Annual learning and development stipend of $1,000.

About the Company

Recognized globally by the Inc 5000 as a fast-growing email marketing company, Flodesk empowers entrepreneurs to build online businesses and design appealing emails. The remote-first organization maintains a globally distributed team with physical hubs in Menlo Park, Barcelona, and Da Nang, fostering a diverse and creative culture committed to leveling the playing field for small business owners through intuitive design.

How to Apply

Interested candidates should visit the official company careers page to submit an application and review further details.

Skills and tech stack

PythonLLMTypeScript

Questions candidates ask

Is this position remote?

Yes, this is a remote-first position at Flodesk.

What is the salary range for this role?

The base salary ranges from $150,000 to $250,000, varying based on candidate experience and geographic location.

Who does this role report to?

This position reports directly to the Head of Product, AI Systems and Core Experience.

What programming languages are required?

Strong software engineering skills in Python, TypeScript, or a similar language are required.

What benefits are provided for US employees?

US employees receive fully paid individual health insurance, parental leave, unlimited flexible time off, a 401(k) match, and an annual learning and development stipend.

Ready to apply?

The application is on the official Flodesk careers page.

Apply on company site