Senior Software Engineer, AI Agent & Experience Team
About this role
Developers working on this team spend their days designing, building, and shipping features across AI platform infrastructure such as the LLM proxy, AI key management, and observability pipelines. Engineers also develop and extend autonomous agents participating in the software development lifecycle, contribute to the internal plugin marketplace, and implement Model Control Protocol services. Toast operates an internal platform team for AI that builds infrastructure, governance, and developer tooling to make artificial intelligence a reliable force multiplier throughout product development.
Candidates joining this group will directly impact developer velocity by crafting systems that internal engineering teams depend on daily. Success in this position requires a strong background in scalable backend services, hands-on experience with LLM-powered agents, and deep customer empathy for developer experiences. Individuals who enjoy tackling complex backend challenges, shaping modern software development lifecycles, and building robust internal platforms will find a strong alignment with this group.
What you will do
- Design, build, and ship features across AI platform infrastructure including the LLM proxy, AI key management, and observability pipelines.
- Develop and extend autonomous agents that participate in the software development lifecycle, such as the AI Review GitHub App and Developer Platform MCP integrations.
- Build on and contribute to the internal plugin marketplace by creating plugins that encode architectural standards and quality practices into AI assistant behavior.
- Implement Model Control Protocol services and service templates to simplify the creation of AI-powered workflows for teams across the company.
- Write high-quality, well-tested code while actively participating in design and code reviews.
- Collaborate closely with engineers, product managers, and platform teams to translate developer needs into reliable platform solutions.
- Contribute to the adoption of agentic development practices and tooling within the engineering organization.
What you need
- Hold a Bachelor’s Degree with relevant experience of 5+ years in designing and implementing scalable backend services.
- Demonstrate hands-on experience building, deploying, or operating LLM-powered agents or AI-assisted developer tooling.
- Bring familiarity with AI coding assistants and an interest in extending them through custom plugins, skills, or hooks.
- Possess experience with Model Control Protocol, tool use patterns, or similar agentic integration frameworks.
- Show working knowledge of prompt engineering and how LLM behavior responds to context and instructions.
- Maintain a strong foundation in Java, Kotlin, or another object-oriented programming language.
- Have professional experience with distributed systems, REST APIs, and cloud-native infrastructure alongside strong customer empathy.
Nice to have
- Familiarity with A2A protocols and multi-agent orchestration patterns
- Experience with machine learning concepts, model evaluation, or ML infrastructure
- Experience building developer experience platforms, internal tooling, or API gateways
- Familiarity with observability tooling for AI and LLM systems such as Langfuse or DataDog
Skills and tech stack
Questions candidates ask
What is the remote or hybrid work policy?
This is a hybrid role requiring in-office presence two days per week. Toast embraces a hybrid work model that fosters in-person collaboration while valuing individual needs.
What tools and technologies will I work with?
You will work with Java, Kotlin, distributed systems, REST APIs, and cloud-native infrastructure. The stack also involves AI tooling such as the LLM proxy, observability pipelines, Model Control Protocol integrations, and AI coding assistants.
What does the team actually build?
The AI Agent and Experience team builds the infrastructure, governance, and developer tooling that makes AI a reliable force multiplier across the product development lifecycle. Products include the LLM proxy, AI key management, observability pipelines, the Claude Plugin Marketplace, and autonomous agents.
How does Toast use AI in the hiring process?
Toast uses AI tools to support recruiters and interviewers with tasks like note-taking, summarization, and documentation of interviews to ensure full focus on the conversation. All hiring decisions are made by people, and candidates are asked to complete interviews and assessments without real-time AI tools unless instructed otherwise.