Senior Staff Machine Learning Engineer
About this role
The AI Platform and Data Science team at Zscaler builds high-fidelity risk identification systems to catch all security threats in customer data while minimizing noise. Engineers on this initiative develop automated security analysis components by combining threat research, data analysis, machine learning, and data engineering.
Reporting to the Manager of AI Platform and Data Science, the Senior Staff Machine Learning Engineer translates complex risk identification methods into agent logic while collaborating with threat research and data engineering teams. Responsibilities include building production-grade LLM-powered agents, handling model orchestration and evaluation, and implementing data-driven quality frameworks like backtesting and precision tuning.
Successful candidates thrive in ambiguous environments, operate with ownership across high-level strategy and hands-on execution, and utilize concrete data analytics to guide engineering decisions. This remote position offers an opportunity to shape cloud security platforms at scale within an enterprise transitioning into an AI-native organization.
What you will do
- Translate risk identification methods into agent logic while understanding the benefits and limitations of agents across risk analysis, explanations, and recommendations.
- Collaborate closely with threat-research teams to understand data, threats, and security heuristics.
- Identify and solve data requirements for analysis by collaborating with data engineering teams on pipelines, enrichments, and aggregations.
- Follow a data-driven quality approach to threat detection involving backtesting, precision versus recall balancing, tuning, and quality control.
- Deploy and monitor solutions in production within the company CI/CD framework.
What you need
- Experience building LLM-powered agents in production, including tool and function calling, prompt engineering, multi-step orchestration frameworks like LangGraph or LangChain, and evaluation against ground truth.
- Eight or more years of professional Python development experience with the ability to design production-quality systems with validated inputs, data contracts, and comprehensive tests.
- Hands-on expertise with SQL and Python data analytics libraries such as pandas, Polars, and NumPy.
- Demonstrated ownership of production reliability, including CI/CD, containerization, logging, metrics, observability, alerting, on-call participation, and cloud infrastructure in AWS.
- Senior Staff level ownership and leadership of emergent requirements, architecture, and complex engineering projects.
Nice to have
- Experience with production AI or ML systems managing competing constraints of cost, latency, accuracy, and model routing.
- Background in cyber threat research, threat modeling, threat hunting, detection engineering, or risk analysis.
- Data engineering experience building pipelines over large-volume event data using SQL, OpenSearch, Elasticsearch, Athena, or Presto.
Benefits
- Various health plans
- Time off plans for vacation and sick time
- Parental leave options
- Retirement options
- Education reimbursement
- In-office perks
Skills and tech stack
Questions candidates ask
What is the remote work policy for this position?
This is a remote role based in the USA, or can be based in Santa Clara, California.
What is the salary range for the Senior Staff Machine Learning Engineer role?
The base salary range for this full-time position is $157,500 to $225,000 USD, which excludes commission, bonus, equity, and benefits.
What technologies and frameworks are required for this role?
Candidates must have experience with Python, SQL, data analytics libraries like pandas and NumPy, and LLM orchestration frameworks such as LangGraph or LangChain.
Who does this position report to?
The role reports to the Manager AI Platform and Data Science in the AI Platform and Data Science department.