Principal Engineer, Distributed Systems
About this role
Guiding the architectural design and execution of security-critical infrastructure, this role involves shaping distributed systems that handle high-scale authorization, bot-defense, anomaly detection, security token services, and API authentication gateways across multiple regions. The incoming engineer establishes multi-region operation standards, formulates consistency models, and designs fault-tolerant dependencies that meet strict availability goals while maintaining low latency under partial failures and traffic spikes.
Operating as a senior individual contributor within Security Products, the person in this position partners closely with engineering, security, infrastructure, networking, platform, product, and customer-facing teams. Daily work includes leading architecture reviews, defining technical standards, guiding capacity planning, and mentoring senior and staff engineers to elevate engineering practices across the organization. Success in the role requires deep technical judgment, the capacity to align diverse stakeholders, and the ability to translate complex distributed-systems challenges into clear, actionable execution plans.
What you will do
- Establish architectural approaches for multi-region distributed systems including service placement, traffic management, disaster recovery, and regional independence.
- Drive decisions around consistency models, caching, replication, concurrency, and ordering where correctness and security are paramount.
- Design systems for tenant isolation, data residency, trust boundaries, and blast-radius reduction while handling sensitive security data.
- Define fault-tolerance strategies and graceful degradation mechanisms for dependencies, networks, storage systems, and control-plane components.
- Improve reliability and operability through SLOs, error budgets, metrics, tracing, audit events, alerting, and post-incident learning.
- Guide teams through architecture reviews, design evaluations, implementation tradeoffs, load testing, and production readiness assessments.
- Mentor senior and staff engineers while communicating risks, tradeoffs, and recommendations clearly to technical and executive audiences.
What you need
- Twelve or more years of experience designing and building production software with substantial background in distributed systems and platform infrastructure.
- A proven track record of serving as a principal, staff-plus, distinguished, or equivalent technical leader across multiple teams or a broad technical domain.
- Deep expertise in distributed-systems design covering availability, latency, scalability, consistency, concurrency, partition tolerance, and failure recovery.
- Hands-on experience designing and operating security-sensitive or reliability-critical services in live production environments.
- Strong software engineering experience utilizing one or more systems languages or backend ecosystems such as Go, Java, Rust, C++, or Python.
- Demonstrated ability to transition from ambiguous requirements to clear architecture, sequenced execution plans, and durable engineering outcomes.
- Strong communication skills with the capability to influence without direct authority and build alignment across organizational boundaries.
Nice to have
- Experience designing and implementing authorization systems at high scale including policy engines, permission models, and enforcement points.
- Experience operating a Security Token Service, identity platform, credential service, or comparable security-critical control plane.
- Experience operating systems with four-nines availability requirements, high request volume, and strict latency objectives.
- Experience with multi-region architecture, regional failover, active-active or active-passive operation, and replication.
- Experience with authentication and identity standards such as OAuth 2.0, OpenID Connect, SAML, JWT, mTLS, or SPIFFE/SPIRE.
Skills and tech stack
Questions candidates ask
What is the salary range for this role?
The salary range for this position is $227,000 to $303,000.
Where is the position located?
The role is located in New York, NY or Sunnyvale, CA.
What type of employment is this?
This is a full-time, senior individual-contributor role.