Role Overview
The Staff Network Production Engineer is a full-time position based in San Francisco, CA, focused on maintaining production reliability across Crusoe's global network infrastructure. This role suits a seasoned network engineer with extensive operational experience in large-scale environments who is ready to handle high-pressure situations and maintain system uptime.
Key Responsibilities
Take ownership of uptime across Crusoe's global edge, backbone, data center, and GPU cluster network to support AI workloads at scale.
Build and maintain Python-based tooling to minimize toil, automate remediation workflows, and accelerate mean time to resolution.
Develop scripts and tooling to expedite detection, triage, and mitigation during high-severity network events, alongside active participation in incident response and postmortem documentation.
Create tooling to surface systemic issues during root cause analyses and drive remediation plans to closure.
Write automation on top of the monitoring stack using streaming telemetry, SNMP, NetFlow, Kentik, Grafana, Prometheus, and ThousandEyes.
Author and maintain runbooks, escalation playbooks, and standard operating procedures with a focus on scripting and automation.
Partner with Architecture and SRE teams to define and track network reliability metrics, building automated dashboards and alerting.
Provide technical guidance to senior engineers regarding automation practices and reduce operational toil while fostering operational excellence.
Requirements
Bring 8 or more years of production network engineering experience focusing on operations, incident response, and reliability within large-scale or internet-scale environments.
Demonstrate strong Python and scripting proficiency to write diagnostic tooling, auto-remediation scripts, and automation pipelines from scratch.
Possess hands-on experience with observability and monitoring tools such as streaming telemetry, SNMP, NetFlow/sFlow, Grafana, Prometheus, and ThousandEyes, including API scripting capabilities.
Bring hands-on experience operating RDMA/RoCE lossless fabrics for GPU or HPC workloads, including familiarity with PFC, ECN, and DCQCN tuning.
Exhibit expert hands-on knowledge of BGP, EVPN-VXLAN, IS-IS, OSPF, MPLS, QoS, and TCP/IP in production data center settings.
Maintain proficiency with Arista (EOS) and Juniper (Junos) platforms across leaf-spine CLOS architectures in multi-vendor environments.
Show comfort operating large device fleets across multi-region environments with on-call responsibility as a critical escalation point.
Hold a Bachelor's degree in Computer Science, Electrical Engineering, or a related field, or demonstrate equivalent practical experience.
Nice to Have
Experience utilizing NVIDIA/Mellanox networking platforms in GPU cluster environments.
Familiarity with Kentik or Arbor for traffic analysis and DDoS visibility.
Experience defining or contributing to SLIs and SLOs alongside SRE or product teams.
Exposure to operating device fleets of 10,000 or more units across hyperscale or cloud environments.
Background contributing to post-incident learning programs or operational excellence initiatives across an organization.
Skills and Tools
Python scripting and diagnostic tooling development
Arista (EOS) and Juniper (Junos) platforms
BGP, EVPN-VXLAN, IS-IS, OSPF, MPLS, QoS, and TCP/IP protocols
RDMA/RoCE lossless fabrics, PFC, ECN, and DCQCN tuning
Monitoring and observability stacks including Grafana, Prometheus, ThousandEyes, SNMP, NetFlow, and streaming telemetry
Salary and Benefits
Annual compensation ranging from $195,000 to $235,000 plus bonus
Restricted Stock Units included in all offers
Comprehensive health, dental, and vision insurance with employer HSA contributions
Paid time off, paid holidays, leave of absence programs, and paid parental life insurance
401(k) retirement plan featuring a company match up to 4% of salary
Professional development, tuition reimbursement, mental health support, and daily meals allowance
Commuter benefits, cell phone stipend, volunteer time off, and global travel insurance
About the Company
Crusoe operates as a vertically integrated AI infrastructure company built from the ground up to own and operate each layer of the stack, from electrons to tokens, powering global AI workloads. The organization approaches energy first to make AI infrastructure better for the world and faster for innovators.
How to Apply
Interested candidates should visit the official Crusoe careers page to submit an application for this position.
