Role Overview
The Senior Network Production Operations Engineer at Crusoe in San Francisco is a full-time production engineering position focused on maintaining the uptime and reliability of global network infrastructure. This role suits an experienced network specialist with a strong background in large-scale operational environments who wants to drive automation and incident response for AI workloads.
Key Responsibilities
Support uptime and reliability across the global edge, backbone, data center fabric, and GPU cluster network supporting AI workloads.
Write and maintain Python-based tooling to automate remediation workflows, reduce operational toil, and speed up resolution times.
Participate in high-severity network event triage, mitigation, communication, and postmortem documentation.
Perform thorough root cause analyses using custom and existing tooling to drive remediation items to closure.
Build automation using the monitoring and observability stack to surface actionable signals faster and minimize manual triage.
Maintain and improve operational standards, SOPs, escalation playbooks, and runbooks.
Partner with Architecture and SRE teams to execute reliability metrics, automated dashboards, and alerting.
Requirements
Minimum of 5 years of production network engineering experience focusing on operations, reliability, and incident response in internet-scale or large-scale environments.
Strong Python and scripting proficiency for writing diagnostic tools and extending automation pipelines.
Hands-on experience with monitoring tools and streaming telemetry, including Grafana, Prometheus, ThousandEyes, SNMP, and NetFlow/sFlow, with API scripting abilities.
Practical experience operating RDMA/RoCE lossless fabrics for HPC or GPU workloads, including DCQCN, PFC, and ECN tuning.
Solid working knowledge of production data center protocols such as BGP, EVPN-VXLAN, IS-IS, OSPF, MPLS, QoS, and TCP/IP.
Proficiency working with Arista EOS and Juniper Junos platforms in leaf-spine CLOS architectures across multi-vendor environments.
Comfort managing large device fleets across multi-region environments while taking on-call responsibilities.
A Bachelor's degree in Computer Science, Electrical Engineering, or a related field, or equivalent practical experience.
Nice to Have
Experience working with NVIDIA or Mellanox networking platforms inside GPU cluster settings.
Familiarity with Arbor or Kentik for traffic analysis and visibility into DDoS.
Exposure to defining SLIs and SLOs alongside SRE or product teams.
Background in operating device fleets exceeding 10,000 units within cloud or hyperscale environments.
Skills and Tools
Python scripting and automation
BGP, EVPN-VXLAN, IS-IS, OSPF, MPLS, QoS, TCP/IP
Arista EOS and Juniper Junos
RDMA/RoCE lossless fabric, PFC, ECN, DCQCN
Grafana, Prometheus, ThousandEyes, SNMP, NetFlow, streaming telemetry
Salary and Benefits
Annual base compensation ranging from $165,000 to $200,000 USD plus bonus
Restricted Stock Units included in all offers
Comprehensive health, dental, and vision insurance with employer HSA contributions
401(k) retirement plan featuring a company match up to 4% of salary
Paid time off, paid holidays, and paid parental leave
Paid life insurance, along with short-term and long-term disability coverage
Professional development, tuition reimbursement, and mental health support
Commuter benefits for parking and transit, plus a cell phone stipend and volunteer time off
About the Company
Crusoe operates as a vertically integrated AI infrastructure company built from the ground up to power massive AI workloads. The organization owns and operates each layer of the technology stack from energy sources to intelligence tokens, utilizing an energy-first approach to make AI infrastructure faster and more sustainable.
How to Apply
Please visit the official Crusoe careers page to submit your application for this position.
