Hardware Failure Analysis Engineer - Memphis
About this role
Engineers who thrive on curiosity and want to solve complex fleet-scale hardware challenges will find a natural home at xAI. Operating with a flat organizational structure, the team expects every member to be hands-on, communicate clearly, and take direct ownership of system reliability.
In this role, the engineer will investigate the root causes of hardware failures across GPU trays, networking gear, and power systems rather than relying on simple component swaps. By driving technical escalations with vendors and analyzing fleet-wide defect clusters, the successful candidate directly safeguards the infrastructure powering advanced artificial intelligence research.
What you will do
- Manage named failure classes spanning GPU trays, baseboards, network interface cards, motherboards, memory, power supplies, thermal modules, and rack-scale patterns.
- Conduct recurrence analysis and fleet-wide defect clustering to identify systemic hardware problems before they escalate.
- Develop evidence-backed technical escalation packages to compel action from OEM, ODM, and component manufacturers.
- Partner directly with external manufacturers to resolve firmware issues, bring-up complications, and manufacturing escapes through committed corrective actions.
- Translate diagnostic findings into actionable RMA policies, spare parts strategies, and definitive troubleshooting stop-rules.
- Handle the failure analysis intake queue on a rotational basis while maintaining performance within established service level agreements.
What you need
- Hold a Bachelor's degree in Systems Engineering, Electrical Engineering, Computer Science, or a related field, or possess equivalent practical experience.
- Bring a minimum of two years of professional experience in hardware reliability engineering, ideally within high-performance computing or data center settings.
- Demonstrate expertise in reviewing hardware specifications, analyzing firmware, and validating releases.
- Possess hands-on experience managing RMA processes, negotiating with vendors, and securing resolutions beyond standard protocols.
- Show a proven capability to diagnose intricate hardware faults, including intermittent or grey failures, utilizing diagnostic software, logic analyzers, and specialized tools.
- Demonstrate familiarity with standard data center hardware components such as servers, GPUs, and networking gear.
- Apply scripting proficiency in Python or Bash for automation alongside general experience in at least one systems language like C, C++, Java, or Rust.
Nice to have
- Experience in AI/ML infrastructure or supercomputing environments
- Knowledge of vendor ecosystems including NVIDIA, Dell, HP, and Supermicro, alongside supply chain management
- Certifications in hardware engineering or reliability such as CRE or CompTIA Server+
- Prior work history in a fast-paced startup or technology company
Skills and tech stack
Questions candidates ask
What team will I be working with?
The organization is small, highly motivated, and focused on engineering excellence with a flat operational structure. All employees are expected to be hands-on and contribute directly to the company's mission.
What are the core duties regarding hardware failures?
You will determine the true root cause of fleet hardware failures at both component and system levels and drive fixes through vendors to the manufacturer. This involves turning swap-it-again routines into permanent vendor redesigns, firmware fixes, or manufacturing escape resolutions.
What programming languages and technical skills are required?
You need proficiency in scripting using Python or Bash for automation and analysis, plus general experience in at least one systems language such as C, C++, Java, or Rust. You are not required to be an expert in all of them.