Senior Software Engineer — Backend (Files Team)
About this role
This role addresses the need to build and scale the foundational filesystem that will power agentic AI workloads. The team at Databricks is focused on creating the Data Intelligence Platform, which enables customers to derive deep data insights for complex business problems. A core challenge is establishing a reliable and performant storage layer that can support the emerging use of the filesystem as the runtime interface for AI agents, managing context, memory, and artifacts.
The Senior Software Engineer on the Files team will be instrumental in innovating at the intersection of storage, performance engineering, and distributed systems. This individual will be responsible for designing, building, and operating the filesystem platform that supports tens of thousands of daily data, AI, and agentic workloads. Key activities include pushing filesystem performance for demanding tasks by profiling and optimizing across the entire stack, from the API layer down to the Linux kernel.
This position exists to define the technical direction for how content is organized across Databricks, standardizing filesystem primitives applicable to all products. The ideal candidate is someone who understands how to build systems from the ground up when existing solutions are insufficient at scale, thrives on tackling complex infrastructure challenges, and possesses a deep technical curiosity about storage and distributed systems.
What you will do
- Design, build, and operate the filesystem platform that powers numerous data, AI, and agentic workloads daily.
- Improve filesystem performance for the most demanding workloads through profiling and optimization across the stack, including the API layer and the Linux kernel.
- Collaborate with various platform teams to design and evolve next-generation compute and storage architectures.
- Set the technical direction for content organization by standardizing filesystem primitives across all Databricks products.
- Ensure the filesystem can effectively support the requirements of agentic AI workloads.
What you need
- A Bachelor's degree in Computer Science or a related field is required.
- Candidates must possess more than five years of production-level experience in languages such as Java, Scala, C++, Go, Rust, or a similar language.
- A minimum of three years of experience developing large-scale distributed systems is necessary.
- Experience with containerization technologies like Kubernetes, Docker, or MicroVMs (e.g., Firecracker) is required.
- A strong understanding of cloud provider internals, such as AWS, Azure, or GCP, is essential.
- Experience with filesystems and FUSE, as well as building object/blob storage (S3, Azure Blob Storage, GCS etc.), is required.
- Proven experience in systems performance engineering across microservices, down to the Linux level is a must.
Nice to have
- Experience with FUSE
Benefits
- Eligibility for annual performance bonus
- Equity
- Comprehensive benefits and perks
Skills and tech stack
Questions candidates ask
What is the remote policy for this role?
The job is listed for San Francisco, California.
What is the compensation range?
The pay range for this role is listed as $166,000 — $225,000 USD for non-commissionable roles or on-target earnings for commissionable roles.
What is the primary technology stack?
The required languages include Java, Scala, C++, Go, Rust, and experience with technologies like Kubernetes, Docker, and cloud provider internals.
What is the team structure like?
The role involves partnering with teams across the platform to design and evolve compute and storage architectures.