Sr. Software Engineer - Ingestion Core team
About this role
Enterprises struggle to deeply understand their own data, creating a need for seamless ingestion of structured and unstructured information into analytics and machine learning environments. The Ingestion Core Team at Databricks addresses this problem by making data ingestion simple, reliable, and efficient across diverse formats and sources. Engineers on this team build distributed platform systems to handle high-volume, petabyte-scale data ingestion from cloud storage, databases, and file sources into Delta Lake with schema evolution and change data capture.
Candidates joining this group will work on streaming ingestion, incremental processing, and replication while optimizing workloads for throughput, cost, latency, reliability, and scale. The role also involves building monitoring and observability capabilities, collaborating with partner teams for retrieval-augmented generation and AI agent use cases, and constructing agentic SDKs with strong evaluations. Professionals who enjoy tackling complex distributed systems challenges and scaling high-throughput data infrastructure will thrive in this environment.
What you will do
- Build distributed infrastructure to ingest data from diverse sources and support streaming ingestion, incremental processing, and replication.
- Reduce end-to-end latency, increase throughput, and lower costs from the time data appears in source systems to when it lands in Delta Lake.
- Design and optimize streaming and distributed workloads for throughput, cost, latency, reliability, and scale.
- Optimize streaming workloads by exploring and applying machine learning techniques.
- Construct customer-facing and internal monitoring and observability capabilities to provide visibility into ingestion workflows.
- Collaborate with partner teams to enable use cases such as retrieval-augmented generation and AI agents.
- Build agentic SDKs backed by strong evaluations.
What you need
- Have at least five years of experience writing production code in languages such as Java, Scala, Go, C++, or Python.
- Possess experience architecting, developing, and deploying large-scale distributed and asynchronous systems.
- Demonstrate familiarity with distributed systems, data streaming, Spark, databases, data processing, or change data capture.
- Bring experience building or operating systems where scale, throughput, latency, reliability, and cost are critical considerations.
- Remain comfortable utilizing AI tools and defining effective evaluations to shorten the development loop.
Skills and tech stack
Questions candidates ask
What is the salary range for this position?
The expected salary range for non-commissionable roles or on-target earnings for commissionable roles is $166,000 to $225,000 USD. Actual compensation packages are based on several factors unique to each candidate, including job-related skills, depth of experience, relevant certifications and training, and specific work location.
What will I actually be working on?
You will build distributed infrastructure to ingest data from diverse sources and support streaming ingestion, incremental processing, and replication. You will also design and optimize streaming workloads, reduce end-to-end latency, and collaborate with partner teams to enable use cases like RAG and AI agents.
What background is required for this role?
Ideal engineers should have five or more years of experience writing production code in Java, Scala, Go, C++, or Python. Candidates also need experience architecting large-scale distributed and asynchronous systems along with familiarity with Spark, databases, streaming, or change data capture.