Staff Software Engineer, Lakeflow Pipelines DR
About this role
Designing and implementing distributed systems that handle cross-region replication and recovery for data pipelines forms the core daily work on the Lakeflow Disaster Recovery team. Databricks builds a unified Lakehouse platform for data engineering, analytics, and AI, and the Lakeflow product specifically helps enterprises manage streaming and batch ETL pipelines. Engineers in this position focus on ensuring business-critical data products continue running smoothly through infrastructure failures and cloud-region outages.
Tackling complex engineering problems involving consistency, idempotency, causal ordering, and safe failover without data duplication occupies much of the schedule. Team members work on streaming checkpoints, stateful operator state, metadata reconciliation, and failure-injection testing to maintain high availability. Candidates who enjoy solving foundational reliability challenges in large-scale distributed databases and storage systems will find a strong operational fit here.
What you will do
- Design and implement distributed systems that replicate and recover pipelines across regions.
- Solve technical challenges involving consistency, idempotency, causal ordering, failover, and safe recovery.
- Develop cross-region replication mechanisms for Lakeflow pipelines, streaming tables, and materialized views.
- Maintain distributed consistency across pipeline dependencies, table versions, and transaction logs.
- Build failover and failback workflows equipped with conservative correctness guardrails.
- Execute deep clone operations, metadata reconciliation, observability improvements, and failure-injection testing.
- Conduct high-fidelity recovery simulations and game-day testing to validate failure modes.
What you need
- Bring strong software engineering skills in production languages such as Java, Scala, C++, Go, or Python.
- Demonstrate understanding of consistency, transactions, idempotency, replication, checkpointing, and data lineage.
- Show the ability to define and work toward a multi-year technical vision with incremental deliverables.
- Possess eight or more years of experience working on databases, distributed systems, storage systems, streaming systems, or reliability engineering.
Nice to have
- PhD or advanced research experience in databases, distributed systems, or storage
Skills and tech stack
Questions candidates ask
What is the salary range for this role?
The expected local pay range for this position is $192,000 to $260,000 USD. Actual compensation packages are based on factors such as job-related skills, depth of experience, relevant certifications, and specific work location.
What technologies and languages will I use?
Engineers write code in production languages such as Java, Scala, C++, Go, or Python. The work centers on distributed systems, databases, storage systems, streaming architectures, and reliability engineering.
What will I actually work on day-to-day?
The focus is on designing distributed systems that replicate and recover pipelines across regions for Lakeflow disaster recovery. This includes solving problems related to consistency, idempotency, failover workflows, streaming checkpoints, and failure-injection testing.