Role Overview
Handshake is seeking a full-time Senior Data Platform Engineer to construct and maintain the infrastructure that powers its career marketplace and AI solutions. Based in San Francisco, CA, this hands-on technical role is designed for professionals eager to direct data infrastructure strategy and support large-scale AI development.
Key Responsibilities
Direct the technical roadmap and operations for the Airflow and Astronomer platform, overseeing production deployments, scheduler capacity, DAG packaging, and incident response.
Create standardized paths for building, testing, deploying, and managing workflows using CI checks, reusable operators, and clear runbooks.
Design and maintain streaming and change data capture pipelines utilizing tools like Pub/Sub, Datastream, and Dataflow or Beam.
Ensure data delivery handles retries, schema modifications, duplicate entries, late-arriving events, and backfills while establishing clear reliability goals.
Enhance cloud foundations for data tasks using Kubernetes, Terraform, secrets management, IAM, CI/CD pipelines, and cost-efficient capacity scaling.
Drive cross-functional architectural choices regarding data contracts, mentor engineers, and unify platform approaches across product, analytics, and ML teams.
Take part in on-call duties, resolve production system outages, and implement permanent fixes based on past incidents.
Construct robust orchestration and data foundations to support AI operations, such as evaluation datasets, agent-facing data, and batch inference.
Requirements
Strong software engineering capability using Python, coupled with a history of building and running production-grade data or distributed systems.
Comprehensive hands-on experience with Apache Airflow beyond writing DAGs, including scaling, deployment, monitoring, upgrades, and troubleshooting.
Familiarity with event-driven or streaming data architectures, featuring a stream processing framework and managed event bus.
Solid comprehension of change data capture, delivery guarantees, idempotency, schema evolution, ordering, and pipeline recovery.
Practical background in cloud infrastructure and infrastructure-as-code principles, handling containers, Kubernetes, CI/CD, and production observability.
Demonstrated success leading complex infrastructure initiatives by establishing technical direction and making pragmatic architecture choices.
Effective communication skills and a history of collaborating across departments while maintaining end-to-end ownership of production systems.
Nice to Have
Familiarity with GCP services including BigQuery, Pub/Sub, Dataflow, Datastream, GKE, and Cloud Storage.
Experience working with Astronomer, Apache Beam, Terraform, Spacelift, Datadog, dbt, Spark, or Dataproc.
Background in supporting both batch consumers and low-latency product features or machine learning properties.
Experience aiding AI or ML workloads like reproducible evaluations, inference pipelines, and governed data access.
Skills and Tools
Python
GCP
Kubernetes
Spark
Airflow
dbt
Terraform
Salary and Benefits
Annual Salary Range: $166,000 to $207,000 in USD
Equity ownership in a rapidly expanding company
Competitive compensation package and 401(k) matching with financial coaching
Comprehensive medical, dental, and vision insurance alongside mental health support and a $500 wellness stipend
Paid parental leave, fertility benefits, and parental coaching
$2,000 annual learning stipend for ongoing professional development
Flexible paid time off, 15 holidays, and 2 flex days
Commuting and internet support, alongside free lunch and gym access at the San Francisco office
About the Company
Handshake launched with the mission to provide every individual with a pathway to a meaningful career regardless of background or connections. Today, the platform connects millions of job seekers with over a million employers and thousands of educational institutions. Having introduced Handshake AI, the organization partners directly with frontier AI labs to publish benchmarks and build advanced data evaluations.
How to Apply
Interested candidates should visit the official Handshake careers page to submit an application for this position.
