About this role
Day-to-day work for this AI Research Intern involves contributing to the development of predictive world models for XPENG's autonomous driving and robotics systems. The intern will collaborate with a research team to tackle complex problems related to learning the dynamics of the physical world from large-scale multimodal data. Specific tasks include designing architectures, training models, and conducting empirical analysis to predict how a scene will evolve based on an agent's actions, effectively creating a learned simulator for policy training. This work is crucial for advancing intelligent mobility by improving the performance of driving and robotic policies.
The role exists to leverage advanced generative modeling techniques to build robust simulation tools necessary for autonomous systems. Interns will focus on areas such as multi-view future prediction, developing shared backbones for prediction and trajectory generation, and extending predictions into 3D scene representations. The goal is to create models that can generalize across different embodiments and sensor configurations.
This position is suited for a PhD student with a strong background in deep learning and generative models who has a proven track record of original research. The ideal candidate is someone who can drive a research project independently from conception through to publication, while also possessing the skills to translate research prototypes into scalable pipelines.
What you will do
- Drive a focused research project on predictive world models, encompassing problem formulation, architecture design, training, evaluation, and empirical analysis in partnership with a mentor.
- Contribute to specific research directions such as high-quality multi-view future prediction and generation.
- Develop architectures where a shared backbone predicts the future and generates trajectories or actions.
- Investigate methods for extending predictions beyond 2D pixels into shared multimodal latent spaces representing 3D scene representations.
- Build comprehensive evaluation methodologies covering prediction accuracy, generation fidelity, and closed-loop policy performance.
- Collaborate with research engineers to transition research prototypes into scalable training and inference pipelines.
- Publish and open-source research results when appropriate.
What you need
- Currently pursuing a PhD in Engineering, Computer Science, or a related field with a focus on Deep Learning, Computer Vision, or Generative Models.
- Must have first-author publications in top-tier venues such as CVPR, ICCV, ECCV, NeurIPS, ICLR, ICML, CoRL, RSS, or SIGGRAPH.
- Possess a strong, current foundation in generative modeling and experimental methodology, including hands-on experience building and evaluating models in PyTorch or JAX.
- Requires strong Python programming and software design skills, coupled with a solid understanding of data structures, algorithms, and large-scale data processing.
- Demonstrated experience building, training, fine-tuning, and evaluating models.
- Must be available to commit to a minimum of 12 weeks and work on-site at the Santa Clara office.
Nice to have
- Hands-on experience with generative models for video or 3D, such as diffusion or flow matching.
- Experience with world models or learned simulators for decision making, including model-based reinforcement learning and VLA models.
- Experience with multimodal foundation models and video tokenizers or VAEs.
- Prior research internship experience in autonomous driving, robotics, or embodied AI.
- Contributions to widely used open-source projects.
Benefits
- Fun, supportive and engaging environment
- Infrastructures and computational resources to support work
- Opportunity to work on cutting edge technologies
- Competitive compensation package
- Snacks, lunches, dinners, and fun activities
- Equal Opportunity Employer
Skills and tech stack
Questions candidates ask
What is the remote policy for this internship?
Interns are expected to work on-site at the Santa Clara office.
What is the compensation structure?
The posting mentions a competitive compensation package.
What is the primary technology focus?
The work focuses on developing world models that learn the dynamics of the physical world from large-scale multimodal data, using generative architectures like diffusion and flow-matching models.
What is the team environment like?
XPENG provides a fun, supportive and engaging environment and opportunities to work with top talents in the field.