Yisong Yue (he/him)
California Institute of Technology
1200 E. California Blvd.
CMS, 305-16
Pasadena, CA 91125

Office: 303 Annenberg

Contact Information >

• SceneCraft: An LLM Agent for Synthesizing 3D Scene as Blender Code -- we developed a Large Language Model (LLM) Agent for converting text descriptions into Blender-executable Python scripts which render complex scenes with up to a hundred 3D assets. This process requires complex spatial planning and arrangemet, which we tackle through a combination of advanced abstraction, strategic planning, and library learning. [arxiv]
  
• Tokenized Time Series Embeddings -- we developed a framework for learning a tokenization for downstream time series modeling, including training generalist models that can be applied to many domains. [arxiv][code]
  
• Symbolic Music Generation with Non-Differentiable Rule Guided Diffusion -- we developed a method for guided diffusion using non-differentiable rules, called Stochastic Control Guidance. Our approach is inspired by path integral control and can be applied in a plug-and-play way to any diffusion model. We demonstrate our approach on symbolic music generation. [arxiv][website]
  
• SustainGym: we have released a suite of environments designed to test the performance of RL algorithms on realistic sustainability tasks. [project][code]
• Farewell! We had three members depart the group during the 2022-2023 Academic Year:
  • Jennifer Sun completed her Ph.D. and will start as an Assistant Professor at Cornell.
  • Cameron Voloshin completed his Ph.D. and has started at Latitude AI.
  • Victor Dorobantu completed his Ph.D. and has started a postdoc at MIT.
  
• ICLR 2024: I will be serving as the Senior Program Chair at ICLR 2024. The rest of Program Chair team includes Swarat Chaudhuri, Katerina Fragkiadaki, Emtiyaz Khan, and Yizhou Sun.
  
• Automatic Gradient Descent: We have developed a new hyperparameter-free optimizer for deep neural networks. Our method has been demonstrated at ImageNet scale using ResNet50 architectures.
  [arxiv][code][blog post]
• Conformal Generative Modeling: We have developed a new framework for generative modeling that works on a wide range of manifolds. [project]
  
• Sabbatical at Latitude AI: I have started my role as Principal Scientist at Latitude AI. Read more here.
• ICML 2023: I will be serving as Communications Co-Chair (with Marco Cuturi) for ICML 2023.
• Invited Talk: I recently gave an invited talk at DeepMath 2022 on developing architecture-aware theory & practical algorithms for neural networks. [slides]
  
• Neurosymbolic Programming for Science: we recently put out a position paper on Neurosymbolic Programming for Science. This position is informed by our experience collaborating with scientists: science is an iterative process of analyzing data, proposing hypotheses, and conducting experiments. Because scientists reason more readily in symbolic terms, it is important to develop frameworks that natively inherit both the flexibility of neural networks and the rich semantics of symbolic models.
  
• Farewell! We had four members depart the group during the 2021-2022 Academic Year:
  • Ugo Rosolia completed his postdoc and has started as a research scientist at Amazon.
  • Guanya Shi completed his Ph.D. and will start as an Assistant Professor at CMU.
  • Eric Zhan completed his Ph.D. and started at Argo AI.
  • Jeremy Bernstein completed his Ph.D. and will start as a postdoc at MIT.
  
• Group Assimilation Exercise: I wrote a post about the Group Assimilation Exercise that my group does annually. The goal of the exercise is the lower the barriers to the group for providing me with constructive feedback.
  
• Neural-Fly published in Science Robotics: our work on Neural-Fly has been published in Science Robotics. [article][press release]
  
• Neurosymbolic Programming Summer School (July 11-13 2022 @Caltech): Neurosymbolic programming is an exciting new area at the intersection of Program Synthesis and Machine Learning that aims to learn models that incorporate program-like structure. More info here.
• Sabbatical at Argo AI: I have started my role as Principal Scientist at Argo AI. Learn more about my new role here.
• Neurosymbolic Programming Survey Paper: published in Foundations and Trends in Programming Languages. (preprint)
• Best Paper Award: "End-to-End Sequential Sampling and Reconstruction for MR Imaging" won Best Paper at ML4Health 2021! [project]
  
• Invited Talk: I presented on Neurosymbolic Programming at the Caltech Explainable AI Virtual Workshop. [slides]
  
• Advances in Machine Learning-Assisted Directed Evolution for Protein Design: new paper showing that smart training set selection can significantly improve machine learning-guided directed evolution in highly epistatic protein fitness landscapes with large low-fitness regions. [online][bioRxiv][code]
  
• Personalized Preference Learning from Spinal Cord Stimulation to Exoskeletons: interactive learning approaches for personalizing medical assistive devices. [slides]
  
• Fine-Grained Identification of Biological Neural Networks: our KDD 2021 paper on "Fine-Grained System Identification of Nonlinear Neural Circuits" demonstrated identifying sparse nonlinear neural networks from real neural recording data. [arxiv][code]
  
• Multi-Agent Behavior Workshop: Video recordings from our Multi-Agent Behavior Workshop (co-located with CVPR 2021) are now available. [playlist]
  
• Bon Voyage! We have four departures in the 2020-2021 academic year:
  • Ellen Novoseller completed her PhD and has started as a postdoc in Ken Goldberg's group at UC Berkeley.
  • Angie Liu completed her postdoc and will start as an assistant professor at Johns Hopkins University.
  • Joe Marino completed his PhD and will start as a research scientist at DeepMind.
  • Jialin Song completed his PhD and has started as a research scientist at NVIDIA AI.
  
• Paper Award: Our work on Task Programming won Best Student Paper at CVPR 2021! [arxiv][code][project]
  
• AI for Science Challenge: We have released the Multi-Agent Behavior Dataset: Mouse Dyadic Social Interactions. [arxiv][dataset][code]
  
• Invited Talk: Video now available of my talk on "Learning to Optimize as Policy Learning" presented at the Princeton Optimization Seminar. [slides]
  
• Invited Talk: I presented on "Learning for Safety-Critical Control in Dynamical Systems" at the Control Meets Learning seminar series.
  
• Invited Talk: I gave the Earnest C. Watson Lecture on January 13th, 2021.
  
• Invited Talk: I gave a presentation on AI for Adaptive Experiment Design at the Directions in ML: AutoML and Automating Algorithms hosted by Microsoft Research. [slides]
  
• Neural-Swarm2: we have released the details of our heterogeneous neural swarm approach! [paper]
  
• Invited Talk: I am presenting on "Competitive Algorithms for Online Control" at the Simons Institute Workshop on Mathematics of Online Decision Making. [slides]
  
• Invited Talk: I am presenting at the Workshop on Imitation Learning: Single & Multi-Agent hosted by DAI 2020. [slides]
• Invited Talk: I am presenting on "Learning to Optimize as Policy Learning" at the Princeton Optimization Seminar. [slides]
• ML for Rover Path Planning: check out this video of our MLNav extension of ENav (the path planner currently on the Mars Perseverance Rover). [arxiv]
  
• NSF Expeditions on Program Learning: we are starting a new research initiative titled Understanding the World Through Code, funded through the NSF Expeditions in Computing Program.
• Controllable Generation of Behaviors: we designed a new method that can generate behaviors calibrated to many different styles. [paper][code][demo]
  
• AI for Swarm Automation: check out this new article titled Machine Learning Helps Robot Swarms Coordinate. [paper 1][paper 2] (videos below)
  
• Invited Talk: I gave a talk on "Learning for Safety-Critical Control in Dynamical Systems" in the Physics ∩ ML seminar. [slides] (video below)
  
• Best Paper Award: Our work on preference learning for exoskeleton gait optimization is appearing at ICRA 2020 with a Best Paper Award! [arxiv][project] (video below).
  
• Partnered with PyTorch to film a short clip on our robotics research at CAST.
  
• Invited Talk: I am giving a talk at University of Chicago on November 13th, 2019.
• Invited Talk: I am giving a talk at the UIUC Computer Science Colloquium on November 11th, 2019.
• Keynote talk at AI-for-Science: I am giving a keynote talk on Adaptive Experiment Design at the Caltech AI-for-Science workshop.
  
• Invited Talk: I am speaking at the PyTorch Developer Conference on October 10th, 2019.
  
• Invited Workshop: I am giving a talk at the Workshop on Automated Algorithm Design hosted by the TTI-Chicago 2019 Summer Workshop Program.
• Invited Workshop: I am giving a talk at the Workshop on Data Collection, Curation, and Labeling for Mining and Learning co-located with KDD 2019.
• Invited Workshop: I am giving a talk at the Adversarial Machine Learning in Real-World Computer Vision Systems co-located with CVPR 2019.
• Invited Workshop: I am giving a talk at the 2019 Workshop on Human In the Loop Learning (HILL) co-located with ICML 2019 [video].
• ICML 2019 Workshop: Please consider attending or submitting to the Real-World Sequential Decision Making Workshop co-located with ICML 2019!
• Simons Symposium: I am attending a Simons Symposium on New Directions in Theoretical Machine Learning on May 5-11, 2019.
• Invited Talk: I am speaking at the Machine Learning Seminar at CMU on April 23rd, 2019.
• Neural Lander: Our work on deep learning for provably stable drone landing control is appearing at ICRA 2019! [arxiv] (video below)
  
• Earthquake Early Detection: State-of-the-art results on earthquake early detection using deep learning! [arxiv]
• Earthquake Localization: State-of-the-art results on earthquake localization using deep learning! [arxiv]
• Invited Workshop: I am presenting at the NeurIPS 2018 workshop on Imitation Learning and its Challenges in Robotics.
• Invited Talk: I am presenting at UCLA on November 20th, 2018.
• Invited Talk: I am presenting at the University of Maryland College Park on October 24th, 2018.
• Invited Talk: I am presenting at Microsoft Research Redmond on October 3rd & 4th, 2018.
• Invited Talk: I am presenting at the University of Washington AI Seminar on October 2nd, 2018.
• Invited Talk: I am presenting at the Rice Computer Science Colloquium on September 27th, 2018.
• Okawa Award: I recently received the Okawa Foundation Research Grant.
• Invited Workshop: I am presenting on Inference+Imitation at the Tractable Probabilistic Models workshop at ICML 2018.
• Invited Workshop: I am presenting on Machine Teaching for Human Learners at the Humanizing AI workshop at IJCAI 2018.
• Tutorial: I am giving a tutorial on imitation learning with Hoang Le at ICML 2018. (video below)
  
• Invited Talk: I am presenting at the Intel AI DevCon. (video below)
  
• Invited Talk: Invited talk at UT Austin AI Seminar on March 30th, 2018.
• Invited Workshop: I am attending the Data-driven Algorithmics workshop on November 5-10, 2017.
• Invited Talk: Invited talk at Southern California Machine Learning Symposium on October 6th, 2017.
• Blog Article: Our paper on smooth imitation learning was covered in an invited blog article titled Beyond Deep Learning: a Case Study in Sports Analytics. [ICML paper]
• Invited Talk: Invited talk at Microsoft Research Colloquium at MSR New England on September 6th, 2017. [video]
• Dagstuhl Seminar: I am attending the Machine Learning and Formal Methods Dagstuhl Seminar on August 28th - September 1st, 2017.
• Press Release: Our work on data-driven speech animation is highlighted on Road to VR! [SIGGRAPH paper][demo video (shown below)]
  
• Press Release & Interview: Neural Networks Model Audience Reactions to Movies, also radio interview. [CVPR paper]
• Microsoft Faculty Summit: I am attending the Edge of AI Microsoft Research Faculty Summit on July 17-18, 2017.
• Invited Talk: Invited talk at Machine Learning Methods for Recommender Systems Workshop to be held at SDM 2017.
• Invited talk: Invited talk at Symposium on Machine Learning and Human Behavior to be held at UC Irvine on March 10th, 2017.
• Best Paper Runner Up: Our paper "Data-Driven Ghosting using Deep Imitation Learning" wins Runner Up to Best Research Paper at the MIT Sloan Sports Analytics Conferece! [pdf][project][press release][demo video (shown below)]
  
• Caltech Computes Alumni College: I gave a talk titled Automatically Improving Automation using Big Data at the Caltech Computes Alumni College.
  
• Southern California Machine Learning Symposium: I am co-organizing the next SoCal ML Symposium, to be held at Caltech on November 18th, 2016.
• Innovation in Artificial Intelligence: I will be moderating a discussion panel on Innovation in Artificial Intelligence on September 15th, 2016.
• Coverage on Sports Illustrated: My collaboration with Disney Research on imitation learning for camera control is featured on Sports Illustrated! [ICML paper][CVPR paper]
• Bloomberg Data Science Research Grant: I'm delighted to be awarded a Data Science Research Grant from Bloomberg Labs!
• Personalization Workshop: Please consider participating in our Computational Frameworks for Personalization Workshop being held at ICML 2016!
• Sports Analytics Workshop: Please consider participating in our Large-Scale Sports Analytics Workshop being held at KDD 2016!
• Blog Post: Thoughts on NeurIPS 2015 and OpenAI.
• Invited Workshop: Algorithms for Human Robot Interaction Workshop.
• Blog Post: Thoughts on KDD 2015.
• Fundraising Chair of AISTATS 2016.
• Invited Talk at Reflections | Projections 2015 organized by ACM@UIUC.
• Invited Workshop: Data-driven Algorithmics Workshop. [slides]
• A Decision Tree Framework for Spatiotemporal Sequence Prediction accepted for publication at KDD 2015. [pdf][demo]
• Sports Analytics Workshop: Please consider participating in our Large-Scale Sports Analytics Workshop being held at KDD 2015!
• Interview by Jessica Stoller-Conrad @Caltech. [link]
• Invited Talk at Human Propelled Machine Learning Workshop at NeurIPS 2014.
• Learning Fine-Grained Spatial Models for Dynamic Sports Play Prediction accepted for publication at ICDM 2014. [pdf][demo][press release]
• Personalization Workshop: Please consider participating in our Personalization Workshop being held at NeurIPS 2014!
• Sports Analytics Workshop: Please consider participating in our Large-Scale Sports Analytics Workshop being held at KDD 2014!
• Personalized Collaborative Clustering accepted for publication at WWW 2014. [pdf][slides][data]
• Invited Talk at DISCML Workshop at NeurIPS 2013.
• Press release of EMS ambulance allocation project with the iLab is now up! [link][research paper]
• Invited talk at WSCD Workshop at WSDM 2012.