Junyao Shi

Hi! My name is Junyao Shi (施钧耀). I'm a PhD student in Computer and Information Science (CIS) at University of Pennsylvania GRASP Labortory, advised by Dinesh Jayaraman.

My research focuses on robot learning, with a particular emphasis on leveraging internet-scale data and foundation models for robotics. My ultimate goal is to free humans from dangerous or mundane tasks by entrusting them to machines.

Previously, I received my B.S. in Computer Science from Columbia University, where I worked with Peter Allen on brain-signal guided robot learning and Tony Dear on reinforcement learning for snake robot locomotion.

Email  /  Google Scholar  /  CV  /  Twitter  /  LinkedIn  /  Github

profile photo

Publications

ZeroMimic: Distilling Robotic Manipulation Skills from Web Videos
Junyao Shi*, Zhuolun Zhao*, Tianyou Wang, Jason Ma, Dinesh Jayaraman
ICRA, 2025
project page / video

We introduce ZeroMimic, a system that distills robotic manipulation skills from egocentric human web videos for zero-shot deployment in diverse environments with a variety of objects.

Composing Pre-Trained Object-Centric Representations for Robotics From "What" and "Where" Foundation Models
Junyao Shi*, Jianing Qian*, Jason Ma, Dinesh Jayaraman
ICRA, 2024
project page / paper / video

POCR chains pre-trained "what" and "where" foundation models to create object-centric representations for robotics. The "where" model identifies object candidates with segmentation masks, which are then bound to slots and encoded by the "what" model, enabling robots to learn policies over these representations.

Don't Yell at Your Robot
Don't Yell at Your Robot: Physical Correction as the Collaborative Interface for Language Model Powered Robots
Chuye Zhang*, Yifei Simon Shao*, Harshil Parekh, Junyao Shi, Pratik Chaudhari, Vijay Kumar, Nadia Figueroa
RSS Genarative Modeling meets HRI Workshop, 2024
project page / paper / video
Maximizing BCI Human Feedback Using Active Learning
Maximizing BCI Human Feedback Using Active Learning
Zizhao Wang*, Junyao Shi*, Iretiayo Akinola*, Peter Allen
IROS, 2020
paper
Deep Reinforcement Learning for Snake Robot Locomotion
Deep Reinforcement Learning for Snake Robot Locomotion
Junyao Shi, Tony Dear, Scott David Kelly
IFAC, 2020
paper
Accelerated Robot Learning via Human Brain Signals
Iretiayo Akinola*, Zizhao Wang*, Junyao Shi, Xiaomin He, Pawan Lapborisuth, Jingxi Xu, David Watkins-Valls, Paul Sajda, Peter Allen
ICRA, 2020
project page / paper / video

Design and source code from Jon Barron's website.