Deployable Decision-Making: Embracing Semantics for Robotic Safety in Everyday Scenarios

Recent breakthroughs in machine learning have opened up opportunities for robots to build a semantic understanding of their operating environment and interact with humans in more natural ways. While machine learning has unlocked new potentials for robot autonomy, as robots venture into the real world, physical interactions with the surrounding environment pose additional challenges. One typical challenge in practical applications is providing safety guarantees in robot decision-making. Much of the safe robot decision-making literature today focuses on explicit safety constraints defined in the robot state and input space. However, in practical applications, robots are often required to infer semantics-grounded safe actions from perception input. While recent machine learning techniques are increasingly capable of distilling semantic information from perception, translating the semantic understanding to explicit safety constraints is non-trivial. In this proposed project, we aim to close the perception-action loop and develop mathematical foundations and algorithmic tools that enable robots to make intelligent and semantically safe decisions.