AI Seminar: Human-Robot Collaboration for Everyday Household Tasks

Dear all, Our next AI seminar is scheduled for Friday, March 7th. Talk details: AI Seminar: Human-Robot Collaboration for Everyday Household Tasks Speaker: Jacob Krantz, Research Scientist, Meta Time: 2:00 PM Location: KEC 1001 and Zoom Zoom link: https://oregonstate.zoom.us/s/98357211915<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Foregonstate.zoom.us%2Fs%2F98357211915&data=05%7C02%7Cai%40engr.oregonstate.edu%7C4443561f2aec48b7dfa208dd5a8ea140%7Cce6d05e13c5e4d6287a84c4a2713c113%7C0%7C0%7C638766290806060981%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=OVRZKoz%2Fy76m21lJxtkhlvaCpPcDsU4CAjH98e4BgYk%3D&reserved=0> Talk Abstract: I want a robot assistant that can work with me to accomplish everyday tasks around the home: tidying up the living room, preparing the table for dinner, etc. This broad capability involves the use of natural language for task specification, embodied multi-agent planning, and robust skill execution. Toward this end, I will present our simulation benchmark called PARTNR: Planning And Reasoning Tasks in humaN-Robot collaboration. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs as planners for these tasks, and reveal limitations in coordination, task tracking, and failure recovery. When paired with real human partners, these LLMs are less efficient than human-human collaboration by 1.5x and less efficient than a single human by 1.1x. Through the PARTNR benchmark, many directions of research can be pursued, for example multi-agent planning, 3D scene understanding, skill coordination, Sim2Real transfer, and even HRI studies. Speaker Bio: Jacob Krantz is a research scientist at the Fundamental AI Research lab at Meta (FAIR), where he works on systems of embodied intelligence with applications toward socially intelligent robotics. In the past he co-organized several community challenges relating to semantically driven navigation (VLN, ImageNav). He received his PhD in Computer Science from Oregon State University in 2023 under Dr. Stefan Lee and his BS in Computer Science with a minor in Physics from Gonzaga University in 2019. For future AI seminars, please visit: https://engineering.oregonstate.edu/EECS/research/AI-seminars<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fengineering.oregonstate.edu%2FEECS%2Fresearch%2FAI-seminars&data=05%7C02%7Cai%40engr.oregonstate.edu%7C4443561f2aec48b7dfa208dd5a8ea140%7Cce6d05e13c5e4d6287a84c4a2713c113%7C0%7C0%7C638766290806075515%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=ANAL3sCutBbvJUwZiSQZbxOV%2BzKaoU46ueXg4F6%2BDP4%3D&reserved=0>. Best, Christian Abou Mrad Graduate Student Oregon State University

Dear all, Reminder that our next AI seminar is scheduled for tomorrow, Friday, March 7th. Talk details: AI Seminar: Human-Robot Collaboration for Everyday Household Tasks Speaker: Jacob Krantz, Research Scientist, Meta Time: 2:00 PM Location: KEC 1001 and Zoom Zoom link: https://oregonstate.zoom.us/s/98357211915<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Foregonstate.zoom.us%2Fs%2F98357211915&data=05%7C02%7Cai%40engr.oregonstate.edu%7Cd49d9a1866ad4c06e4d608dd5d09d4d0%7Cce6d05e13c5e4d6287a84c4a2713c113%7C0%7C0%7C638769018981100634%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=3QO73noYCXzL1m%2Fx0rS3Upl6d%2FF1YECMDFDHaQLZdv4%3D&reserved=0> Talk Abstract: I want a robot assistant that can work with me to accomplish everyday tasks around the home: tidying up the living room, preparing the table for dinner, etc. This broad capability involves the use of natural language for task specification, embodied multi-agent planning, and robust skill execution. Toward this end, I will present our simulation benchmark called PARTNR: Planning And Reasoning Tasks in humaN-Robot collaboration. PARTNR stands as the largest benchmark of its kind, comprising 100,000 natural language tasks spanning 60 houses and 5,819 unique objects. We analyze state-of-the-art LLMs as planners for these tasks, and reveal limitations in coordination, task tracking, and failure recovery. When paired with real human partners, these LLMs are less efficient than human-human collaboration by 1.5x and less efficient than a single human by 1.1x. Through the PARTNR benchmark, many directions of research can be pursued, for example multi-agent planning, 3D scene understanding, skill coordination, Sim2Real transfer, and even HRI studies. Speaker Bio: Jacob Krantz is a research scientist at the Fundamental AI Research lab at Meta (FAIR), where he works on systems of embodied intelligence with applications toward socially intelligent robotics. In the past he co-organized several community challenges relating to semantically driven navigation (VLN, ImageNav). He received his PhD in Computer Science from Oregon State University in 2023 under Dr. Stefan Lee and his BS in Computer Science with a minor in Physics from Gonzaga University in 2019. For future AI seminars, please visit: https://engineering.oregonstate.edu/EECS/research/AI-seminars<https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fengineering.oregonstate.edu%2FEECS%2Fresearch%2FAI-seminars&data=05%7C02%7Cai%40engr.oregonstate.edu%7Cd49d9a1866ad4c06e4d608dd5d09d4d0%7Cce6d05e13c5e4d6287a84c4a2713c113%7C0%7C0%7C638769018981114796%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C&sdata=c9nJsRb4S8AMUmknn2%2B4M2bqgsOnxaZ5BjaEEhMujDk%3D&reserved=0>. Best, Christian Abou Mrad Graduate Student Oregon State University
participants (1)
-
Abou Mrad, Christian