Ego4D: Around the World in 3,000 Hours of Egocentric Video

Kristen Grauman; Andrew Westbury; Eugene Byrne; Zachary Chavis; Antonino Furnari; Rohit Girdhar; Jackson Hamburger; Hao Jiang; Miao Liu; Xingyu Liu; Miguel Martin; Tushar Nagarajan; Ilija Radosavovic; Santhosh Kumar Ramakrishnan; Fiona Ryan; Jayant Sharma; Michael Wray; Mengmeng Xu; Eric Zhongcong Xu; Chen Zhao; Siddhant Bansal; Dhruv Batra; Vincent Cartillier; Sean Crane; Tien Do; Morrie Doulaty; Akshay Erapalli; Christoph Feichtenhofer; Adriano Fragomeni; Qichen Fu; Abrham Gebreselasie; James Hillis; Xuhua Huang; Yifei Huang; Wenqi Jia; Weslie Khoo; Satwik Kottur; Anurag Kumar; Federico Landini; Chao Li; Yanghao Li; Zhenqiang Li; Karttikeya Mangalam; Raghava Modhugu; Jonathan Munro; Tullie Murrell; Takumi Nishiyasu; Will Price; Paola Ruiz Puentes; Merey Ramazanova; Leda Sari; Kiran Somasundaram; Audrey Southerland; Yusuke Sugano; Ruijie Tao; Minh Vo; Yuchen Wang; Xindi Wu; Takuma Yagi; Ziwei Zhao; Yunyi Zhu; David Crandall; Dima Damen; Giovanni Maria Farinella; Christian Fuegen; Bernard Ghanem; Vamsi Krishna Ithapu; C. Jawahar; Hanbyul Joo; Kris Kitani; Haizhou Li; Richard Newcombe; Aude Oliva; Hyun Soo Park; James M. Rehg; Yoichi Sato; Jianbo Shi; Mike Zheng Shou; Antonio Torralba; Lorenzo Torresani; Mingfei Yan; Jitendra Malik; IEEE COMP SOC

doi:10.1109/CVPR52688.2022.01842

Conference proceeding

Ego4D: Around the World in 3,000 Hours of Egocentric Video

Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, …

2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2022), Vol.2022-, pp.18973-18990

IEEE Conference on Computer Vision and Pattern Recognition

01/01/2022

DOI: https://doi.org/10.1109/CVPR52688.2022.01842

Abstract

Computer Science

Computer Science, Artificial Intelligence

Imaging Science & Photographic Technology

Science & Technology

Technology

We introduce Ego4D, a massive-scale egocentric video dataset and benchmark suite. It offers 3,670 hours of daily-life activity video spanning hundreds of scenarios (household, outdoor, workplace, leisure, etc.) captured by 931 unique camera wearers from 74 worldwide locations and 9 different countries. The approach to collection is designed to uphold rigorous privacy and ethics standards, with consenting participants and robust de-identification procedures where relevant. Ego4D dramatically expands the volume of diverse egocentric video footage publicly available to the research community. Portions of the video are accompanied by audio, 3D meshes of the environment, eye gaze, stereo, and/or synchronized videos from multiple egocentric cameras at the same event. Furthermore, we present a host of new benchmark challenges centered around understanding the first-person visual experience in the past (querying an episodic memory), present (analyzing hand-object manipulation, audio-visual conversation, and social interactions), and future (forecasting activities). By publicly sharing this massive annotated dataset and benchmark suite, we aim to push the frontier of first-person perception.

Metrics

1 Record Views

Details

Title: Ego4D: Around the World in 3,000 Hours of Egocentric Video
Creators - without role: Kristen Grauman - The University of Texas at Austin
Andrew Westbury - Meta AI, New York, NY 10003 USA
Eugene Byrne - Meta AI, New York, NY 10003 USA
Zachary Chavis - University of Minnesota System
Antonino Furnari - University of Catania
Rohit Girdhar - Meta AI, New York, NY 10003 USA
Jackson Hamburger - Meta AI, New York, NY 10003 USA
Hao Jiang - META Health
Miao Liu - Georgia Institute of Technology
Xingyu Liu - Carnegie Mellon University
Miguel Martin - Meta AI, New York, NY 10003 USA
Tushar Nagarajan - The University of Texas at Austin
Ilija Radosavovic - Berkeley College
Santhosh Kumar Ramakrishnan - The University of Texas at Austin
Fiona Ryan - Georgia Institute of Technology
Jayant Sharma - University of Minnesota System
Michael Wray - University of Bristol
Mengmeng Xu - King Abdullah Univ Sci & Technol, Riyadh, Saudi Arabia
Eric Zhongcong Xu - National University of Singapore
Chen Zhao - King Abdullah University of Science and Technology
Siddhant Bansal - International Institute of Information Technology, Hyderabad
Dhruv Batra - Meta AI, New York, NY 10003 USA
Vincent Cartillier - Georgia Institute of Technology
Sean Crane - Carnegie Mellon University
Tien Do - University of Minnesota
Morrie Doulaty - Meta, New York, NY USA
Akshay Erapalli - Meta, New York, NY USA
Christoph Feichtenhofer - Meta AI, New York, NY 10003 USA
Adriano Fragomeni - University of Bristol
Qichen Fu - Carnegie Mellon University
Abrham Gebreselasie - Carnegie Mellon University Africa
James Hillis - META Health
Xuhua Huang - Carnegie Mellon University
Yifei Huang - The University of Tokyo
Wenqi Jia - Georgia Institute of Technology
Weslie Khoo - Indiana University
Satwik Kottur - Meta, New York, NY USA
Anurag Kumar - META Health
Federico Landini - Meta, New York, NY USA
Chao Li - META Health
Yanghao Li - Meta AI, New York, NY 10003 USA
Zhenqiang Li - The University of Tokyo
Karttikeya Mangalam - Berkeley College
Raghava Modhugu - International Institute of Information Technology, Hyderabad
Jonathan Munro - University of Bristol
Tullie Murrell - Meta AI, New York, NY 10003 USA
Takumi Nishiyasu - The University of Tokyo
Will Price - University of Bristol
Paola Ruiz Puentes - Universidad de Los Andes
Merey Ramazanova - King Abdullah University of Science and Technology
Leda Sari - META Health
Kiran Somasundaram - META Health
Audrey Southerland - Georgia Institute of Technology
Yusuke Sugano - The University of Tokyo
Ruijie Tao - National University of Singapore
Minh Vo - META Health
Yuchen Wang - Indiana University
Xindi Wu - Carnegie Mellon University
Takuma Yagi - The University of Tokyo
Ziwei Zhao - Indiana University
Yunyi Zhu - National University of Singapore
David Crandall - Indiana University
Dima Damen - University of Bristol
Giovanni Maria Farinella - University of Catania
Christian Fuegen - Meta, New York, NY USA
Bernard Ghanem - King Abdullah University of Science and Technology
Vamsi Krishna Ithapu - META Health
C. Jawahar - International Institute of Information Technology, Hyderabad
Hanbyul Joo - Meta AI, New York, NY 10003 USA
Kris Kitani - Carnegie Mellon University
Haizhou Li - National University of Singapore
Richard Newcombe - META Health
Aude Oliva - Moscow Institute of Thermal Technology
Hyun Soo Park - University of Minnesota System
James M. Rehg - Georgia Institute of Technology
Yoichi Sato - The University of Tokyo
Jianbo Shi - California University of Pennsylvania
Mike Zheng Shou - National University of Singapore
Antonio Torralba - Moscow Institute of Thermal Technology
Lorenzo Torresani - Dartmouth Hospital
Mingfei Yan - META Health
Jitendra Malik - Berkeley College
IEEE COMP SOC
Publication Details: 2022 IEEE/CVF CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION (CVPR 2022), Vol.2022-, pp.18973-18990
Series: IEEE Conference on Computer Vision and Pattern Recognition
Publisher: IEEE
Number of pages: 18
Grant note: AIM1893589 - CUP E64118002540007 / MIUR AIM -Attrazione e MobilitaInternazionale Linea 1 Mike Shou's Start-Up Grant 2033413 / Innovation and Technology Ecosystems; Dir for Tech, Innovation, & Partnerships; National Science Foundation (NSF); NSF - Directorate for Technology, Innovation and Partnerships (TIP); NSF - Innovation & Technology Ecosystems (ITE) KAUST Office of Sponsored Research through the Visual Computing Center (VCC) R01MH114999 / NIH; United States Department of Health & Human Services; National Institutes of Health (NIH) - USA EP/T004991/1 / UKRIEngineering and Physical Sciences Research Council (EPSRC) Doctoral Training Program (DTP), EPSRC Fellowship UMPIRE; UK Research & Innovation (UKRI); Engineering & Physical Sciences Research Council (EPSRC) 2033413 / NSF; National Science Foundation (NSF)
Identifiers: 9942417708331
Academic Unit: King Abdullah University of Science & Technology
Language: English
Resource Type: Conference proceeding