Embodied First-Person Vision
Models that interpret intention, social interaction, and sensorimotor behavior from egocentric video.
I work on computer vision, machine learning, and broader questions in dynamic intelligence. My research has studied image segmentation, grouping, object recognition, human behavior analysis, first-person vision, and computational models that connect perception with action and memory.
I received my B.A. in Computer Science and Mathematics from Cornell University in 1994 and my Ph.D. in Computer Science from UC Berkeley in 1998. Before joining Penn, I was a research faculty member at the Robotics Institute at Carnegie Mellon University.
- Dynamic intelligence: visual memory, learning, and systems that reason from visual experience.
- Perception: normalized cuts, spectral grouping, segmentation, object recognition, and visual organization.
- People: human activity, body analysis, social interactions, and first-person visual understanding.
Research Areas
Image Segmentation and Grouping
Graph-based visual grouping, normalized cuts, contour organization, and spectral methods.
Human Recognition and Activity
Algorithms for understanding human motion, activity, identity, gesture, and behavior in video.
Open-source Packages
ncut-pytorch
Nyström Normalized Cuts PyTorch
Normalized Cut with Nyström approximation, run on million-scale graph in milliseconds. O(n) time complexity, O(1) space complexity.
Selected Projects and Publications
Artifacts and attention sinks: Structured approximations for efficient vision transformers
arXiv
Object Detection in Video with Spatiotemporal Sampling Networks
European Conference on Computer Vision, 2018
Egocentric Basketball Motion Planning from a Single Image
Computer Vision and Pattern Recognition, 2018
Unsupervised Learning of Important Objects from First-Person Videos
International Conference on Computer Vision, 2017
Normalized Cuts
EgoNet
Random Walks
Convolutional Random Walk Networks for Semantic Image Segmentation
Computer Vision and Pattern Recognition, 2017
Grouping
Segmentation Given Partial Grouping Constraints
IEEE Transactions on Pattern Analysis and Machine Intelligence, 2004
Resources
Current Ph.D. Students
Huzheng Yang
Current Ph.D. student
Katherine Xu
Current Ph.D. student
Andrew Lu
Current Ph.D. student
Postdoctoral Researchers
Hyun Soo Park
Associate Professor, McKnight Presidential Fellow, University of Minnesota
Ph.D. Dissertation/ Thesis Supervised
-
Stella Yu, Professor, University of Michigan; Adjunct Professor, UC Berkeley
"Computational Models of Perceptual Organization", Carnegie Mellon University, May 2003.
-
Abhinav Gupta, Professor, Carnegie Mellon University
"Beyond Nouns and Verbs", 2009, U. Maryland. Co-advisor with Larry Davis.
- Qihui Zhu, Nine Chapters Capital Management "Shape Detection by Packing Contours and Regions", December 2010.
- Alexander Toshev, Google "Shape Representations for Object Recognition", December 2010. Co-advisor with K. Daniilidis, B. Taskar, Google.
- Praveen Srinivasan, TLM, Perception at Aurora "Holistic Shape-Based Object Recognition Using Bottom-up Image Structures", February 2011.
- Elena Bernardis, Assistant Professor of Medical School, UPenn "Finding Dots in Microscopic Images", April 2011. Co-advisor with S. Yu.
- Song Gang, Google "Quantitative Analysis of Thoracic Computed Tomography Images", May 2013. Co-advisor with J. Gee.
-
Katerina Fragkiadaki, Associate Professor, Carnegie Mellon University
"Multi-granularity Representations for Human Interactions: Pose, Motion and Intention", September 2013.
- Jeff Byrne, STR "Shape Representations using Nested Descriptors", April 2014.
- Weiyu Zhang, Apple "Visual Recognition via Matching Discriminative Deformable Patterns", Proposal May 2014.
-
Gedas Bertasius, Assistant Professor, UNC
"Embodied Visual Perception Models for Human Behavior", 2019.
- Jyh-Jing Hwang, Research Scientist at Waymo Research "Learning Image Segmentation with Relation-Centric Loss and Representation", 2020.
- Lingzhi Zhang, Research Scientist at Adobe "Bridging Visual Generation and Recognition", 2023.
Video
