Dobb-E for Web

Dobb-E — Practical Imitation Learning for Home Robots

Dobb-E is an open-source system that improves the real-world abilities of home robots by using imitation learning. It tackles common shortcomings in domestic robotics by simplifying how demonstrations are gathered and how robots learn from them, enabling robots to generalize to new tasks and unfamiliar environments.

Low-cost demonstration hardware

A key part of the system is an inexpensive, easy-to-use demonstration tool called the Stick. It’s built from off-the-shelf and custom parts to make data collection accessible to non-experts.

• iPhone used for recording demonstrations and capturing visual streams
• 3D-printed connectors and holders to adapt the device to different homes
• an inexpensive $25 reacher/grabber as the primary manipulation aid

The HoNY interaction collection

Dobb-E trains from a collected dataset called Homes of New York (HoNY), which captures realistic human-robot interaction examples across multiple residences.

• Data gathered across 22 distinct apartments and houses
• Roughly 13 hours of recorded interactions in total
• Multiple sensor streams including RGB and depth video
• Action labels paired with demonstrations for supervised signals

Learning representations for new tasks

At the core of the framework is a representation model named Home Pretrained Representations (HPR). HPR uses a ResNet-34 backbone and self-supervised objectives to learn visual features that help robots plan and execute actions in novel contexts.

• ResNet-34 architecture forms the feature extractor
• Self-supervised pretraining to transfer knowledge across environments
• Learned embeddings enable rapid adaptation to unfamiliar tasks

Results and how to access the project

Dobb-E demonstrates strong performance when given a short amount of time to adapt: on average, robots solve about 81% of previously unseen tasks after 15 minutes of setup. The project is openly available so others can reproduce and build on the work.

• Research paper describing methods and experiments is available
• Full documentation to reproduce experiments and use the software
• Source code and model weights hosted for public access on GitHub