Abstract
Shared control systems for robotic manipulation still struggle to generalize beyond their training distribution, limiting their utility in real-world assistive applications. In this paper, we propose Human-In-The-Loop Highly Generalizable Diffusion (HITL-HGD), an improved shared control framework that enhances the limitations of previous research combining autonomous diffusion policies and human input. This approach to conditioning the policy leverages privileged spatial information available via prompted image segmentation. HITL-HGD achieves an average in-distribution success rate of 100% at 10 demonstrations versus 33.3% for the prior method, a 300% improvement and reaches 100% success on two of three tasks with as few as 7 demonstrations. We additionally showcase the ability to completely switch objects under the same pick and place policy while maintaining high success rates despite different execution requirements. These results illustrate the importance of spatial constraints for generalization in robot learning policies.
Methodology
HITL-HGD builds on the shared control framework of HITL-D by introducing explicit geometric understanding through automated spatial constraints. Rather than relying on implicit learning from point clouds alone, HITL-HGD leverages privileged spatial information via automatic object segmentation, dramatically improving generalization to novel objects and scenarios. The key innovation is the integration of SAM-3 (Segment Anything Model 3) for real-time segmentation, which eliminates the need for manual bounding box definitions and enables the policy to learn generalizable spatial relationships between task-relevant objects and the end effector.
Key Improvements Over HITL-D:
- Explicit Spatial Constraints: Centroid-based relationships instead of implicit point cloud patterns
- Automatic Segmentation: SAM-3 eliminates hand-defined bounding boxes for true generalization
- 300% Generalization Improvement: Reaches 100% success at 10 demonstrations vs. 33.3% for HITL-D
- Data Efficiency: Achieves 100% success on two of three tasks with as few as 7 demonstrations
- Object Switching: Ability to switch objects under the same policy while maintaining high success rates
Architecture Components:
- Segmentation Pipeline: SAM-3 with object prompts and sub-prompting for complex objects
- Visual Input: Segmented, colorless point cloud (1024 points after FPS)
- Spatial Features: Centroid differences—direct distances between object centroids and end effector
- State Input: Centroid-based alignment constraints and relative positioning
- Action Output: 6D SO(3) representation for full 3D rotation
- Encoding: 3-layer MLP encoding → 64-dimensional latent vector (more efficient than HITL-D)