Enhancing Assistive Robotics with Simplified Shared Control via Imitation Learning
Abstract
Autonomous manipulation systems have achieved remarkable capabilities, yet integrating human expertise with diffusion-based policies in shared control remains relatively unexplored. Additionally, finding efficient combinations of human input with autonomous policies remains an unsolved problem. Physically impaired individuals demonstrate a strong preference for maintaining control authority and personal agency in their daily lives. To strike this balance, this thesis proposes a novel diffusion-assisted shared control approach that reduces mental workload while maintaining an intuitive control interface and user autonomy.
We present two complementary contributions. First, Human-In-The-Loop Diffusion (HITL-D) is a shared control framework dividing the 6 degrees of Cartesian control into two distinct components. Users control end effector position via a joystick while a diffusion policy autonomously determines orientation based on scene point clouds and Cartesian position. In a multi-task user study with 12 participants, HITL-D reduced average task completion times by 40\%, decreased perceived workload by 37\%, and improved Likert-scale ratings for independence, intuitiveness, and confidence compared to traditional teleoperation methods.
Second, we address the generalization limitations of shared control systems by proposing Human-In-The-Loop Highly Generalizable Diffusion (HITL-HGD), which leverages privileged spatial information via prompted image segmentation to condition the diffusion policy. Using alignment constraints and object centroids, we create an autonomous system that encodes implicit rules about the world during inference. HITL-HGD achieves up to 300\% improvement in generalization compared to HITL-D, reaching 100\% success with as few as 7 demonstrations. Additionally, we demonstrate the ability to switch object instances under a single policy or task context while maintaining high success rates.
Together, these contributions demonstrate that shared control fundamentally enables practical assistive robotics by combining human strengths in task reasoning with autonomous capabilities in fine manipulation, reducing operator workload while preserving independence and enabling real-world deployment.
Methodology Overview
This thesis presents a shared control framework that combines human expertise with autonomous diffusion-based policies to enable practical assistive robotics. The framework is built on a key principle: humans maintain control of end effector position via joystick input, while a diffusion policy autonomously determines end effector orientation based on scene observations. Both contributions—HITL-D and HITL-HGD—follow this architectural paradigm but differ in their approach to generalization and scene understanding.
HITL-D (Chapter 3) introduces the foundational shared control architecture with simplicity and reactivity at its core. The diffusion policy predicts orientation in a single step, conditioned on hand-cropped point clouds and end effector position. This reactive approach eliminates the need for future prediction while keeping the human in direct control of position, maximizing user safety and agency. HITL-HGD (Chapter 4) builds on this foundation by introducing explicit geometric understanding through automated spatial constraints. Using SAM-3 for automatic object segmentation, HITL-HGD replaces hand-defined bounding boxes with learned centroid relationships, achieving 300% improvement in generalization while reducing data requirements to as few as 7 demonstrations.
Key Methodology Components:
- Shared Control Architecture: Human controls end effector position; diffusion policy predicts orientation
- Visual Perception: Point cloud processing from scene observations with spatial constraints
- Diffusion Policy: Convolutional diffusion policies with DDIM scheduling for orientation prediction
- Spatial Reasoning: Point cloud segmentation and centroid-based constraints for explicit geometric understanding
- Control Integration: Proportional control with velocity capping and deadband thresholds
- Data Efficiency: HGD variant achieves success with minimal demonstrations through better spatial conditioning