Abstract
Autonomous manipulation systems have achieved remarkable capabilities, yet the integration of human expertise with diffusion-based policies in shared control remains relatively unexplored. In this paper, we propose Human-In-The-Loop Diffusion (HITL-D), a shared control framework that enhances user performance in multi-step, insertion, and fine manipulation tasks. HITL-D leverages a novel combination of diffusion-based policies and human control to provide autonomous end effector orientation updates conditioned on a scene point cloud and the Cartesian position of the end effector. This approach reduces the number of joystick control axes required, thereby lowering mental workload. In a multi-task user study with 12 participants, HITL-D reduced average task completion times by 40%, decreased perceived workload by 37%, and improved Likert-scale ratings for independence, intuitiveness, and confidence compared to traditional teleoperation methods. These results demonstrate that HITL-D effectively integrates human expertise with autonomous assistance, improving both objective and subjective aspects of teleoperation.
Methodology
HITL-D implements a shared control framework where the human operator maintains direct control of end effector position through joystick input, while a diffusion policy autonomously predicts end effector orientation. The key design philosophy emphasizes reactivity over prediction—the policy operates on a single-step horizon, observing the current scene and providing immediate orientation guidance rather than planning multiple steps ahead. This reactive approach maximizes user safety and maintains intuitive control, as humans retain position authority throughout the interaction.
Key Design Principles:
- Reactivity: Single-step (horizon=1) prediction for immediate responsiveness
- User Safety: Human maintains position control; robot only adjusts orientation
- Simplicity: Point cloud conditioning with hand-cropped bounding boxes
- Proportional Control: Orientation error converted to angular velocity with deadband thresholds
Architecture Components:
- Visual Input: Colored point cloud (2048 points after farthest point sampling)
- State Input: End effector 3D position relative to robot base
- Action Output: 3D orientation (radians)
- Encoding: 3-layer MLP encoding → 128-dimensional latent vector
- Policy: Convolutional diffusion policy with DDIM scheduler
- Training: 3000 epochs with observation window of 1
Poster
BibTeX
@article{zilka2026hitl,
title={HITL-D: Human In The Loop Diffusion Assisted Shared Control},
author={Zilka, Riley and Khlynovskiy, Sergey and Wang, Allie and Jagersand, Martin},
journal={arXiv preprint arXiv:2605.21460},
year={2026}
}