Towards Miniature Humanoid Tele-Loco-Manipulation Using Virtual Reality and Reinforcement Learning

Technical Breakdown§

Overview§

The paper introduces a control framework for miniature humanoid robots (specifically ROBOTIS OP3) that combines:

• Upper-body teleoperation via Virtual Reality (VR) headset and hand controllers, mapping human arm motions to robot arm joints.
• Lower-body locomotion using a Reinforcement Learning (RL)-trained policy for walking and balance, independent of arm movements.

Methodology§

RL for Locomotion

The lower-body control is formulated as a Markov Decision Process (MDP) with state space $s_t$ including joint angles, velocities, IMU readings, and base velocity commands from the operator. The action space $a_t$ comprises target joint positions for the legs, executed by a low-level PD controller. The reward function $r_t$ encourages forward velocity, energy efficiency, and stability: $$ r_t = w_v \cdot v_x - w_{\tau} \cdot \|\tau\|^2 - w_{penalty} \cdot \text{fall} $$ where $v_x$ is forward speed, $\tau$ are joint torques, and fall is a binary penalty.

The policy $\pi_{\theta}(a_t|s_t)$ is trained using Proximal Policy Optimization (PPO) in simulation (PyBullet) and then transferred to real hardware with domain randomization (mass, friction, motor gains).

Teleoperation Stack

The operator wears an HTC Vive headset and controllers. Upper-body joint angles are computed via inverse kinematics from the tracked hand and head poses. These are sent as desired positions for the robot's arms. The RL policy runs at 50 Hz on the robot's onboard computer, while VR commands are sent at 90 Hz over a low-latency wireless link.

Implementation Details§

• Robot: ROBOTIS OP3 (20 DOF: 12 legs, 8 arms) with an Intel NUC i7 onboard.
• Simulator: PyBullet with domain randomization.
• Training: PPO with clipping parameter $\epsilon = 0.2$, learning rate $3e-4$, horizon 2048, mini-batch size 64, 10 epochs per update.
• Observation space: 84-dim (joint angles, velocities, IMU angular velocity, linear acceleration, base twist command).
• Action space: 12-dim (target joint angles for 6 leg joints each side).

Code Snippet (PyTorch Policy and PPO Update)§

import torch
import torch.nn as nn
import torch.optim as optim

class ActorCritic(nn.Module):
    def __init__(self, obs_dim, act_dim):
        super().__init__()
        self.actor = nn.Sequential(
            nn.Linear(obs_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, act_dim),
            nn.Tanh()
        )
        self.critic = nn.Sequential(
            nn.Linear(obs_dim, 256), nn.ReLU(),
            nn.Linear(256, 256), nn.ReLU(),
            nn.Linear(256, 1)
        )
    def forward(self, obs):
        return self.actor(obs), self.critic(obs)

# PPO update pseudocode (simplified)
# for epoch in range(ppo_epochs):
#     log_probs, values = model(batch_obs)
#     ratios = torch.exp(log_probs - old_log_probs)
#     advantages = returns - values
#     surr1 = ratios * advantages
#     surr2 = torch.clamp(ratios, 1-eps, 1+eps) * advantages
#     actor_loss = -torch.min(surr1, surr2).mean()
#     critic_loss = 0.5 * (returns - values).pow(2).mean()
#     loss = actor_loss + 0.5 * critic_loss - entropy_coef * entropy
#     optimizer.zero_grad(); loss.backward(); optimizer.step()

Results§

• Walking speed up to 0.45 m/s independent of arm motion.
• Tele-loco-manipulation: human operator relocated two 40g cubes in 10 minutes over a total walking distance of 5m.
• The compliant control stack allows safe physical interaction without explicit force sensing.

Key Takeaways§

• First demonstration of full-body tele-loco-manipulation on a miniature humanoid.
• The modular approach decouples arm teleoperation from learned leg locomotion, enabling intuitive control.
• Limitations: slow walking speed, limited payload, and reliance on external VR infrastructure.