RecoverTwin: Train-Free Failure Recovery for Dual-Arm Manipulation

This was my undergraduate senior thesis. I wanted to see whether a small supervisor could help a frozen dual-arm policy that stalls at the grasp. It steps in and re-grasps, with no training at all. It helped on a few tasks and hurt on others. This page records what I tried and what I learned.

Qixuan Huang · Advised by Prof. Liangjing Yang

Overview

Imitation-learning and vision-language-action policies for dual-arm manipulation often break when the scene changes. On the RoboTwin 2.0 benchmark, tasks that succeed over 75% of the time in clean scenes drop below 30% in randomized ones. Many of these failures happen at the grasp.

Retraining a policy for every new condition is expensive, so I asked a narrower question: without changing the policy, can an outside module notice a stalled grasp and fix it? RecoverTwin is my attempt. It combines existing pretrained models with a hand-written rule, and nothing in it is trained.

I tested it on RoboTwin 2.0 with two base policies, ACT and RDT-1B. A cell is one task in one scene type, clean or randomized. Each cell runs the same 100 seeds with and without RecoverTwin, so every seed gives a direct before-and-after comparison.

In short, the supervisor helped in two randomized grasping cells and made little difference in most others. It clearly hurt where the base policy already worked or where both arms had to cooperate.

Method

The design is simple. At every step the frozen base policy proposes a 14-D joint action, and a rule-based detector decides whether to interrupt. When it fires, a recovery pathway takes over one arm. It finds the target, samples six-DoF grasps, ranks them, plans a path, and executes for a bounded number of steps before handing control back.

Architecture

A frozen IL or VLA base policy proposes actions. The analytic detector either passes them through or triggers the recovery pathway, which is itself built from frozen modules. An action gate picks the action that is executed.

Failure detector

The detector is a hand-written rule, not a learned classifier. It reads gripper widths, end-effector positions, the cached grasp candidates, a cooldown counter, and whether a recovery is already running. It fires only when four conditions hold: no recovery is in progress, the cooldown has elapsed, grasp candidates exist, and the arm looks like an attempted but unconfirmed grasp. That last condition means the gripper is still open, the hand is near a candidate, and the gripper has not moved toward closing in twenty steps. After firing, an 80-step cooldown prevents immediate re-triggering.

Grasp candidates

Florence-2, an open-vocabulary localizer, is prompted with the task's target description. Its bounding box is back-projected into the head-camera depth to get a 3D target region, and the partial point cloud goes to GraspNet, which returns ranked six-DoF grasps. Candidates outside the reachable workspace are discarded, and those inconsistent with the object's expected grasp axis are demoted. Adapting it to a new task mostly means changing the prompt.

Recovery execution

The top-ranked candidate becomes the target pose. An analytic IK solver is tried first, with an RRT-Connect planner as the fallback for poorly conditioned cases. Each recovery has a 60-step budget. On reaching the pose, the gripper closes and holds briefly to confirm the grasp, and control returns to the base policy. A failed grasp is retried on the next-ranked candidate; if the budget runs out or no candidates are left, the supervisor gives control back.

Base policies

The supervisor talks to both policies only through the 14-D action interface they share. ACT is a CVAE-Transformer imitation policy that predicts action chunks with temporal aggregation. RDT-1B is a 1.2B-parameter diffusion transformer that emits 64-step action chunks. For RDT-1B, the supervisor keeps an action buffer, clears it after a recovery, and loads a cached language embedding in place of the T5-XXL encoder.

Results

Each cell compares the success rate of the base policy alone, SR₀, with that of the base policy plus RecoverTwin, SR₁, on the same 100 seeds. Δ is their difference in percentage points. b counts saved seeds, which fail alone but succeed with RecoverTwin; c counts hurt seeds, which go the other way. Stars mark significance after Holm–Bonferroni correction across all 16 cells: *** for p < 0.001, ** for p < 0.01, * for p < 0.05. I set each cell's scope from its task structure before evaluation: in for single-arm grasp tasks, coord for bimanual coordination, multi for long-horizon multi-step tasks.

ACT: imitation learning, 14 cells

Most cells did not change much. Two randomized grasping cells improved. On shake_bottle, successes rose from 9 to 28, and all 19 changed seeds were saves. On beat_block_hammer they rose from 0 to 6, but six seeds are too few to be significant after correction.

Four cells got significantly worse. On clean grab_roller, where ACT already succeeds, the detector fired on episodes that were about to finish. On the two bimanual tasks, overriding one arm broke the coordination between the arms. On clean place_cans_plasticbox, the detector misfired during the placement step.

A small ablation suggests where the saves came from. On shake_bottle, a recovery that only opens the gripper and retreats adds 2 points, and the GraspNet pose adds the other 17. On beat_block_hammer, the split is 1 point and 5. Most of the benefit seems to come from the grasp pose rather than from simply interrupting the policy.

One recovery, step by step

beat_block_hammer, randomized, seed 100000. Without the supervisor, shown in red, the hand stalls about 0.5 m from the object and the gripper never closes. With it, shown in blue, the detector fires at step 61, the hand moves in and closes, and a second activation at step 140 re-grasps before the strike. The episode succeeds.

RDT-1B: vision-language-action, 2 cells

I only ran RDT-1B on two cells. On beat_block_hammer it improved by 13 points, more than ACT's 6, likely because RDT-1B's 17% baseline leaves more nearly finished attempts for a re-grasp to complete. On adjust_bottle it got 5 points worse; RDT-1B starts at 63% there, against 19% for ACT.

When it helps, when it hurts

Supervisor quality vs. base success rate

Quality q = b / (b + c), the share of changed outcomes that were saves. In-scope cells fall monotonically with base success rate, with Spearman ρ = −0.86 and permutation p = 0.004. Out-of-scope cells sit at or near zero.

Across cells, the supervisor was useful mainly when the base policy was weak. Below about 30% base success, nearly every changed outcome was a save. Around 75%, saves and hurts roughly balanced, and above 90% every change was a hurt. In hindsight this makes sense, since a simple trigger cannot tell a near-success from a near-failure. On tasks outside its scope, it either changed nothing or broke the task.

Looking back

For the time I spent on it, the method is simple and the evaluation is narrow. The main gaps:

• Simulation only. Every result comes from RoboTwin 2.0; I never tested on a real robot.
• A hand-written trigger. The detector uses fixed thresholds, and it causes most of the hurts.
• Single-arm recovery. It cannot help, and often harms, tasks where both arms must coordinate.
• Limited coverage. Fourteen cells on one imitation-learning policy and two on a VLA are not enough for general claims.

If I continued

• Make the trigger task-aware, so it stays quiet on tasks it cannot help.
• Add a two-arm recovery for coordination tasks.
• Test more base policies, such as π0 and DP3.
• Try it on a real bimanual robot.

What I learned

• Pretrained perception and grasp models can recover some grasp failures without any training, but only under narrow conditions.
• When to intervene mattered as much as how to recover. A recovery that helps a weak policy can hurt a strong one.
• Counting saves and hurts per seed, not just comparing success rates, showed me where the method helps and where it does damage.

Senior thesis at the ZJU-UIUC Institute, Zhejiang University, advised by Prof. Liangjing Yang. Built on RoboTwin 2.0, ACT, RDT-1B, Florence-2 and GraspNet.