Overview

This is our entry for the LeHome Challenge 2026, a competition for robotic garment manipulation. The goal is a single policy that takes a piece of clothing from an arbitrary initial state, crumpled, twisted, or flat, and manipulates it into a correctly folded state using a pair of bimanual arms. Hardware is the LeRobot SO-ARM101, and submissions are scored by a mesh-keypoint pairwise-proximity metric against a target fold.

This was a team project: Conor Hayes, Andnet DeBoer, and myself, run as a Northwestern CS396 final project and robotics research project.

Our best policy, SmolVLA, folding a garment in the LeHome Isaac Sim environment

System Overview

Our pipeline has three stages:

1. Data collection: teleoperation in simulation using the SO-101 arms, plus procedural trajectory generation to scale up demonstrations.
2. Pre-training + policy training: we trained three policy families, SmolVLA, Diffusion Policy, and ACT, under domain randomization.
3. Simulation evaluation: SO-101 in Isaac Sim, scored against the competition’s keypoint metric.

Augmenting the Data with NVIDIA Cosmos

Collecting real teleoperation episodes by hand is slow, so we used NVIDIA Cosmos to grow our dataset. Cosmos takes one of our clips and re-renders it in a completely different setting while keeping the robot’s motion exactly the same. From a single demo we can spin up dozens of variants with new backgrounds, lighting, colors, and textures. All that variety is what teaches the policy to handle scenes it never actually saw while training.

An example of our NVIDIA Cosmos augmentation. The same manipulation, re-rendered with a different background and lighting.

Comparing the Policies

We trained and tested three imitation-learning policies: SmolVLA, ACT, and Diffusion Policy. SmolVLA, shown in the overview above, and ACT both folded the garment cleanly and did it the same way every time. Diffusion Policy struggled. It usually bunched or dragged the cloth instead of folding it, and once the fabric ended up in an odd shape it almost never recovered.

ACT folds cleanly and decisively.

Diffusion Policy often leaves the garment bunched instead of folded.

So why the gap? ACT plans a short burst of motion at a time, which fits the smooth, repetitive rhythm of folding really well. SmolVLA leans on a big pre-trained vision-language model, so it handles new starting positions with ease. Diffusion Policy builds each move through many small refinement steps, and that process gets fragile when the cloth is partly hidden or hard to read. A small mistake early on snowballs into the messy folds you see here.

Results

Submissions are scored with a mesh-keypoint metric. We place a set of check points on the garment, and a fold passes a check when each point lands close enough to where it should be. The more points that match their targets, the higher the score.

How the scoring works. Each check point on the garment has to land close to its target position.

Across our evaluation episodes, the three policies landed right where the folds above suggested:

SmolVLA came out on top, ACT was close behind, and Diffusion Policy trailed well back. That lines up with what the clips show.