AFFECT: Action Effect Matching
Learning What Actions Change: Action Effect Matching for Dynamics-Faithful World Models
Learning What Actions Change:
Action Effect Matching for Dynamics-Faithful World Models
Preprint TL;DR
AFFECT is an auxiliary training objective that teaches robot world models how small action changes alter the future: generated videos are trained to respond to logged action perturbations the same way recorded videos do, making world models more faithful for policy evaluation and planning.
Motivation
To serve as policy evaluators or planners, robot world models must reflect how small action changes alter future outcomes. A slight change in reaching motion can decide whether a gripper grasps a cube, misses it, or pushes it away. Yet existing models tend to produce visually plausible videos that ignore such differences, predicting similar futures for actions with very different real consequences. We therefore treat action sensitivity as a distributional problem: a world model should capture how the distribution of future observations shifts as actions vary under a given context.
Method
AFFECT asks generated videos to respond to action perturbations the same way recorded videos do, rather than asking each generated video to match its recorded counterpart. This needs interaction data in which actions are perturbed in a known way, and a loss that measures the response to those perturbations.
1. Perturbation play
We train a play policy on task-agnostic human teleoperation and roll it out autonomously, which needs no scene resets or task-specific procedures. Before execution, we inject temporally correlated noise into each nominal action chunk and log the perturbation; the robot executes the perturbed chunk and we record the resulting future. Each window needs just a single recorded future, so no repeated executions from the same initial state are required.
2. Action effect encoding
The world model sees only the context frame and the executed actions, and generates a future from fresh noise with a short differentiable sampler, never conditioned on the recorded future. Two frozen encoders then summarize the videos (φ) and the logged perturbation (ψ) with simple, hand-specified features. After centering the perturbation features by their expected value, the action effect is the product of perturbation and video features: positive when larger-than-expected perturbations come with larger values of a video feature. Keeping the encoders frozen prevents a trivial solution, and lets AFFECT plug into existing world models without architectural changes.
3. Distributional matching
Averaged over many windows, the action effects reveal how the scene responds to actions. AFFECT trains the world model so that the action effects of generated videos match those of recorded videos:
Centering turns each action effect into a conditional covariance between perturbation and video features, so scene-only errors (e.g. rendering a scene slightly too dark) cancel out, and the loss compares only the direction and strength of the response to actions. The full objective is a finite-feature maximum mean discrepancy over action effects. Unlike plain feature matching, it catches a model that moves an object left whenever the real object moves right.
Policy Evaluation Results
We apply AFFECT to two state-of-the-art robot world models, DreamDojo and Cosmos3, and run real policies in closed loop inside each world model. If the world model is faithful, policies that succeed more often on the real robot should also succeed more often in imagination.
Does the world model rank policies like the real robot?
Pearson correlation between policy success rates in the world model and in the real environment. AFFECT improves it in all six backbone–embodiment combinations.
Are the generated videos closer to reality?
Gain in long-horizon PSNR (dB) after adding AFFECT. AFFECT improves it in all six combinations.
Show full results tables (PSNR, LPIPS, FVD, Pearson, and other baselines)
Real robots: DROID (Franka) and ALOHA-2
| Method | DROID (Real) | ALOHA-2 (Real) | ||||||
|---|---|---|---|---|---|---|---|---|
| PSNR ↑ | LPIPS ↓ | FVD ↓ | Pearson ↑ | PSNR ↑ | LPIPS ↓ | FVD ↓ | Pearson ↑ | |
| iVideoGPT | 16.16 | 0.52 | 1946 | 0.151 | 16.77 | 0.52 | 3808 | 0.155 |
| CoCo | 18.13 | 0.50 | 1676 | 0.212 | 19.09 | 0.49 | 3959 | 0.225 |
| Ctrl-World | 17.48 | 0.36 | 1141 | 0.236 | 20.32 | 0.11 | 396 | 0.399 |
| Cosmos3 | 18.69 | 0.24 | 910 | 0.425 | 22.94 | 0.08 | 333 | 0.298 |
| + AFFECT | 19.67 | 0.22 | 744 | 0.583 | 24.05 | 0.05 | 257 | 0.694 |
| DreamDojo | 18.53 | 0.16 | 807 | 0.468 | 24.20 | 0.06 | 183 | 0.325 |
| + AFFECT | 19.04 | 0.14 | 622 | 0.790 | 24.91 | 0.05 | 188 | 0.490 |
Simulation: OGBench UR5e
| Method | Pearson ↑ | Short Horizon | Long Horizon | ||||
|---|---|---|---|---|---|---|---|
| PSNR ↑ | LPIPS ↓ | FVD ↓ | PSNR ↑ | LPIPS ↓ | FVD ↓ | ||
| iVideoGPT | 0.334 | 21.53 | 0.49 | 1134 | 20.20 | 0.53 | 1314 |
| CoCo | 0.130 | 21.69 | 0.49 | 1094 | 20.29 | 0.52 | 1331 |
| Ctrl-World | 0.310 | 24.99 | 0.10 | 156 | 24.27 | 0.11 | 196 |
| Cosmos3 | 0.091 | 19.07 | 0.19 | 470 | 18.72 | 0.22 | 562 |
| + AFFECT | 0.334 | 19.95 | 0.18 | 463 | 19.21 | 0.20 | 540 |
| DreamDojo | 0.451 | 26.03 | 0.06 | 85 | 24.87 | 0.08 | 108 |
| + AFFECT | 0.686 | 26.78 | 0.05 | 73 | 25.58 | 0.07 | 93 |
Planning with Action-Faithful World Models
Five policy checkpoints propose candidate action chunks; the world model imagines a video for each, and a learned success estimator picks the best one. With the proposals and estimator held fixed, AFFECT raises average success from 60.0% to 68.3% on UR5e pick-and-place and from 72.2% to 80.6% on a long-horizon ALOHA-2 task (open a trash bin, then throw away two items).
Show planning results table
| Task | Uniform | DreamDojo | DreamDojo + AFFECT |
|---|---|---|---|
| UR5e · Vertical2 | 33.3 | 25.0 | 33.3 |
| UR5e · Horizontal | 50.0 | 83.3 | 83.3 |
| UR5e · Diagonal1 | 33.3 | 58.3 | 66.7 |
| UR5e · Diagonal2 | 50.0 | 58.3 | 75.0 |
| UR5e · Lateral Near | 66.7 | 75.0 | 83.3 |
| UR5e · Average | 46.7 | 60.0 | 68.3 |
| ALOHA-2 · Subtask 1 | 83.3 | 91.7 | 91.7 |
| ALOHA-2 · Subtask 2 | 58.3 | 66.7 | 83.3 |
| ALOHA-2 · Subtask 3 | 25.0 | 58.3 | 66.7 |
| ALOHA-2 · Average | 55.6 | 72.2 | 80.6 |
Where Do the Features See Action Effects?
On recorded UR5e videos, we measure how consistently perturbation–feature products keep the same sign across episodes at each latent cell, relative to a baseline that pairs perturbations with videos from other windows. Responses are 1.6–2.2× stronger than this shuffled baseline in regions traversed by the arm, cube, or shadow, and stay at chance on the static table and backdrop, so the frozen features provide a localized action-effect signal.
BibTeX
@article{won2026affect,
title={Learning What Actions Change: Action Effect Matching for Dynamics-Faithful World Models},
author={Won, John and Dong, Yonghoon and Shin, Jinwoo},
journal={arXiv preprint arXiv:2611.25980},
year={2026}
}