Evaluating Robot Policies in a World Model

Abstract

Robotics has broad applications from automating house chores to taking care of patients. However, evaluating robot control policies is challenging, as real-world testing is expensive, while handcrafted simulations often fail to accurately reflect real-world conditions, resulting in poor correlation between simulated evaluation and real-world outcomes. In this work, we investigate World-model-based Policy Evaluation (WPE). We first train an action-conditioned video generation model as a proxy to real-world environments. To enable efficient rollouts of hundreds of interactive steps while mitigating error accumulation in the world model, we propose an inference scheme which we call Blockwise-Autoregressive Diffusion Transformer with adjustable context and decoding horizon lengths. To ensure that the world model indeed follows action input, we propose metrics based on the agreement between the ground truth video and generated video conditioned on the same sequence of actions to evaluate the world model. We then use the world model for policy evaluation by performing Monte Carlo rollouts in the world model while employing a vision-language model (VLM) as a reward function. Interestingly, we found that WPE tends to underestimate the policy values for in-distribution actions and overestimate policy values for out-of-distribution actions. Nevertheless, WPE preserves the relative rankings of different policies. In emulating real robot executions, WPE achieves high fidelity in mimicing robot arm movements as in real videos, while emulating highly realistic object interaction remains challenging. Despite this limitation, we show that a world model can serve as a starting point for evaluating robot policies before real-world deployment.

Estimated policy value ρ̂(π) for in-distribution policies (mean and standard error across 4 runs) using WPE compared to the true policy value ρ(π) using the ground truth LIBERO simulator, where π is the OpenVLA policy. WPE achieves policy values close to the ground truth in tasks such as LIBERO-Object. WPE generally underestimates the policy value, while preserving the relative performance ordering across tasks.

Evaluated policy value ρ̂(π) for out-of-distribution policies (mean and standard error across 4 runs) aggregated across all tasks in LIBERO. π is the OpenVLA policy with different levels of Gaussian noise. WPE overestimates the policy values across all noise levels, suggesting that evaluating out-of-distribution policies using a world model can be challenging. Nevertheless, WPE preserves the ranking of the out-of-distribution policies.

Citation

If you find this work useful, please cite:

@misc{quevedo2025evaluatingrobotpoliciesworld,
      title={Evaluating Robot Policies in a World Model}, 
      author={Julian Quevedo and Percy Liang and Sherry Yang},
      year={2025},
      eprint={2506.00613},
      archivePrefix={arXiv},
      primaryClass={cs.RO},
      url={https://arxiv.org/abs/2506.00613}, 
}