eVTA0 · RLER arXiv 2026

Demonstration-free reward learning

Demonstration-Free Success-Probability Reward Learning
for Generalist Robot Policies

Duo Wu1 Haifeng Wang1,3 Rongwei Lu1 Jinghe Wang1 Tianyi Xiong1 Zhimin Wang1 Chao Yu1 Shuai Ma2 Zhi Wang1

1Tsinghua University 2Pengcheng Laboratory 3Yuanxing Robotics

Paper Code Model Video BibTeX

Reinforcement learning (RL) enables generalist robot policies to improve through trial-and-error interaction, yet its effectiveness is fundamentally constrained by sparse task rewards. Existing general-purpose reward models typically alleviate this issue by learning task progress from expert demonstrations, but introduce a distribution mismatch with the mixed-quality rollouts encountered during policy optimization, making their estimates unreliable on suboptimal and failed behaviors from which the policy must learn.

We introduce eVTA0, a demonstration-free reward learning paradigm where dense reward feedback can be learned directly from sparse task outcomes and policy experience.

Three approaches to reward feedback for real-world robot learning. Sparse task outcomes give feedback only at the end. Demonstration-trained reward models score policy rollouts from expert examples but can face a mismatch on imperfect behavior. eVTA0 learns per-step success probabilities from mixed-quality policy rollouts and terminal outcomes, and RLER updates it with new experience.

Highlights

  1. 01

    Theoretical foundation. We theoretically reveal that terminal task outcomes implicitly define intermediate reward signals, where rewards can be modeled as conditional task success probabilities and recursively learned through bootstrapping.

  2. 02

    Demonstration-free paradigm. Based on this principle, we establish a demonstration-free reward learning paradigm by developing eVTA0, a general-purpose reward model trained through a simple TD-style bootstrapping recipe to estimate success probabilities from manipulation histories.

  3. 03

    RL with Evolving Rewards (RLER). We further introduce RL with Evolving Rewards (RLER), a closed-loop learning framework that adapts eVTA0 using newly collected policy rollouts, allowing reward feedback to evolve with the changing behavior distribution of the policy.

  4. 04

    Extensive validation. We conduct extensive evaluations on both reward modeling and policy learning. eVTA0 consistently provides high-quality reward feedback and improves policy performance under the same RL training budget. Real-world experiments further show that RLER improves overall success rates by 20%–26% over the initial SFT policies, while achieving 35%–36% improvements under OOD conditions.

01 / Reward quality

Higher Reward Quality

eVTA0 produces reward signals that more reliably track task evolution and distinguish successful from failed trajectories than state-of-the-art reward models.

Quantitative analysis

Benchmark Method Temporal Consistency Outcome Distinction
VOC ↑VROC ↑MSE ↓Kendall τa ↑
LIBEROGVL (Qwen3-VL-4B)0.280.060.510.21
GVL (GPT-5)0.740.480.480.31
TOPReward (Qwen3-VL-4B)0.71−0.580.640.20
VLAC-8B (Fine-tuned)0.530.470.540.23
Robometer-4B (Fine-tuned)0.980.610.100.85
eVTA0 (Qwen3-VL-4B)0.840.770.060.98
MetaWorldGVL (Qwen3-VL-4B)0.18−0.010.420.02
GVL (GPT-5)0.640.350.290.39
TOPReward (Qwen3-VL-4B)0.79−0.810.630.14
VLAC-8B (Fine-tuned)0.420.430.590.42
Robometer-4B (Fine-tuned)0.770.150.130.65
eVTA0 (Qwen3-VL-4B)0.800.440.040.94

Quantitative analysis

Reward predictions of eVTA0, Robometer, and GVL on a successful and a failed rollout of the same bowl-placement task.
Successful rollout

Confidence grows with reliable task progress

eVTA0 remains conservative during early manipulation and sharply increases its predicted success probability once the bowl is reliably moved toward and placed on the plate, eventually approaching one at task completion.

Failed rollout

Confidence fades with emerging failure cues

eVTA0 initially increases its prediction when the robot successfully grasps and lifts the bowl, but decreases after the placement becomes unstable and remains low when the bowl is eventually tipped over. In contrast, Robometer continues to assign high scores even after the failed trajectory deviates from successful completion.

02 / Policy learning

Better Rewards, Better Policies

eVTA0 provides consistently effective reward guidance across diverse settings and enables more sample-efficient policy learning under the same training budget of 6,400 episodes.

Open
Bar charts of GRPO success rates on the four LIBERO suites: eVTA0 reaches 92.6, 99.0, 85.8, and 54.0 percent, ahead of SFT, binary, handcrafted, and Robometer rewards.
Controlled comparison: π0.5 policy; GRPO; 6,400 training episodes; Same RL training configurations except rewards.
Spatial92.6%+13.8 pp
Object99.0%+5.6 pp
Goal85.8%+5.4 pp
Long54.0%+10.8 pp
Average82.9%+8.9pp

A separate look

Where binary rewards stall, eVTA0 keeps climbing.

Training with sparse binary rewards fails to improve and eventually degrades, while eVTA0 rewards support steady improvement.

Learning curvesOpen
PPO learning curves: eVTA0 success keeps improving while binary-reward training degrades; side panel compares estimation quality with the PPO critic.
LIBERO-Spatial; PPO; 1,280 training episodes
03 / Reward Evolution

Evolving Rewards, Better Generalization

As the policy evolves, RLER adapts eVTA0 with newly collected rollouts so that reward feedback stays aligned with the policy's changing behavior distribution — improving real-world success rates by 20–26 pp over initial policy and by 35–36 pp under distribution shifts.

Real-world Evaluation

Evaluation under real-world distribution shifts in object distance, pose, position, and appearance.

TaskInitial SFTRLER
Round 1		RLER
Round 2		Fixed eVTA0Gain over SFT
Pick CarrotDynamic Environment 32.0%40.0%68.0%48.0%+36 pp
Put ShuttlecockPrecision Control 45.0%60.0%80.0%65.0%+35 pp

Evaluated under unseen object configurations (25 trials for Pick Carrot and 20 for Put Shuttlecock), RLER Round 2 consistently outperforms the fixed eVTA0 baseline trained with the same rollout budget, achieving +15–20 pp improvement under OOD scenarios.

From Failure to Robust Execution

Previously unsuccessful condition

Put the shuttlecock at an unseen position

The object is moved to an unseen position where the initial policy fails in every trial. Through iterative reward evolution, RLER gradually adapts its behavior to the new configuration and recovers successful execution.

  1. Initial SFT0%
  2. RLER · Round 130%
  3. RLER · Round 270%
  4. Fixed eVTA040%

Real-world Rollouts

Initial SFT policyFailed rollout
eVTA0-RLERSuccessful rollout

Put the shuttlecock from the table onto the shuttlecock in the plate.

04 / MethodeVTA0 · RLER

Theoretical Foundation

The reward at every step r at time t equals the probability of eventual success S equals 1, given script F at time t, the information revealed by the trajectory prefix

Reward is defined as the success probability conditioned on ℱt, the information revealed by the trajectory history at timestep t.

Open
Illustration of the theoretical foundation: sparse terminal outcomes implicitly define dense success-probability rewards that can be recursively learned through bootstrapping.

eVTA0 Design

Open
A vision-language model reads the task instruction and a window of recent observations, predicts success probability through a reward head, and learns from TD-style bootstrapping with a target network.
Overview of eVTA0. The left panel illustrates the model architecture, which estimates the probability of eventual task success conditioned on the reward-prediction instruction, manipulation task description, recent observation history and special reward token. The right panel illustrates the training procedure, where eVTA0 is trained on mixed-quality policy rollouts through temporal-difference (TD) style bootstrapping.

Integration into RL. The predicted success probability is transformed into a dense reward signal as r superscript eVTA subscript t equals p-hat subscript t minus 1. This linear transformation preserves the ordering and pairwise differences of reward estimates while avoiding length bias that may favor unnecessarily long rollouts.

RL with Evolving Rewards (RLER)

Enabled by the demonstration-free learning ability of eVTA0, we introduce RL with Evolving Rewards (RLER), where newly collected policy rollouts are used to update eVTA0, and eVTA0 in turn provides adapted rewards for subsequent policy optimization. In this way, reward learning can evolve with the changing behavior distribution of the policy.

Open
Diagram contrasting a fixed eVTA0 reward with the RLER loop, where rollouts update eVTA0 before each round of policy optimization.
Illustration of RL training with a fixed eVTA0 (left) and the proposed RL with Evolving Rewards (RLER) framework (right). RLER adapts eVTA0 using newly collected policy rollouts, enabling reward learning and policy optimization to evolve together.

Video

Title slide of the eVTA0 English talk video. Recorded Video

eVTA0 Talk

Toward General Embodied Reward System — Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies

Watch the video

Toward general embodied reward system

Let the robot’s own experience
discover what success looks like.

Inspired by the brain’s reward system, eVTA (embodied ventral tegmental area) aims to build a general reward mechanism for embodied intelligence — one that learns from experience, adapts with policy evolution, and scales across tasks. eVTA0 takes the first step toward this vision by learning success-probability signals directly from the robot’s own experience.

Citation

Download .bib
arXiv:2609.33653 · 2026
@article{wu2026evta0,
            title={Demonstration-Free Success-Probability Reward Learning for Generalist Robot Policies},
            author={Wu, Duo and Wang, Haifeng and Lu, Rongwei and Wang, Jinghe and Xiong, Tianyi and Wang, Zhimin and Yu, Chao and Ma, Shuai and Wang, Zhi},
            journal={arXiv preprint arXiv:2609.33653},
            year={2026}
          }

Open original ↗