FIG. 06 / IN-CONTEXT ROBOT LEARNING

One-Shot Skill DIY with GLOW

Task-centric spatiotemporal encoding turns one demonstration into reusable skill context. GLOW conditions action generation on this context and the current scene.

01 / DIY DEMONSTRATION

Observe Once

A continuous visual experience

A frame from the human demonstration with the black kettle and plant — Black kettle human demonstration

Another frame from the same human demonstration — Black kettle human demonstration

Physical Understanding

Objects & scene geometry

Physical interactions

Temporal state changes

D = (I₁, …, I_T)Visual sequence

One demonstration defines the task

Understand objects, interactions, and state changes jointly across frames.

02 / SKILL ENCODING

Reusable Skill Context

From a DIY demonstration

Task-Centric Spatiotemporal EncodingEncode physical interactions across time into reusable task context.

Reusable skill context

C_D

Task intent·Physical interactions·Language alignment

C_D=\Phi_\theta(D)

Task knowledge that transfers

Reuse task intent and interaction knowledge as objects and scene conditions change.

03 / CONTEXT CONDITIONING

Context Conditioning

Ground the skill in the current scene

REUSABLE TASK KNOWLEDGEC_D

Reuse the demonstrated task intent;
combine it with current execution conditions.

LANGUAGE L / EDITABLE

“Aim at the plant, then pour slowly.”

Refine goals or execution style through language.

The green watering can and target plant in the current scene — Current view O_t: green watering can

Robot state s_tHistory H_t

C_t=(O_t,L,s_t,H_t,C_D)

Reusable skill context + current scene ↓ one GLOW model

04 / IN-CONTEXT LEARNING

GLOW

Unified multimodal autoregressive model

ICL: reuse task knowledge, adapt actions to the scene

CONDITIONING PREFIX / C_t

Observation O_tLanguage LState s_tHistory H_tSkill context C_D

Autoregressive reasoning & action

CoT
Reasoning

→

Action
Token

→

…

CoT-aligned task intent guides scene-dependent action.

p_\theta(Y\mid C_t)=\prod_{i=1}^{N}p_\theta(y_i\mid C_t,y_{<i})

CONTEXT-CONDITIONED SKILL EXECUTION

a_t\sim\pi_\theta(\cdot\mid C_t)

Generate scene-dependent actions from reusable task context.

TRAINING-FREE ADAPTATION

\theta_{\mathrm{test}}=\theta_{\mathrm{base}}

Adapt to the task through context; keep model weights fixed.

05 / SCENE-ADAPTIVE ACTION

Execute & Generalize

Task-consistent, scene-adaptive actions

The robot pours water onto the plant with a green watering can — Green watering can robot execution

The robot returns the green watering can upright and places it down — Adapt execution using current feedback.

Object Generalization

Black kettle → Green watering can
Task intent transfers across objects.

Actions generated for the current scene
No trajectory replay

Closed-loop feedback → Update C_t+1New observation O_t+1 · Robot state s_t+1 · History H_t+1Reuse skill context C_D; update current execution conditions for the next decision.

GENERALIZATION THROUGH CONTEXT

Task knowledge is reused; actions are generated for the current scene.

Same Skill Context

C_D

Reusable task intentPhysical interaction knowledge

Different Objects

O_t

Black kettleDemonstration object

Green watering canExecution object

Current scene + language requirements

Adapted Actions

a_t

a_ta_t+1a_t+2a_t+3…

Scene-dependent action generation · Task intent carried into execution

In-context adaptation · Fixed model weights