Box Storage: Adapting to New Objects and Positions

Box stor­age con­sists of open­ing the lid, plac­ing objects inside, and clos­ing the lid. Three sets of exam­ples pair human demon­stra­tions with robot exe­cu­tion. GLOW locates the lid, objects, and box inte­rior from cur­rent obser­va­tions, then com­pletes stor­age in the demon­strated order.

Box Storage · Reproduction at Standard and Unusual Orientations

01 / TEACH

Demonstration · Standard orientation

02 / REPRODUCE

Robot execution · Standard orientation

03 / TEACH

Demonstration · Unusual orientation

04 / REPRODUCE

Robot execution · Unusual orientation

Box Storage · Reproducing Operations With the Same Object

01 / TEACH

Demonstration · Same object

02 / REPRODUCE

Robot execution · Same object

Box Storage · Transferring Actions Across Objects

01 / TEACH

Demonstration · Original object

02 / REPRODUCE

Robot execution · New object and position

Teach­ing estab­lishes inter­ac­tion goals and oper­a­tion order, while real-time obser­va­tions deter­mine exe­cu­tion posi­tions and end-effector ori­en­ta­tion. The state after open­ing the lid informs object-placement plan­ning, and the place­ment result enables lid clo­sure. GLOW pre­serves task struc­ture while using on-site feed­back to com­plete each stage.

Watering Transfer: From Action Reproduction to Tool Adaptation

Two sets of water­ing exam­ples show action repro­duc­tion with the same tool and trans­fer between dif­fer­ent tools. The first uses a green water­ing can for both demon­stra­tion and exe­cu­tion; the second uses a black narrow-spout kettle in the demon­stra­tion and a green water­ing can during robot exe­cu­tion. GLOW maps demon­strated grasp­ing, ori­en­ta­tion, and tilt­ing rela­tion­ships to the cur­rent tool, then grasps it, moves toward the plant, tilts, returns upright, and puts it back.

Watering · Reproduction With the Same Type of Tool

01 / TEACH

Demonstration · Green watering can

02 / REPRODUCE

Robot execution · Same tool type

Watering · Action Transfer Across Tools

01 / TEACH

Demonstration · Black kettle

02 / REPRODUCE

Robot execution · New tool and position

GLOW reuses manip­u­la­tion rela­tion­ships from the demon­stra­tion and gen­er­ates approach posi­tions and end-effector ori­en­ta­tions for the cur­rent tool. Obser­va­tions after each action update tool and target states, allow­ing sub­se­quent actions to adapt to changes in the scene.

Table Wiping: The Emergence of Personalized Actions

In the wiping demon­stra­tion, a person moves a cloth along a heart-shaped path. After learn­ing, the robot repro­duces the heart-shaped wiping motion. This exam­ple illus­trates GLOW’s learn­ing of a per­son­al­ized manip­u­la­tion style: the demon­stra­tion con­veys both the goal of wiping the table and the motion char­ac­ter­is­tics of how to wipe it.

01 / TEACH

Demonstration · Heart-shaped wiping

02 / REPRODUCE

Robot execution · Personalized motion

The dis­tinc­tive heart-shaped motion is pre­served from demon­stra­tion to repro­duc­tion, illus­trat­ing how demon­stra­tions can convey per­son­al­ized actions. A user can express a manip­u­la­tion pref­er­ence through one demon­stra­tion, allow­ing the robot to reflect that style while com­plet­ing the task.

Drink Mixing: Sequence Understanding and Continuous Execution With Multiple Glasses

In a scene con­tain­ing mul­ti­ple glasses, GLOW iden­ti­fies the object of each step, pour­ing rela­tion­ships, and oper­a­tion order from the demon­stra­tion. It then selects the cor­re­spond­ing glass from cur­rent obser­va­tions and per­forms the grasp, pour, return upright, and replace­ment steps in sequence, con­nect­ing mul­ti­ple oper­a­tions into a com­plete drink-mixing pro­ce­dure.

01 / TEACH

Demonstration · Multi-glass mixing sequence

02 / REPRODUCE

Robot execution · Continuous multi-step actions

The demon­stra­tion pro­vides oper­a­tion order, while on-site obser­va­tions iden­tify cur­rent objects and task progress. GLOW turns mixing sequences from dif­fer­ent demon­stra­tions into con­tin­u­ous action, main­tain­ing con­ti­nu­ity across mul­ti­ple objects and steps and demon­strat­ing under­stand­ing and reuse of task struc­ture.

Each demon­stra­tion video is paired with its cor­re­spond­ing exe­cu­tion. Simul­ta­ne­ous play­back pre­serves each video’s orig­i­nal dura­tion; action stages are not forced into syn­chro­niza­tion.