S2M-Trek: From Single to Multi-Sphere Transport via Per-Frame Deep Sets on a Wheel-Legged Robot
1School of Mechanical Science and Engineering, Huazhong University of Science and Technology
2School of Mathematics, Harbin Institute of Technology
†skelon_chan@hust.edu.cn
· *liyiqun@hust.edu.cn Corresponding author
Abstract
Conventional object transport typically relies on containers, fences, or tethers to secure the payload. Here, a wheel-legged robot transports up to five identical, free-rolling steel spheres on an open back plate without physical restraints. Because the spheres are indistinguishable, their observation-slot assignments may change independently across history frames, creating a per-frame permutation symmetry that standard history-concatenation set encoders do not enforce. This mismatch becomes a bottleneck beyond two spheres: under the same PPO budget, flat and branch-wise MLPs plateau at or below two spheres, while a history-concatenation Deep Sets baseline (DSHC) also stalls unless slot assignments are randomized during training. We introduce Per-Frame Deep Sets (PFDS), which pools objects invariantly within each frame before temporal fusion. PFDS is Gframe-invariant by construction, and we prove universal approximation of continuous Gframe-invariant policies. In a 2×2 ablation over architecture and slot randomization, PFDS reaches five spheres across all five seeds and achieves 100% no-drop transport, whereas DSHC relies on augmentation for reliable curriculum progression. We also distill the privileged PFDS teacher into TactSet, which observes only a Gframe-invariant binary union contact map and retains 75% no-drop performance with five spheres. Finally, we run the S2M-Trek deployment stack closed-loop on a physical wheel-legged robot for every sphere count k=1,…,5. The measured tactile maps contain one connected component per sphere but no persistent identity, complementing the quantitative simulation results.
Main Contributions
- A symmetry-mismatch diagnosis. We formalize multi-sphere transport as a Gframe-invariant policy learning problem and derive the quotient ratio (N!)H−1— about 1.7×106 for N=5 spheres and H=4 history frames — as a quantitative measure of the non-physical redundancy that standard history-concatenation encoders fail to collapse.
- The PFDS encoder. A minimal modification of Deep Sets that pools within each history frame before temporal readout, achieving intrinsic per-frame permutation invariance. We back this with formal proofs of Gframe-invariance and universal approximation.
- Architecture vs. augmentation, untangled. An eight-encoder comparison plus a 2×2 ablation (encoder × slot randomization) cleanly separates the architectural pathway from the data-augmentation pathway to multi-sphere robustness. Five-seed runs confirm PFDS reliability.
- TactSet and physical validation. We distill the privileged PFDS teacher into a student using a 16×16 binary union tactile map. TactSet retains 75% no-drop transport with five spheres under the original simulator assets and 85% in the matched evaluation harness. We also run the same deployment stack closed-loop on the physical robot for k=1,…,5; its tactile maps show one connected component per sphere without persistent identity.
Our Method in a Nutshell
The problem. When a quadruped carries multiple identical free-rolling spheres, the observation tensor X ∈ ℝH×N×d indexes balls by an arbitrary slot id at each of H history frames. Because the balls are identical, swapping two slot assignments in any single frame produces a physically equivalent state — so the correct symmetry group is the per-frame product Gframe = (SN)H. Standard Deep Sets applied to history-concatenated tokens only respect the much smaller diagonal subgroup Gdiag, leaving (N!)H−1-fold non-physical variation that the policy must learn to ignore from scratch.
PFDS (Per-Frame Deep Sets). We change where the pooling happens, not how big the network is. A shared MLP φ embeds each per-ball feature, masked sum pools within each frame t to a frame embedding ht, and a temporal MLP ρ reads out [h1; …; hH]. This single re-ordering of operations makes the encoder intrinsically invariant to independent per-frame slot permutations, collapsing the entire Gframe orbit to one representation.
Training & distillation. The PFDS teacher is trained in Isaac Lab with PPO under a multi-criterion curriculum that gradually increases the sphere count from 1 to 5. We then distill the teacher into TactSet via DAgger, replacing the privileged ball state with a 16×16 Boolean union contact map — the same physical contacts visible to a real tactile array. Because the Boolean union is symmetric in its arguments, the tactile observation is Gframe-invariant by construction, semantically aligning the student input with the teacher's quotient representation.
Per-Frame Deep Sets (PFDS) Encoder
The core idea. Per-object observations xt,i are pooled independently within each history frame t to produce frame embeddings ht; these are then concatenated and processed by the temporal readout MLP ρ. Because pooling occurs within each frame, the encoder is intrinsically invariant to independent per-frame permutations (Gframe-invariant) — collapsing the (N!)H−1 non-physical redundancy left by standard history-concatenation encoders.
System Architecture
Top: privileged teacher training. The PFDS teacher is trained in Isaac Lab with PPO using privileged object-state observations (position, linear/angular velocity).
Bottom: tactile-student distillation. TactSet is distilled via DAgger, replacing the privileged observation with a 16×16 Boolean union tactile contact map.
Multi-Sphere Transport Demos (Simulation)
A single PFDS teacher policy transports up to five identical free-rolling spheres simultaneously on the unfenced back plate of the S3 wheel-legged robot. Each video shows the same policy evaluated in Isaac Lab with a different sphere count, played at real-time speed.
Real-Robot Rollouts with the 16×16 Tactile Array
Hardware recordings on the physical S3 wheel-legged robot carrying one to five free-rolling steel spheres on the unfenced tactile plate. Each clip is a synchronized four-panel view of the same rollout, and all five sphere-count clips are played back at 3× speed (the on-screen clock and timeline still show real elapsed seconds).
Panel layout. Top left: overhead camera view of the robot and spheres. Top right: live 16×16 Boolean contact map (5 g threshold) with the active taxel list. Bottom left: number of active cells over the full rollout, with the playhead marking the current instant. Bottom right: body-pitch indicator (tilt drawn ×5).
Quadruped Baseline: Five Spheres Held in Place
The same capture setup on the quadruped platform: the robot holds its position in the original legged stance while carrying five spheres on the tactile plate. All five stay on the plate for the whole clip; the active-taxel count fluctuates as the spheres shift under the stepping gait. Shown at real-time speed rather than 3×.
Comparison-Method Failure Cases
Hardware rollouts of the comparison methods on the same quadruped platform and the same five-sphere payload. In both runs a sphere rolls off the plate mid-transport, and the contact map drops from five active taxels to four.
Encoder Comparison
Eight-encoder comparison. Flat MLP, Branch-MLP, history-concatenation Deep Sets (DSHC), and PFDS differ in how they impose permutation structure across history. Only PFDS achieves intrinsic Gframe-invariance.
Beyond the Training Range: Zero-Shot k = 6, 7, 8
The curriculum stops at five spheres, but the policies also extrapolate zero-shot to six, seven, and eight. The PFDS teacher retains every sphere in 98% / 95% / 93% of episodes, while the TactSet student reaches 80% / 59% / 33% full-retention rates, respectively.
The table also reports strict success, which adds motion and support-margin requirements to full-horizon payload retention. The widening gap at larger counts shows that retaining every sphere and maintaining the stricter transport criteria measure different aspects of performance.
| Spheres (zero-shot) | PFDS teacher | TactSet student | ||
|---|---|---|---|---|
| no-drop | strict | no-drop | strict | |
| k = 6 | 98% | 54% | 80% | 26% |
| k = 7 | 95% | 3% | 59% | 5% |
| k = 8 | 93% | 1% | 33% | 0% |
Evaluations use 100 episodes per model and sphere count in MuJoCo. As in the paper, this is the same robot equipped with twice the tactile sensing area: two 16×16 panels represented as a 32×16 map. Counts k=6, 7, and 8 are zero-shot and were not used during training.
Results
A summary of the four headline experiments. Full numbers, ablations, and confidence intervals are in the paper.
BibTeX
@article{chen2026s2m,
title={S2M-Trek: From Single to Multi-Sphere Transport via Per-Frame Deep Sets on a Wheel-Legged Robot},
author={Chen, Zong and Li, Xuebin and Xiao, Jinpeng and Li, Shaoyang and Liu, Ben and Li, Min and Yin, Zhouping and Li, Yiqun},
journal={arXiv preprint arXiv:2606.01332},
year={2026}
}