SeMoCo · A Semantic-First Motion Codec for Motion Language Modeling

01Overview

Motion tokens that know
what they mean.

Most motion tokenizers are trained for reconstruction alone, so action-level meaning and fine-grained kinematics share one distortion-ordered hierarchy. SeMoCo splits the two: a dedicated semantic quantizer is aligned to a frozen TMR-SOMA teacher, while a parallel residual quantizer keeps the detail needed to reconstruct the body.

i

Split quantization

A single semantic VQ runs beside — not inside — the kinematic RVQ chain, so language alignment never reshapes the residual geometry.

1 + 15codes per packet, 12.5 Hz

ii

Dual-axis generation

A temporal Transformer predicts the next semantic code; a lightweight depth decoder completes its kinematic residuals along the codebook axis.

188 / 391 MLite & Base generators

iii

Ω-MotionVerse

Five motion sources unified under the SOMA skeleton at 50 Hz, with provenance retained for source-wise evaluation.

1,006 h909,913 text–motion pairs

02Tokenizer

The semantic-first packet.

A stride-four convolutional encoder turns 50 Hz transition records into 12.5 Hz latents. Two projections split each latent: one is quantized by a single semantic codebook and aligned to the frozen teacher by a window-level cosine loss; the other is consumed by a 15-stage residual quantizer. Both are summed back into one decoder space.

SeMoCo tokenizer architecture: encoder, split semantic VQ and kinematic RVQ, TMR-SOMA distillation, decoder. — **Figure 1 — SeMoCo tokenizer.** The encoder latent is factorized into a semantic stream and a kinematic residual stream. Semantic supervision is applied only during codec training.

m_t packet layout

q^sem: One code from a 1,024-entry semantic codebook, aligned to TMR-SOMA by cosine distance over a 16-packet window.
q^kin,1…15: Fifteen residual codes, each 1,024 entries, ordered by reconstruction error with quantizer dropout p = 0.2.

Placing the constraint on a parallel branch costs 0.40 mm of reconstruction against an unsupervised reference, versus 2.65 mm when it is applied inside the residual chain — while reaching the closest agreement with the teacher (.878 cosine).

03Generator

Two axes, not one long stream.

Flattening sixteen codes per step into a single sequence makes the context sixteen times longer. Instead, the sixteen embeddings are summed into one packet token, so the temporal Transformer still runs at 12.5 Hz.

Temporal axis. Text tokens from a frozen Flan-T5-XL prefix the sequence; the backbone predicts the next packet's semantic code and an EOS target.
Codebook axis. Conditioned on that hidden state, a five-layer depth decoder emits q^kin,1…q^kin,15 autoregressively.
Decode. Completed packets pass through the frozen SeMoCo decoder into full-body SOMA motion.

Temporal layers12 / 24

Hidden width768 / 1,024

Context budget300 packets ≈ 24 s

Dual-axis motion language model: text encoder, temporal Transformer backbone, MTP module and motion decoder. — **Figure 2 — Dual-axis motion language model.** The MTP module completes each packet from its semantic code down to the kinematic residuals.

04Ω-MotionVerse

One skeleton, five sources.

Every clip is converted to the SOMA-77 convention, resampled to 50 Hz, floor-aligned and canonicalized at its first frame. Splits are drawn over source recordings, so no two clips cut from the same take land in different partitions.

0text–motion pairs

0hours of motion

0source recordings

0mean clip seconds

Composition of the Ω-MotionVerse corpus by source group and duration. — **Figure 3 — Composition.** Around 1,000 hours of text-paired motion.

Source	Pairs	Hours	Capture
MotionGV	542,787	690.7	Monocular video
BONES-SEED	351,422	281.2	Optical marker
HumanML3D	14,094	28.1	Optical marker
Fit3D	922	5.1	Optical marker
HumanSC3D	688	0.9	Optical marker
Total	909,913	1,006.0	—

05Results

Reconstruction leads;
generation follows.

12.83mm

MPJPE-22 on the shared test split, where the compared codecs land at 78.95, 90.02 and 96.01 mm after a round trip through their native representations.

T1 Motion reconstruction · HumanML3D

Method	MPJPE ↓	Med. ↓	PA-MPJPE ↓
MoMask	32.39	25.54	20.14
MotionGPT3	42.38	31.58	33.49
MotionMillion	42.54	31.51	19.84
SeMoCo	19.22	17.36	16.42

Millimetres, 22 SMPL joints. Baselines on the official test set; SeMoCo on the HumanML3D portion of our split, which preserves the original partition.

T2 Motion prediction · observe 0.5 s, predict 2 s

Method	ADE ↓	FDE ↓	minADE₅₀ ↓	minFDE₅₀ ↓
MotionGPT3	2.103	3.155	1.333	1.830
MDM	2.146	3.973	0.877	1.252
MotionGPT	2.637	3.943	1.531	2.086
Ours-Lite	1.228	2.449	0.695	1.095
Ours-Base	1.273	2.483	0.759	1.220

Metres. Lite outperforms Base here, reversing the text-to-motion ordering.

T3 Tokenizer ablation · matched 150 M generators

RVQ layout	Sem.	FID ↓	R@1 ↑	R@3 ↑	MedR ↓	MPJPE-77 ↓
Single-chain	✕	.202	.483	.663	1.65	15.60
Single-chain	✓	.211	.451	.637	1.93	18.54
Split-branch	✕	.226	.319	.490	3.83	13.70
Split-branch	✓	.186	.484	.656	1.70	15.93

TMR-SOMA overall pool, mean over 30 repeats. Semantic supervision inside the split layout moves FID from .226 to .186 and R@1 from .319 to .484, at a cost of 2.23 mm reconstruction — a trade-off the single-chain layout does not reproduce.

Qualitative text-to-motion comparison

Generated motion from HY-Motion, Kimodo and Ours-Base for three text prompts. — **Figure 4.** HY-Motion, Kimodo and Ours-Base at matched timestamps for three prompts.

06Stylized avatars

From text to a character.

07Real-world transfer

On a Unitree G1.

Take 01of 18

↓ Three takes, side by side

08Citation

Cite this work.

@article{huang2026semoco,
  title   = {SeMoCo: A Semantic-First Motion Codec for Motion Language Modeling},
  author  = {Huang, Tianlv and Guo, Hetian and Cai, Ziyi and Wang, Song and
             Zhang, Yanping and Fan, Zipei and Song, Xuan and Wu, Guangming and
             Zheng, Xin},
  journal = {arXiv preprint arXiv:2608.24334},
  year    = {2026}
}