Open VLA Monthly · September 2026 · First issue (with Inference Cost Benchmark, round one)

This month's judgements

• "Open VLA" is a layered claim: open weights, a commercial licence and a maintained repository are three different things. Of 40 indexed models, 17 have a licence that clearly allows commercial use, 17 have had a push in the last six months, and only 7 have both.
• Downloads have decoupled from maintenance. 54% of last month's Hugging Face downloads went to OpenVLA, whose repository has had no push for 544 days. It is downloaded because it is the baseline in papers, not because anyone is putting it on a robot.
• For single-stream inference, consumer GPUs beat the datacenter card. None of the five models ran faster on an A800 80GB than on an RTX 4090; four were 15–33% slower, at 3–4× the cost per thousand inferences. For batch-1 VLA inference, 80 GB buys VRAM headroom and nothing else.
• The constraint is latency, not money. On a 4090 the five models cost ¥0.07–0.57 per thousand inferences; but a robot that owns a GPU pays that GPU's daily rent (about ¥17.5) whatever model it runs. "Cost per 1k" misleads in dedicated-GPU settings, which is why our workbench now shows cost per day and per month.
• A VLA that reasons runs at about 1 Hz: it can be the brain, not the cerebellum. MolmoAct emits 64 tokens per step including a reasoning trace and lands at 0.7–0.94 s p50 on all three cards. SmolVLA, at 0.9 GB and 5 Hz, is the only one of the five that could fit an edge board. This matches the industry's three-layer view.
• The friction between "can download" and "can run" is the largest hidden cost for developers in China. The first nine models needed four mutually exclusive Python environments; four could not be reproduced, because of gated tokenizers and private runtimes, not GPUs.
• Data is polarised. Of 32 public datasets only 7 state their hours; the largest real-robot corpora come from Chinese organisations (InternData-A1 at 7,433 h, AgiBot World at 1.0 M trajectories), while the other pole is over ten thousand minutes-long SO-101 community datasets on Hugging Face. In between, datasets that are commercially licensed, downloadable and state their scale can be counted on one hand; 44% have an unverified licence.

The index: 40 models, more than half from Chinese organisations

As of 18 September the index holds 40 open-weight VLA models, 12 robot bodies and 6 inference GPUs. By organisation, 23 models come from Chinese institutions and 17 from elsewhere. By release year: 24 in 2025, 7 in the first nine months of 2026, 8 in 2024, 1 in 2023. The curve peaked in 2025 and has slowed, but the sample is too small to call a trend.

By size, among the 34 models with a stated parameter count, 6 are under 1B, 14 between 1 and 4B, 10 between 4 and 8B, 4 at 8B or above; the median sits near 3B. That matches the constraint of running on a robot or a single consumer GPU: bigger models exist in papers, not on robots.

Licence is the dirtiest field in the index. 17 clearly allow commercial use, 8 clearly forbid it, and 15 we mark as unverified: code and weight licences disagree, the model card says only "research use", or there is no LICENSE file at all. Those 15 are not our laziness; the sources themselves do not say. We do not conclude on their behalf.

Activity: half the repositories idle for six months, downloads concentrated in three models

This month we started a daily fetch of public counters (GitHub, Hugging Face, ModelScope). The first snapshot quantifies two things that were previously anecdotes.

First, of 39 models with a GitHub repository, 20 have had no push for more than 180 days and 28 had zero commits in the last 90 days. Only 11 had any commit: lerobot (SmolVLA) over 100, LingBot-VLA 2.0 with 23, InternVLA-A1.5 with 17, GR00T and Galaxea G0.5 with 10 each, the rest in single digits. "There are many open embodied models" shrinks to a quarter once filtered by maintenance.

Second, downloads are extremely concentrated. Hugging Face reports 809k downloads in 30 days across the index; the top three (OpenVLA, SmolVLA, GR00T N1.7) take 85.5%. OpenVLA alone has 435k, or 54%, while its repository has not been pushed for 544 days and its 7,039 stars stopped growing long ago. The plausible reading: it is the comparison baseline in almost every later paper, and evaluation scripts pull its weights automatically. Downloads here measure citation inertia, not production adoption. SmolVLA's 159k downloads and lerobot's 100+ commits are two sides of the same fact: it is currently the only model that is both used and repaired.

On ModelScope, 23 of 40 models have a same-name mirror, 14 uploaded by the vendor's own organisation and 9 by third parties. More than 40% of models still require the Hugging Face mirror route for developers in China.

Inference Cost Benchmark, round one: three cards, five models, 15 measurements

Method: one RTX 4090, one RTX 5090 and one A800 80GB rented on demand from AutoDL; bf16, batch 1, a fixed 224-pixel synthetic image and the same instruction; 50 warm-up steps then 500 timed steps; per-step p50 and p95 latency, throughput and peak VRAM recorded. Scripts, environments and raw result files are public, and every number traces back to a JSON file. We measure inference overhead only; we do not judge action quality.

Three readings.

Reading one: differences between cards are smaller than differences between models. On the same card the fastest and slowest models differ 8×; the same model differs at most 33% across the three cards. Model choice moves latency by an order of magnitude more than GPU choice.

Reading two: the 5090's gain over the 4090 is uneven. Models that generate more tokens gain more (MolmoAct 28% faster); small models break even or lose (NORA 15% slower). That fits the 5090's advantage being memory bandwidth rather than single-thread latency. The A800 trails across the board: it is built for multi-GPU training and batched inference, gives no single-request latency advantage, and costs 3–4× more per thousand inferences.

Reading three: cost is not the problem; frequency is. At the 4090's on-demand price of ¥2.19/h, the five models cost ¥0.07–0.57 per thousand inferences. But robots do not buy inference by the call: one robot occupies one card, and eight hours is ¥17.5 whichever model runs. What differs is how many hertz that ¥17.5 buys: about 8 Hz for NORA, 6 for OpenVLA, 5 for SmolVLA, 2.5 for SpatialVLA, 1 for MolmoAct. With a 5 Hz target, three of five qualify.

Reproduction has to be recorded honestly. Nine models were planned for round one; five ran. π0 and π0.5 need the gated PaliGemma tokenizer; MiniVLA and VLA-Adapter depend on the private prismatic runtime rather than transformers remote code. The five that ran needed four mutually exclusive environments: transformers 4.40.1, 4.47.0 and 4.57.1, plus a separate lerobot venv. Every pitfall is written up on the recipes pages.

Licence: read the text, not the impression

17 models clearly allow commercial use; only 7 of those were also maintained in the last six months: GR00T N1.7, LingBot-VLA 2.0, X-VLA, MolmoAct, Being-H0, RoboVLMs, Spirit v1.5. Most of the 8 models that forbid commercial use are academic weights released under CC-BY-NC. The commonest pattern among the 15 unverified is "code Apache-2.0, weights unstated"; we do not extrapolate the code licence to the weights.

Benchmark table: LIBERO is saturated, and a score belongs to a recipe

This month the public benchmark table went live: 340 scores across 39 models, from three kinds of source: 270 from papers, 38 from official repositories, 32 reprinted by third-party papers. Every number is copied as printed, with its table reference and protocol notes, grouped by benchmark, never aggregated across benchmarks. Three observations.

LIBERO no longer separates models. Of 26 LIBERO four-suite averages, the highest is 98.9 (Galaxea G0.5, InternVLA-A1.5) and the top eight all fall between 97 and 99, closer than the evaluation noise. From OpenVLA's 76.5 in 2024 to 98.9 today, the benchmark went in two years from "discriminating" to "everyone near the ceiling". A high LIBERO score now tells you a model was fine-tuned on LIBERO, not how strong it is.

The same model, 20 points apart. OpenVLA reports a LIBERO average of 76.5 in its own paper; with the OFT fine-tuning recipe (same weight family) it reaches 97.1. The difference is action decoding, fine-tuning steps and input setup, not the model. This is why we refuse to publish a composite score: recording both 76.5 and 97.1 with their sources is more honest than picking one as "OpenVLA's score".

SimplerEnv still discriminates; real-robot numbers do not compare. SimplerEnv Google Robot visual-matching scores spread from 52 to 83.5 (X-VLA 83.5, InternVLA-M1 80.7, CogACT 76.7, EO-1 76.5), making it one of the few public simulation benchmarks that still separates models. The 98 real-robot results come from 30 models on their own robots, tasks and trial counts; we record them, but they cannot be compared across papers, so they sit in their own group with a warning.

Two gaps: Spirit v1.5 has no printed number anywhere (only a leaderboard rank); every π0 score comes from third-party papers, because its own paper shows charts without tables.

Dataset index: unstated scale is the norm, and the middle is empty

The dataset index opens with 32 public datasets (real robot, human video, simulation), every field sourced. It is the other half of the model index: whether a model can be fine-tuned to your robot depends first on whether data exists. The first batch yields four facts.

Unstated scale is the norm. Of 32 datasets, 7 state total hours, 18 state trajectory counts, and 12 state neither, including LIBERO and CALVIN, used as benchmarks by dozens of papers, and vendor releases such as Galaxea and Unitree. The stated parts add up to 13,028 hours and 7.57 million trajectories, but the total is meaningless: InternData-A1 alone is 7,433 hours and Ego4D (human egocentric video, not robots) 3,670 hours; together they are 85% of the hours.

Chinese organisations supply the largest real-robot data. The 8 datasets from Chinese organisations include AgiBot World (1.0 M trajectories), InternData-A1 (630 k trajectories, 7,433 h), ARIO (3.03 M, aggregated from many sources), RoboMIND, Galaxea and Fourier ActionNet. By public real-robot trajectory count the top two are Chinese vendors. Yet most of these are gated on Hugging Face (access on request), and only 2 have a confirmed ModelScope page: for developers in China, data is harder to obtain than models.

Licences are as messy as for models. 11 clearly allow commercial use, 7 clearly forbid it, 14 are unverified (44%). The common pattern is a CC-BY-NC-SA dataset card next to an Apache-2.0 code repository, or no licence at all. We classify literally and do not extrapolate.

Formats are fragmented; LeRobot is not yet the standard. Of 32, 6 are in LeRobot format, 6 in HDF5, 7 custom, 8 unstated, the rest RLDS, zarr or webdataset. LeRobot v2/v3 is the fastest-growing format, but by stock it is a quarter; most of the time the first step with a new dataset is writing a converter.

One last number: searching Hugging Face for "so101" returns 10,235 datasets and "so100" returns 4,101 (observed 2026-09-18; search matches, not format-filtered). That is the long tail created by the SO-101 arm plus the LeRobot toolchain: tens of thousands of minutes-long datasets, worthless alone, and nobody has counted their total hours. That gap is worth filling.

What this means for developers

1. Screen in three steps: the licence text allows commercial use, the repository had a push in the last six months, and the model has a measured frequency on your card. Fewer than ten indexed models pass all three today.
2. Spend the GPU budget on consumer cards first: for a single robot with single-stream inference, a 4090-class card beats an 80 GB datacenter card on both latency and cost; only fine-tuning and multi-stream serving need the large VRAM.
3. Let the target frequency define the candidate set: skill layers that need 5 Hz or more are currently served only by models under 4B; models with reasoning traces belong in a 1 Hz planning layer.
4. Budget for environments: treat every model as its own Python environment and do not try to share; put weight-download time into the first day's schedule.

What to watch next month

• Once 30 days of activity data exist, whether weekly changes in stars and downloads track commit activity or keep decoupling.
• Round two of measurements: π0 / π0.5 (solving the gated tokenizer), GR00T N1.7, RDT-1B, CogACT; an int8 precision comparison.
• The first cross-reference after the dataset index and benchmark table go live: how many models are measured, scored, backed by data and commercially licensed at once.
• Filing an issue on each of the 15 licence-unverified repositories to ask about the weight licence, and recording the replies.

Method and limits

All figures come from data files dated 18 September 2026: model and body fields from repository READMEs, model cards and papers with sources attached; activity numbers from same-day snapshots of the GitHub, Hugging Face and ModelScope public APIs; measurements from GPUs we rented ourselves. Measurements reflect inference overhead on fixed synthetic input only; they do not reflect a real camera pipeline and do not judge action quality. The judgement sections are the maintainers' opinions and are labelled as such. All data is downloadable under /data/.