Measured
Tests and results
What has been measured, on what hardware, and what is still open. The numbers are from the 0.1.4 preview unless stated otherwise; the weak spots are listed with the good ones.
The suite
- pytest, whole project
- 4867 passed, 3 skipped, 0 failed
- real-app smoke
- 23 / 23
- command-line mode
- 28 / 28
- from the unpacked package
- 23 / 23
- what it covers
- memory, attention, the voice pipeline, games, the panel, privacy, the scripted conversations
With the real model (gemma 12B, one run)
- who-is-who
- 11 / 11
- attention
- 17 / 17
- callbacks
- 6 / 6
- pirate (role kept)
- 4 / 4
- replies of four words or fewer
- 1 / 8 — weak
- preferences recalled
- 3 / 7 — weak
- "forget me" wrote the tag
- 4 / 4 for real requests, 0 / 4 for "just kidding"
- a cut-off note
- dropped or continued sensibly, both tried
Answers: 35 conversations (2026-10-17)
- answered
- 35 / 35
- words per answer
- median 24, p90 31, max 47
- seconds per answer
- median 1.3, p90 1.9; a web lookup 10-12 s
- assistant phrases
- 0.06 per answer
- stage directions spoken aloud
- 0
- emoji
- 0
- repeats of an earlier answer
- 0
- five people at once
- all answered, 1.7-7.7 s (one model, so one after the other)
- ten lines in one second
- no error; put together and answered once
- empty input
- HTTP 400, as it should be
- 6,000 characters, control characters, her own tags typed by a person
- all answered, nothing executed
These move between runs: the same battery the day before read median 25-27 words and 1.4-1.5 s. Read them as ranges, not as a score.
Not verified
- Discord voice with real people (DAVE)
- not tried
- her joining a voice channel
- not tried
- the sound board
- not tried
- VRChat, osu!, Minecraft in the real games
- not tried
- runs of many hours
- not tried
- the recogniser on other voices
- not tried
- a second Live2D model
- not tried
- a real Claude Code or Codex session using her
- not tried
Run it yourself
- python3 -m pytest
- the whole suite; no model needed
- scripts/smoke.py
- a sandbox app, started with AURA_NO_PLAYBACK=1
- scripts/quality_battery.py --quick --json out.json
- the 35 conversations, about a third; needs a running Aura and her key
- scripts/people_and_bits.py
- who-is-who, attention, callbacks and pirate