asist-p — local voice assistant

Español

It listens through the microphone, thinks and answers out loud. Everything runs on the machine; only the optional web search goes online.

It joins three existing engines (Canary to hear, llama.cpp to think and qwentts.cpp to speak) in a Rust pipeline of threads and channels where no stage waits for the next.

  microphone ──samples──> segmenter ──utterance──> ASR ──text──┐
  (cpal, real time)       (VAD, turns)             (Canary)    │
                                                                v
  speaker <──samples── synthesis <──sentences── conversation <─┘
  (cpal, ring)          (qwentts)                (llama.cpp + tools)

Each box is a thread; each arrow, a channel. Two decisions carry the rest:

• Nothing blocks the stage ahead. The microphone never waits for the recognizer, and the model never waits for the synthesizer. Whoever has spare capacity drops work instead of accumulating delay.
• Answers go out per sentence, not per answer. Each sentence goes to the synthesizer as soon as it closes, so the assistant starts talking while the model is still writing. That is what separates half a second from five.

The assistant speaks Spanish: its prompts, tool descriptions and spoken acknowledgements are in Spanish on purpose. The code is in English.

Getting started

git clone --recursive https://git.all.ar/pub/asist-p.git && cd asist-p
scripts/bootstrap.sh          # submodules, patches, binaries and models
cargo run --release -- check  # check that everything is in place
cargo run --release -- run    # start talking

bootstrap.sh is idempotent and does not copy models: it links them from wherever they already are ($ASIST_MIRROR, default ~/GIT-MIRRO, and $ASIST_HF, default ~/HF). If it cannot find the engines built, it builds them, and that does take a while.

cargo run --release -- devices          # list audio devices
cargo run --release -- run --barge-in   # allow interrupting the assistant
cargo run --release -- run --shell      # enable system commands
cargo run --release -- run --no-manage  # do not start the servers

While developing, keep the servers running separately (loading the models takes more than a minute) and restart only the Rust binary:

scripts/servers.sh start
cargo run --release -- run --no-manage

Layout

The three engines live in vendor/ as submodules pinned to a specific commit. The local changes on them are in vendor/patches/, and bootstrap.sh applies them: without them the recognizer does not behave like the one that was measured.

Configuration

Everything is in config/asistente.toml, with comments. Values that carry a number in their comment come from a specific measurement, not an assumption; the details are in docs/RENDIMIENTO.md (in Spanish).

The most tweaked settings:

[vad]
silence_hold = 0.8     # how much silence ends your utterance
barge_in = false       # interrupt the assistant by talking over it

[general]
history_turns = 8      # conversation memory
system_prompt = "..."  # personality and style

[tools]
shell = false          # run system commands

Interrupting and speaker echo

By default the assistant is half duplex: while it talks, the microphone is closed. That is not laziness: with open speakers the microphone hears itself, and the assistant transcribes itself and answers itself.

With headphones, --barge-in lets you talk over it to interrupt. The threshold is raised (vad.barge_in_factor) so the echo is not enough but a real voice is.

Latency

A real turn, measured end to end:

The binary measures this on every turn and flags the slowest stage. A few configuration settings were worth more than any code change:

• The TTS decoded in 24 s blocks: 4948 ms → 585 ms to first audio.
• The model template left <think> open: 8630 ms → 413 ms to first token.
• The style prompt suppressed tool calls: 0/8 → 8/8.
• The camera captured at 720p: 7.8 s → 2.9 s per image at 640x480.
• Tools ran in silence: ~8.4 s → 4.1 s until something is heard.
• The screen at 640 px made the text up: 1 of 3 right → 3 of 3 at 1280 px.

All of them, with the reasons and how they were measured, are in docs/RENDIMIENTO.md (in Spanish).

Tools

The assistant can call functions. It ships with these:

The tool names are Spanish because the model reads them in a Spanish conversation.

While a tool works, the assistant says «Déjame que lo busque» or «Voy a mirar». It is not decoration: without it the turn spends six seconds in silence and reads as a hang. With the acknowledgement, the first audio arrives in 3–4 s.

Search supports three backends: tavily (with a key, returns an already written answer), ddgs (no key, the same library behind duckduckgo-mcp) and searxng. The command backend runs a script that writes JSON, so adding another engine means writing a script, not touching Rust.

Vision (camera and screen) takes advantage of the server already loading the multimodal projector: the same model that converses describes the image. Neither the photo nor the capture is saved to disk, and they do not enter the history.

The screen uses 1280 px and the camera 640, and the difference matters: a scene can be understood, but text has to be read. Measured, at 640 px the model does not say it cannot read the screen, it makes up what it says (it answered that the meeting was «at 10:00» when it said 15:30). Hence the threefold cost.

ejecutar_comando is off on purpose: giving a shell to a model that obeys what it hears through the microphone is a change of security posture. When it is on, only allowlisted commands get through, with no shell to interpret metacharacters and with a deadline.

Adding your own takes about twenty lines: docs/EXTENDER.md (in Spanish).

Voice

The repository ships no cloned voice: a voice belongs to someone. To make the assistant speak with one, record a few seconds of audio with its transcript and extract the latents:

scripts/clone-voice.sh recording.wav transcript.txt

They end up in assets/voices/ (outside git). Without a cloned voice, remove the [tts.reference] section from config/asistente.toml and the assistant uses one of the model's own voices.

Tests

More than a hundred unit tests that need neither models nor a microphone, plus integration tests against the real servers.

cargo test                                    # unit tests, no models
scripts/servers.sh start
cargo test --release -p asist-app --test integration -- --nocapture
cargo test --release -p asist-app --test integration -- --ignored  # full loop

The integration tests skip themselves when no server is listening, and they take turns on the GPU through a mutex: both servers share a 4 GB card, and in parallel they measure contention instead of latency.

The --ignored test closes the loop without a microphone: it synthesizes a sentence and checks that the recognizer gets it back.

Requirements

• Rust 1.85 or later
• CUDA for the C++ engines (they run on CPU, but barely)
• PipeWire or ALSA
• ~6 GB of disk for the models
• ffmpeg for the camera and to scale down captures
• For the screen: grim (Wayland) or maim/ImageMagick (X11)
• For search: a Tavily key in $TAVILY_API_KEY, or uv tool install ddgs

License

MIT