---

name: inference-evaluation description: "Guides RWKV-LM RWKV-7 inference modes, checkpoint/tokenizer setup, prompt sampling, and MMLU-style evaluation troubleshooting." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

RWKV inference and evaluation

Use this route when the request mentions running an RWKV checkpoint, GPT-mode, RNN-mode, fast decoding, rwkv_v7_demo, RWKV_RNN, sampling settings, model state, tokenizer setup, MMLU, multiple-choice scoring, or generation debugging.

Route by task

• Choose a demo path: read inference-workflows.md to decide between GPT-mode prefill, RNN-mode token-by-token decoding, and the fast mixed path.
• Build a safe config: use build_inference_config.py to emit a JSON config with checkpoint, architecture, sampling, and mode fields.
• Adapt MMLU or another MCQ evaluation: read evaluation-workflows.md and use render_mmlu_prompt.py to validate the prompt shape.
• Debug a failed run: read troubleshooting.md before changing dtype, backend, or tokenizer.

Inference decisions

1. Confirm checkpoint family and dimensions (n_layer, n_embd, vocab_size, head size). The demo scripts hard-code these values; a mismatch can produce shape errors or nonsense logits.
2. Confirm tokenizer family. Current RWKV-7 demos use rwkv_vocab_v20230424 with vocabulary size 65,536. Older v4/v4neo examples may use the 20B JSON tokenizer and 50k vocabulary.
3. Choose mode:
   • GPT-mode is convenient for full-sequence prefill and comparison.
   • RNN-mode is appropriate for stateful autoregressive generation.
   • Fast mixed mode combines prefill and decoding but depends on CUDA custom extension compilation and a compatible GPU/toolkit.
4. Keep prompt, temperature, top_p, top_k, length, and trial count explicit in a config file or command record.
5. Treat local model paths from repository examples as placeholders. Replace them with the user's actual checkpoint and do not publish private paths.

Evaluation decisions

The MMLU script formats each sample as a four-choice prompt and scores the last-token probabilities for tokens corresponding to " A", " B", " C", and " D". Adapt that pattern carefully when choices are not single tokens or when a different tokenizer is used.

Full evaluation requires a checkpoint, tokenizer, dataset, and often GPU time. For local validation, test prompt rendering and answer-token mapping before running the complete benchmark.

Handoff

Route data preparation or training commands to training-data. Route tensor names, Qwen comparison, or context-parallel state composition to architecture-reference. Route RWKV-8/ROSA toy scripts to rosa-experiments.