---

name: language-models description: "Routes AXLearn GPT-family trainer catalogs, tokenizer variants, MoE configs, and flash-attention workflows." disable-model-invocation: true metadata: disco-role: operating license: Apache 2.0

language-models

Use this sub-skill for AXLearn's decoder-only language-model catalogs and related tokenizer/model-family helpers.

Typical triggers:

• GPT, C4, Pajama, Fuji, Gala, Honeycrisp, Qwen, or MoE model names.
• Long-context, flash-attention, RoPE, ALiBi, or mesh-shape questions in axlearn.experiments.text.gpt.
• Tokenizer files such as bpe_32k.json, bpe_128k.json, Llama-3-tokenizer.json, or Fuji v3 vocabulary helpers.
• Queries about tokamax, qwix, FlashAttention, or the model-specific trainer catalogs.

If the user is only asking about the shared trainer runtime, config_for_function, or fake-data smoke checks, use ../training-core/ first. If the user is asking about cloud launch or GCP job execution, use ../cli-cloud/.

What to read

• references/overview.md for the GPT catalog structure and major model families.
• references/troubleshooting.md for optional dependency and tokenizer-path failures.
• scripts/inspect_gpt_configs.py for a safe config-inspection helper.

Families covered

• c4_trainer for C4-based training catalogs.
• fuji, gala, honeycrisp, gspmd, and qwen for model-family builders and trainer variants.
• deterministic_trainer and the Pajama configs for dataset-specific catalog variants.
• vocabulary_fuji_v3 for tokenizer compatibility and Llama-3-style tokenizer files.
• gala_sigmoid for sigmoid-attention-related config manipulation.

Typical workflows

Inspect exported config names

Use the bundled helper to list the named trainer configs for a module and to inspect one resolved config:

python scripts/inspect_gpt_configs.py --module axlearn.experiments.text.gpt.gala --config 7B

Understand tokenizer wiring

The GPT helpers read tokenizer files from the configured data directory. When DATA_DIR=FAKE, they fall back to the packaged repository data under axlearn/data/tokenizers/.

Route around optional dependencies

Some GPT-family modules pull in extra MoE or flash-attention dependencies at import time. If the import fails, check references/troubleshooting.md before assuming the catalog is unavailable.

Decision points

• Use this sub-skill when the user names a concrete GPT-family architecture or tokenizer file.
• Keep reusable trainer mechanics in training-core.
• Do not send vision or ASR questions here just because they also use trainer configs.