---

name: corpora-and-vector-spaces description: "Guides Gensim dictionaries, bag-of-words vectors, streaming corpora, corpus persistence formats, and preprocessing workflows." disable-model-invocation: true metadata: disco-role: operating license: LGPL 2.1

Corpora and Vector Spaces

Use this sub-skill when the task starts from raw text, token lists, corpus files, Wikipedia dumps, or sparse vector streams and needs Gensim's corpus/vector-space abstractions before modeling or similarity indexing.

Read when

• A user asks to create or update a Dictionary, call doc2bow, or filter tokens by document frequency.
• A corpus must be streamed instead of loaded fully into memory.
• A task mentions Matrix Market, SVMlight, Blei/LDA-C, UCI, Mallet, Low, text directory, or Wikipedia corpus formats.
• A downstream model fails because query vectors, training vectors, or persisted corpora are in the wrong feature space.

Quick workflow

1. Tokenize documents with a stable preprocessing function such as gensim.utils.simple_preprocess or gensim.parsing.preprocess_string.
2. Build dictionary = corpora.Dictionary(tokenized_documents) or stream token lists into the constructor.
3. Convert token lists, not raw strings, with dictionary.doc2bow(tokens).
4. Filter only after deciding what must remain: filter_extremes, filter_tokens, filter_n_most_frequent, then compactify when needed.
5. Expose large data as an iterable yielding one BoW vector at a time.
6. Persist corpora with the format-specific serialize() method and reload with the matching corpus class.

Read references/workflows.md for end-to-end corpus recipes and references/api-reference.md for verified signatures and parameter notes. Read references/data-formats.md when selecting an IO format or handling compression/metadata.

API anchors

• corpora.Dictionary(documents=None, prune_at=2000000) builds token-id and frequency mappings.
• Dictionary.doc2bow(document, allow_update=False, return_missing=False) takes an iterable of string tokens.
• Dictionary.filter_extremes(no_below=5, no_above=0.5, keep_n=100000, keep_tokens=None) removes rare/common tokens and limits vocabulary size.
• corpora.MmCorpus.serialize(fname, corpus, id2word=None, index_fname=None, progress_cnt=None, labels=None, metadata=False) writes a lazy corpus with optional offset metadata.
• TextCorpus, TextDirectoryCorpus, and WikiCorpus help turn text files or XML dumps into streamed vectors.

Bundled helper

Run scripts/corpus_io_smoke.py to verify that the target environment can build a dictionary, serialize/reload Matrix Market data, and process a tiny local text-directory corpus without network access.

Boundaries and routing

• After the corpus is vectorized, route TF-IDF, BM25, LSI, LDA, HDP, NMF, and coherence tasks to ../topic-modeling-and-transformations/SKILL.md.
• Route Word2Vec/FastText/Doc2Vec/Phrases to ../embeddings-and-phrases/SKILL.md.
• Route document retrieval/indexing to ../similarity-and-search/SKILL.md.
• Route downloader/cache/vector-conversion/Wikipedia command-line utilities to ../data-and-cli-utilities/SKILL.md.

Common decisions

• Prefer Dictionary when the model needs stable token ids and document frequencies. Use HashDictionary only when bounded hash ids are acceptable.
• Use TextDirectoryCorpus when files are already laid out in directories and optional filename filtering matters.
• Use custom iterators for application-specific parsing, databases, or streams; Gensim only requires iteration over BoW vectors for most models.
• Use WikiCorpus or the data/CLI utilities only after acknowledging full dump size, compression, and runtime constraints.
• Save both the dictionary and the model/index that depends on it; a model alone cannot reconstruct the original feature mapping.

Troubleshooting

Read references/troubleshooting.md for raw string doc2bow errors, empty vectors, dictionary drift, malformed corpus files, encoding/compression problems, and large Wikipedia dump constraints.