---

name: topic-modeling-and-transformations description: "Guides Gensim TF-IDF, BM25, LSI, LDA, HDP, NMF, coherence, and transformation workflows." disable-model-invocation: true metadata: disco-role: operating license: LGPL 2.1

Topic Modeling and Transformations

Use this sub-skill when the task takes a bag-of-words corpus and needs a transformation or topic model such as TF-IDF, BM25, LSI/LSA, LDA, HDP, NMF, or coherence scoring.

Read when

• A task asks to train or apply TfidfModel, LsiModel, LdaModel, or LdaMulticore.
• A task needs CoherenceModel, topic comparison, or model persistence across topic transforms.
• A user wants to understand parameters such as num_topics, passes, iterations, chunksize, alpha, eta, random_state, or num_workers.
• A workflow is failing because the model was trained on a different dictionary or vector space than the query corpus.

Quick workflow

1. Vectorize text through corpora.Dictionary and doc2bow first.
2. Apply a transform such as TfidfModel to weight the bag-of-words corpus.
3. Train an LsiModel, LdaModel, LdaMulticore, HdpModel, or Nmf on the transformed corpus using the same id2word mapping.
4. Save the model after training, and if needed save the transformed corpus too.
5. Use CoherenceModel only after you know which texts, corpus, dictionary, or keyed vectors correspond to the topics you want to score.

Read references/workflows.md for guided recipes, references/api-reference.md for verified signatures, references/lda-workflows.md for LDA parameter strategy, and references/evaluation-and-coherence.md for coherence evaluation.

API anchors

• TfidfModel(corpus=None, id2word=None, dictionary=None, wlocal=..., wglobal=..., normalize=True, smartirs=None, pivot=None, slope=0.25).
• OkapiBM25Model(corpus=None, dictionary=None, k1=1.5, b=0.75, epsilon=0.25).
• LsiModel(corpus=None, num_topics=200, id2word=None, chunksize=20000, decay=1.0, distributed=False, onepass=True, power_iters=2, extra_samples=100, dtype=numpy.float64, random_seed=None).
• LdaModel(corpus=None, num_topics=100, id2word=None, distributed=False, chunksize=2000, passes=1, update_every=1, alpha='symmetric', eta=None, decay=0.5, offset=1.0, eval_every=10, iterations=50, gamma_threshold=0.001, minimum_probability=0.01, random_state=None, ns_conf=None, minimum_phi_value=0.01, per_word_topics=False, callbacks=None, dtype=numpy.float32).
• LdaMulticore(...) mirrors LdaModel but uses multiple CPU workers.
• HdpModel, Nmf, RpModel, LogEntropyModel, NormModel, and CoherenceModel are useful when the task needs alternative topic extraction or evaluation.

Bundled helper

Run scripts/topic_transform_smoke.py for a safe tiny-corpus transform and topic-model smoke. It proves the environment can train simple transformations without downloading external data.

Boundaries and routing

• If the task is about raw text preprocessing, dictionaries, or corpus format handling, route to ../corpora-and-vector-spaces/SKILL.md.
• If the task is about embeddings or phrase detection, route to ../embeddings-and-phrases/SKILL.md.
• If the task is about querying or indexing transformed vectors, route to ../similarity-and-search/SKILL.md.
• If the task is about downloading data or conversion CLIs, route to ../data-and-cli-utilities/SKILL.md.

Common decisions

• Use TfidfModel when you need a lightweight inverse-document-frequency weight before another transformation.
• Use LsiModel when an SVD-like latent space is helpful and the corpus is sparse or streaming.
• Use LdaModel or LdaMulticore when a probabilistic topic mixture is needed; LdaMulticore is the CPU multicore variant.
• Use HdpModel when the topic count should be inferred automatically rather than fixed.
• Use Nmf when non-negative factorization is the requested family.
• Use CoherenceModel to compare or tune topic quality, not as a substitute for model training.

Troubleshooting

Read references/troubleshooting.md for feature space mismatch, missing id2word, slow or stochastic LDA, coherence input mismatch, and optional distributed Pyro4 issues.