training-and-compressionThis skill teaches a Researcher to train, compose, inspect, and troubleshoot Faiss compressed and binary indexes while preserving codec, dimension, and backend constraints.
Independent GitHub snapshot · Automated classification · Original documentation governs
Use this branch when the request involves training or adding compressed
codes, PQ/SQ/IVF-PQ, residual or additive quantization, RaBitQ, OPQ and
other transforms, PQ fast scan, standalone codecs, reconstruction, code size,
or binary/Hamming indexes. The CPU baseline is the verified Faiss 1.15.0
package (Python >=3.10, NumPy float32 arrays, OPTIMIZE DD AVX2). Treat every
CUDA, cuVS, ROCm, Metal, or SVS claim as conditional until the runtime probe
in the accelerated sibling succeeds.
nprobe, and
search behavior, read index-selection-and-search.IndexPreTransform, ID ownership, selectors, and IndexRefine*
composition, read composition-and-filtering.This branch owns the codec lifecycle and the compressed representation. It does not choose a general index family, own custom IDs/selectors, or certify optional accelerators.
Normalize the input as a contiguous float32 matrix of shape (n, d) for
float indexes, or a contiguous uint8 matrix of shape (n, d // 8) for
binary indexes. Binary dimension must be a multiple of eight; do not pass
unpacked boolean/float bits to an IndexBinary.
Select a factory or explicit class only after checking the constraints and code-size formulas in api-reference.md. Keep metric and transform dimensions consistent through the whole chain.
Keep a representative, finite training sample separate from the database.
Call train(training_vectors) before add(database_vectors) whenever
is_trained is false. Training data needs enough examples for the coarse
centroids and every codec; a small smoke may warn, but production training
should not ignore those warnings.
Add only vectors with the input dimension expected by the outer index. For an IVF codec, Faiss trains the coarse quantizer and encoder as part of the outer lifecycle; do not manually add to an untrained IVF-PQ/SQ index.
Search with a deliberately chosen nprobe (IVF) and k; then measure
against an exact baseline before trading bits, subquantizers, or probes.
Compression changes recall and reconstructed values even when the API
succeeds.
If a caller needs bytes, use sa_encode/sa_decode with uint8 arrays of
exactly sa_code_size() columns. reconstruct and decoded values are
approximations for lossy PQ/SQ/RQ/AQ/RaBitQ codecs; they are not the
original vectors. RaBitQ's sa_decode is suitable for its IP-oriented
codec representation but is explicitly not a reliable L2 reconstruction
oracle; use its search path for L2.
For a recall/latency compromise, first improve the candidate set (for
example IVF nprobe), then add a refinement stage owned by the composition
sibling. Refinement requires a compatible reconstructable/exact storage
choice and must not be used to hide an untrained or dimension-invalid base
index.
Run the bundled no-network check when validating a small configuration:
python /path/to/training-and-compression/scripts/smoke_codecs.py --help
python /path/to/training-and-compression/scripts/smoke_codecs.py
It uses deterministic tiny data and returns nonzero for invalid factory, dimension, metric, or binary input. It is a smoke, not a benchmark.
PQ/AQ fast scan is a storage/layout specialization, not a replacement for training or recall evaluation. The verified family is 4-bit PQ/AQ with a block size divisible by 32 (default 32); it repacks codes into blocks and may pad the final block. SIMD dispatch can select a different implementation or a scalar fallback. Do not infer that a fast-scan factory is available merely because the host has a GPU or because an unrelated SIMD level is present.
Residual, additive, product-residual, LSQ, and RaBitQ families expose more parameters than the compact examples here. Use the API and workflow tables, then run a tiny CPU trial before adopting one. GPU codec behavior belongs to the accelerated sibling and is unverified by this CPU branch.
For invalid IVF.../PQ strings, wrong d % M, too few training points,
unsupported fast-scan blocks, bad byte shapes, or reconstruction surprises,
start with troubleshooting.md. Preserve the
original failing configuration and error; repair one constraint at a time.
For a high-level factory replacement route to the index-selection sibling,
for staged refinement route to composition, and for recall/memory evidence
route to persistence-and-evaluation.