界面已本地化;技能介绍与说明文档按所选语言由 AI 自动翻译,代码、命令、路径和专业术语保留原文,以原作者文档为准。
探索技能
效率与协作 · SKILL.md

importance_rank_allocator

importance_rank_allocator

【效率与协作】Compute AdaLoRA sensitivity-uncertainty triplet scores and globally mask singular values to a target rank budget.

独立 GitHub 快照 · 自动分类 · 以原作者文档为准

技能说明

GitHub

name: importance_rank_allocator description: Compute AdaLoRA sensitivity-uncertainty triplet scores and globally mask singular values to a target rank budget.

Importance-Aware Rank Allocator

Use this skill when a recovery or implementation needs AdaLoRA's adaptive rank allocation rather than fixed LoRA ranks.

Inputs

  • A list of adapted matrices with A, E, B parameters and matching gradients.
  • Previous EMA state for sensitivity and uncertainty, or empty state for the first step.
  • Hyperparameters beta1, beta2, and target global rank.

Outputs

  • Updated EMA state.
  • Per-triplet importance scores.
  • Mask threshold and masked singular values.
  • Rank pattern for every adapted matrix.

Workflow

  1. Compute instantaneous sensitivity abs(parameter * gradient) for every AdaLoRA parameter.
  2. Update smoothed sensitivity and uncertainty EMAs.
  3. Score each entry with smoothed_sensitivity * uncertainty.
  4. Combine each triplet as singular-value score plus mean scores from the corresponding A row and B column.
  5. Retain the top target-rank triplets across all matrices and set other singular values to zero.
  6. Log rank pattern and threshold so downstream recovery can verify budget compliance.

Validation

Run python scripts/rank_allocator.py --self-test or the provided tests.

Limitations

This script is deterministic and standard-library only. It is intended as executable mechanism evidence and a reference contract for tensor implementations.

我的收藏