monkeypatch.relora

monkeypatch.relora

Implements the ReLoRA training procedure from https://arxiv.org/abs/2307.05695, minus the initial full fine-tune.

Classes

Name	Description
ReLoRACallback	Callback to merge LoRA weights into the base model and save full-weight checkpoints

ReLoRACallback

monkeypatch.relora.ReLoRACallback(cfg)

Callback to merge LoRA weights into the base model and save full-weight checkpoints

Functions

Name	Description
magnitude_pruning_	Zero the lowest prune_ratio fraction of values by absolute magnitude, in place.
random_pruning_	Zero a random prune_ratio fraction of values, in place.
reset_optimizer	Prune optimizer state for reset_params only.

magnitude_pruning_

monkeypatch.relora.magnitude_pruning_(tensor, prune_ratio)

Zero the lowest prune_ratio fraction of values by absolute magnitude, in place.

random_pruning_

monkeypatch.relora.random_pruning_(tensor, prune_ratio)

Zero a random prune_ratio fraction of values, in place.

reset_optimizer

monkeypatch.relora.reset_optimizer(
    optimizer,
    *,
    reset_params,
    optimizer_state_keys,
    prune_method='magnitude',
    prune_ratio=0.9,
)

Prune optimizer state for reset_params only.