FedIT (Average)
Applies FedAvg independently to A and B adapters. Fast and simple, but the product (ΣBk)(ΣAk) ≠ ΔW introduces cross-term noise. Requires homogeneous client ranks.
Integrating Low-Rank Adaptation (LoRA) into federated learning offers a promising solution for parameter-efficient fine-tuning of Large Language Models (LLMs) without sharing local data. However, several methods designed for federated LoRA present significant challenges in balancing communication efficiency, model accuracy, and computational cost, particularly among heterogeneous clients. These methods either rely on simplistic averaging of local adapters, which introduces aggregation noise, require transmitting large stacked local adapters leading to poor download communication efficiency, or necessitate reconstructing memory-dense global weight-update matrices and performing computationally expensive decompositions to design client-specific low-rank adapters.
In this work, we propose FLoRIST, a federated fine-tuning framework that achieves mathematically accurate aggregation without incurring high communication or computational overhead. Instead of constructing the full global weight-update matrix at the server, FLoRIST employs an efficient decomposition pipeline by performing singular value decomposition on stacked local adapters separately, operating within a compact intermediate space. We introduce tunable singular value thresholding for server-side optimal rank selection to construct a pair of global low-rank adapters shared by all clients. Extensive empirical evaluations across multiple datasets and LLMs demonstrate that FLoRIST consistently strikes the best balance, achieving superior download communication efficiency while maintaining competitively better performance than baselines in both homogeneous and heterogeneous setups.
Applies FedAvg independently to A and B adapters. Fast and simple, but the product (ΣBk)(ΣAk) ≠ ΔW introduces cross-term noise. Requires homogeneous client ranks.
Stacks adapters for exact, noise-free aggregation with heterogeneous rank support. However, the download rank grows as ∑rk, causing high communication overhead.
Freezes adapter A at initialization so that only B is trained and transmitted— exact aggregation with half the communication cost. Still limited to homogeneous ranks and slower to converge.
Reconstructs the full ΔW and applies SVD to redistribute components to each client. Supports heterogeneous ranks but requires O(mn) memory and O(LKmn) compute on the server.
The singular value spectrum of the aggregated global weight update ΔW decays extremely rapidly. Despite clients using LoRA ranks up to 64, only 6–10 singular components are significant across all layers of TinyLlaMA fine-tuned on the Wizard dataset. Existing methods like FLoRA and FlexLoRA ignore this redundancy, transmitting far more information than necessary. This motivates FLoRIST's energy-based thresholding approach.
Singular value spectrum of q_proj layers in TinyLlaMA (Wizard dataset). Most singular values decay within the first 6–10 components despite a maximum client rank of 64.
Prior federated LoRA methods each address only a subset of the key desiderata. FedIT introduces cross-term aggregation noise and lacks heterogeneity support. FFA-LoRA freezes one adapter to eliminate noise but suffers slower convergence. FLoRA supports heterogeneous ranks via stacking but the global rank grows linearly, causing high download overhead. FlexLoRA reduces download cost but requires constructing and decomposing the full dense matrix ΔW ∈ ℝm×n, incurring large server compute.
Comparison of federated LoRA methods across aggregation accuracy, heterogeneity support, and key challenges. FLoRIST is the only method that addresses all limitations simultaneously.
FLoRIST stacks local adapters Bk and Ak with dataset-size weighting into Bstack and Astack. This ensures mathematically exact, noise-free aggregation equivalent to the true weighted sum ΔW = BstackAstack, while natively supporting heterogeneous client ranks.
Rather than constructing the prohibitively large ΔW ∈ ℝm×n, FLoRIST applies SVD independently to Bstack and Astack, then forms an intermediate product P = SBQSA ∈ ℝr×r. Crucially, the singular values of P are exactly the singular values of ΔW, computed entirely in the compact r×r space. This makes FLoRIST 7.5× faster than FlexLoRA's full-matrix SVD.
An energy-based threshold τ retains the smallest rank p such that the top-p singular values capture at least a τ-fraction of the total energy. The resulting single global adapter pair (Bg, Ag) is broadcast to all clients. Since p is typically far smaller than any client rank, this yields:
Rank: FLoRIST < FFA-LoRA < FlexLoRA ≤ FedIT < FLoRA
FLoRIST pipeline: (1) Upload local LoRA adapters → (2) Efficient SVD on stacked adapters → (3) Singular value thresholding for optimal rank p → (4) Construct global LoRA adapters → (5) Broadcast to all clients.
FLoRIST achieves the best accuracy–efficiency tradeoff across all 12 model–dataset–client configurations. The practical fixed threshold τ=0.9 stays within ±1% accuracy of the optimally tuned τ*, while often achieving even higher efficiency.
FLoRIST [τ=0.9] achieves the lowest normalized per-round download communication cost, a 400× reduction vs. full fine-tuning, 70× vs. FLoRA, and 5× vs. FFA-LoRA.
FLoRIST converges fastest and achieves the highest final MMLU accuracy. Singular value thresholding acts as a denoising regularizer, suppressing client-specific drift and retaining only the shared signal across clients.
As τ decreases from 1.0, performance initially improves (denoising effect), peaks around τ = 0.91, then degrades as useful components are discarded. For τ ≤ 0.99, FLoRIST achieves the lowest total global rank across all datasets. A fixed τ = 0.9 is a robust practical default with accuracy within ±1% of the optimally tuned τ* across all 12 configurations.
Total rank (log scale) vs. τ
MMLU score vs. τ (TinyLlaMA–Wizard)
@inproceedings{MLSYS2026_f38cb4cf,
author = {Ramesh, Hariharan and Dass, Jyotikrishna},
booktitle = {Proceedings of Machine Learning and Systems},
editor = {A. Chowdhery and Z. Jia},
pages = {1--21},
publisher = {MLSys},
title = {FLoRIST: Singular Value Thresholding for Efficient and Accurate Federated Fine-Tuning of Large Language Models},
url = {https://proceedings.mlsys.org/paper_files/paper/2026/file/f38cb4cf9a5eaa92b3cfa481832719c6-Paper-Conference.pdf},
volume = {8},
year = {2026}
}