Fed-SB introduces a novel approach for federated fine-tuning of large language models using Low-Rank Adaptation (LoRA), addressing the challenges of high communication costs and performance degradation in traditional methods. By leveraging a small square matrix to optimize updates, Fed-SB significantly reduces communication costs while enhancing performance across various reasoning tasks, establishing a new balance between efficiency and effectiveness in both private and non-private settings.