paper-conference

Zeroth-Order Fine-Tuning of LLMs with Transferable Static Sparsity
Zeroth-order optimization (ZO) is a memory-efficient strategy for fine-tuning Large Language Models using only forward passes. However, …
Wentao Guo, Jikai Long, Yimeng Zeng, Zirui Liu, Xinyu Yang, Yide Ran, Jacob R. Gardner, Osbert Bastani, Christopher De Sa, Xiaodong Yu, Beidi Chen, Zhaozhuo Xu
PDF Cite
Zeroth-Order Fine-Tuning of LLMs with Transferable Static Sparsity
Coordinating Distributed Example Orders for Provably Accelerated Training
Recent research on online Gradient Balancing (GraB) has revealed that there exist permutation-based example orderings that are …
A. Feder Cooper, Wentao Guo, Khiem Pham, Tiancheng Yuan, Charlie F. Ruan, Yucheng Lu, Christopher De Sa
PDF Cite Code Poster
Coordinating Distributed Example Orders for Provably Accelerated Training
GraB: Finding Provably Better Data Permutations than Random Reshuffling
Random reshuffling, which randomly permutes the dataset each epoch, is widely adopted in model training because it yields faster …
Yucheng Lu, Wentao Guo, Christopher De Sa
PDF Cite Poster
GraB: Finding Provably Better Data Permutations than Random Reshuffling