Timing Trick Cuts Energy Used in LLM Training by Up to 14 Percent
Researchers at the University of Twente in the Netherlands cut the energy used to train a large language model by up to 14 percent without slowing it down, by finely tuning how fast a GPU's clocks tick during computation. The method, presented at the Computing Frontiers conference in Sicily, adjusts the chip's speed at the level of tiny tasks called kernels rather than across whole training steps. In tests training a 1.3 billion parameter model, the approach saved roughly 14.6 percent of energy with only a 0.6 percent increase in training time.
Key Takeaways
- OpenAI 's fourth large language model (LLM), GPT-4 , took an estimated 50 gigawatt-hours to train, or the equivalent of 5,000 American homes ' yearly power consumption.
Since then, the computational resources used to train frontier LLMs have only increased , though direct power usage numbers are hard to come by.
- Every chip-including the GPUs commonly used for training frontier models-uses at least one clock to orchestrate computations.
- DVFS has been a well-known technique that goes back to at least the 1990s.
But Spaan says other researchers haven't been able to usefully apply it to LLM training because their methods either slowed down calculations too much or were not fine-grained enough to improve energy usage.
- For example, a single vector-vector multiplication can make up a single kernel.
The kernels are fed to the GPU to be processed many times in parallel.
- Less energy, same time The team performed their experiment by training GPT-3-XL, a 1.3 billion parameter model, on an Nvidia RTX 3080 Ti GPU.
Stats & Key Facts
- #Researchers at the University of Twente in the Netherlands cut the energy used to train a large language model by up to 14 percent without slowing it down, by finely tuning how fast a GPU's clocks tick during computation.
- #In tests training a 1.3 billion parameter model, the approach saved roughly 14.6 percent of energy with only a 0.6 percent increase in training time.
- #OpenAI 's fourth large language model (LLM), GPT-4 , took an estimated 50 gigawatt-hours to train, or the equivalent of 5,000 American homes ' yearly power consumption.
- #Less energy, same time The team performed their experiment by training GPT-3-XL, a 1.3 billion parameter model, on an Nvidia RTX 3080 Ti GPU.
OpenAI 's fourth large language model (LLM), GPT-4 , took an estimated 50 gigawatt-hours to train, or the equivalent of 5,000 American homes ' yearly power consumption. Since then, the computational resources used to train frontier LLMs have only increased , though direct power usage numbers are hard to come by. Now, a research group at the University of Twente in the Netherlands has shown that you can save up to 14 percent of the energy used in LLM training without sacrificing speed by cleverly adjusting the clock frequency of the GPU during computation.
Jeffrey Spaan , Ph.D. candidate at University of Twente and lead author on the article, presented the results at the Computing Frontiers conference in Catania, Sicily, last month. "My research is about finding computing waste," Spaan says. "It's similar to underutilization of the hardware, but instead of optimizing the software for the hardware, we try to optimize the hardware for the software."
Making the GPU tick Spaan and his collaborators accomplished this by using a technique known as dynamic voltage and frequency scaling ( DVFS ). Every chip-including the GPUs commonly used for training frontier models-uses at least one clock to orchestrate computations. Each operation in the chip is triggered by a clock pulse.
For more details please read the original article at IEEE Spectrum AI.
Continue Learning
Comments
Comments appear only after moderation. Your email identifies your submission to the moderator and is never displayed here.
No approved comments yet.