Are AI Models Working Harder Than They Need to?
Much of modern AI runs on multiplication. Neural networks behind everything from generated answers to photo organization and song recommendations perform millions or billions of operations that multiply inputs by learned weights. John thinks that's more work than the job requires.
Key Takeaways
- Instead of repeatedly multiplying inputs by weights, these networks pass binary inputs through interconnected lookup tables-closer to consulting a collection of stored answers than solving the same arithmetic problem repeatedly.
Depending on the task, she says, the networks can be 1,000 times as small or as fast as conventional alternatives while maintaining comparable accuracy.
- had built a commercial product around it in the '80s for pattern recognition, and then it just disappeared.
A couple of professors at the Federal University of Rio [de Janeiro] kept working on it quietly for years, just one research group, so it wasn't a lot of work.
- We could get them 1,000 times smaller than what everyone else was getting.
Energy-Efficient Weightless Neural Networks Why is now the time to look into weightless neural networks?
- The model behind most popular networks today is based on a neuron model from a 1943 paper, the McCulloch-Pitts model .
The industry took that and expanded it, millions and billions of neurons, to get something that works.
- What have you been able to demonstrate so far?
Stats & Key Facts
- #Depending on the task, she says, the networks can be 1,000 times as small or as fast as conventional alternatives while maintaining comparable accuracy.
- #We could get them 1,000 times smaller than what everyone else was getting.
- #The model behind most popular networks today is based on a neuron model from a 1943 paper, the McCulloch-Pitts model .
John, a professor of electrical and computer engineering at the University of Texas at Austin, has spent the past five years working on a class of models called weightless neural networks . Instead of repeatedly multiplying inputs by weights, these networks pass binary inputs through interconnected lookup tables-closer to consulting a collection of stored answers than solving the same arithmetic problem repeatedly. Depending on the task, she says, the networks can be 1,000 times as small or as fast as conventional alternatives while maintaining comparable accuracy.
Her team's work has so far focused on small, specific problems: medical sensors, activity tracking, keyword spotting. But she thinks the same approach could eventually reach much bigger targets, including the transformer models behind today's chatbots. What made you walk away from weights in the first place?
Was there a specific moment that pushed you toward lookups instead? John: A friend casually invited me to a weekly meeting a few years ago to talk about using lookups instead of weights, a technique that wasn't new. had built a commercial product around it in the '80s for pattern recognition, and then it just disappeared.
For more details please read the original article at IEEE Spectrum AI.
Continue Learning
Comments
Sign in to join the conversation