Researchers explore the possibility of decreasing the layer count for each token in large language models (LLMs) to speed up inference and reduce energy and financial expenditures. They introduce a self-speculative decoding method that combines early departure with speculative decoding, and experiment with layer dropout to minimize computation and increase prediction accuracy.
6 Impressions