Enterprise artificial intelligence applications processing lengthy documents or complex tasks are confronting a critical memory challenge. As contextual data expands, the KV cache—where a model's working memory resides—grows proportionally, creating severe bottlenecks. Researchers at MIT have developed a groundbreaking solution called Attention Matching that compresses this cache by up to 50 times whilst maintaining accuracy, and it accomplishes this feat in mere seconds rather than hours.
Large language models generate responses token by token, storing mathematical representations of previous tokens as key-value pairs in the KV cache to avoid recalculating entire conversation histories. This memory consumption becomes problematic in enterprise scenarios such as analysing massive legal contracts or maintaining extended customer dialogues, where a single user request can balloon to gigabytes of memory. Traditional solutions like dropping older context or text summarisation prove inadequate—the former loses critical information, whilst the latter damages downstream performance significantly.
Attention Matching distinguishes itself from previous compression methods through its remarkable speed and quality preservation. Unlike gradient-based techniques like Cartridges, which require hours of expensive GPU computation to compress a single context, Attention Matching achieves comparable results in seconds. The technique preserves two crucial mathematical properties: the attention output (information extracted when querying memory) and attention mass (the relative weight of tokens). By generating reference queries that proxy the model's likely internal searches and using simple algebraic techniques rather than compute-intensive optimisation, the system maintains behaviour whilst dramatically reducing memory footprint.
Testing on open-source models like Llama 3.1 and Qwen-3 demonstrated impressive real-world performance. On QuALITY reading comprehension benchmarks and dense medical records from LongHealth datasets, Attention Matching achieved 50x compression without accuracy loss. Standard text summarisation failed catastrophically on complex medical records, performing no better than models without any context. The researchers even demonstrated 200x compression when combining Attention Matching with summarisation, and successfully tested online compaction where models hitting memory limits were compressed mid-task without performance degradation.
Whilst the technique requires access to model weights and integration into existing infrastructure demands engineering effort, MIT researchers have released the code publicly. Enterprise applications include compacting large tool outputs or lengthy documents immediately after processing. This approach aligns with industry trends, as major AI providers increasingly offer built-in compaction solutions rather than requiring enterprises to implement their own workarounds.
Fuente Original: https://venturebeat.com/orchestration/new-kv-cache-compaction-technique-cuts-llm-memory-50x-without-accuracy-loss
Artículos relacionados de LaRebelión:
- Zuckerberg Grilled Metas Usage Goals Underage Users
- LLM-Guardian Fortifying Your AI with Smart Defense
- Microsofts AI Scanner Uncovering Open-Weight LLM Backdoors
- Kimi K25 El LLM Open Source que Revoluciona las Abejas de Agentes
- Apple Slashes App Store Fees For Mini-Apps
Artículo generado mediante LaRebelionBOT
No hay comentarios:
Publicar un comentario