Close Menu
    DevStackTipsDevStackTips
    • Home
    • News & Updates
      1. Tech & Work
      2. View All

      CodeSOD: A Unique Way to Primary Key

      July 22, 2025

      BrowserStack launches Figma plugin for detecting accessibility issues in design phase

      July 22, 2025

      Parasoft brings agentic AI to service virtualization in latest release

      July 22, 2025

      Node.js vs. Python for Backend: 7 Reasons C-Level Leaders Choose Node.js Talent

      July 21, 2025

      The best CRM software with email marketing in 2025: Expert tested and reviewed

      July 22, 2025

      This multi-port car charger can power 4 gadgets at once – and it’s surprisingly cheap

      July 22, 2025

      I’m a wearables editor and here are the 7 Pixel Watch 4 rumors I’m most curious about

      July 22, 2025

      8 ways I quickly leveled up my Linux skills – and you can too

      July 22, 2025
    • Development
      1. Algorithms & Data Structures
      2. Artificial Intelligence
      3. Back-End Development
      4. Databases
      5. Front-End Development
      6. Libraries & Frameworks
      7. Machine Learning
      8. Security
      9. Software Engineering
      10. Tools & IDEs
      11. Web Design
      12. Web Development
      13. Web Security
      14. Programming Languages
        • PHP
        • JavaScript
      Featured

      The Intersection of Agile and Accessibility – A Series on Designing for Everyone

      July 22, 2025
      Recent

      The Intersection of Agile and Accessibility – A Series on Designing for Everyone

      July 22, 2025

      Zero Trust & Cybersecurity Mesh: Your Org’s Survival Guide

      July 22, 2025

      Execute Ping Commands and Get Back Structured Data in PHP

      July 22, 2025
    • Operating Systems
      1. Windows
      2. Linux
      3. macOS
      Featured

      A Tomb Raider composer has been jailed — His legacy overshadowed by $75k+ in loan fraud

      July 22, 2025
      Recent

      A Tomb Raider composer has been jailed — His legacy overshadowed by $75k+ in loan fraud

      July 22, 2025

      “I don’t think I changed his mind” — NVIDIA CEO comments on H20 AI GPU sales resuming in China following a meeting with President Trump

      July 22, 2025

      Galaxy Z Fold 7 review: Six years later — Samsung finally cracks the foldable code

      July 22, 2025
    • Learning Resources
      • Books
      • Cheatsheets
      • Tutorials & Guides
    Home»Development»Machine Learning»NVIDIA Researchers Introduce Dynamic Memory Sparsification (DMS) for 8× KV Cache Compression in Transformer LLMs

    NVIDIA Researchers Introduce Dynamic Memory Sparsification (DMS) for 8× KV Cache Compression in Transformer LLMs

    June 11, 2025

    As the demand for reasoning-heavy tasks grows, large language models (LLMs) are increasingly expected to generate longer sequences or parallel chains of reasoning. However, inference-time performance is severely limited by the memory footprint of the key–value (KV) cache, not just the number of tokens produced. In a recent paper, researchers from NVIDIA and the University of Edinburgh introduce Dynamic Memory Sparsification (DMS)—a data-efficient, retrofit-friendly method that compresses KV caches and unlocks inference-time hyper-scaling without degrading model accuracy.

    The Bottleneck: KV Cache in Transformer Inference

    Transformer-based models like GPT, LLaMA, and Qwen use KV caches to store past token representations for autoregressive generation. This cache grows linearly with sequence length and width (parallel threads), consuming large amounts of GPU memory and leading to slower inference due to frequent memory access.

    Existing techniques for KV cache optimization either rely on training-free heuristics—such as attention weight-based token eviction—or require heavy post-training retrofits like Dynamic Memory Compression (DMC). Both have significant downsides: the former tends to hurt accuracy, while the latter is computationally expensive.

    Dynamic Memory Sparsification DMS: Compression Without Compromise

    Dynamic Memory Sparsification DMS addresses these limitations with a hybrid approach: it sparsifies the KV cache like traditional pruning methods but does so with a minimal training overhead (~1,000 steps) and delayed eviction, which retains tokens temporarily after they’re marked for removal. This design preserves important context information and avoids abrupt accuracy drops.

    The core idea is to make eviction decisions differentiable during training using a Gumbel-sigmoid-based sampling mechanism. Tokens predicted for future eviction remain usable for a sliding window duration before being discarded, allowing the model to absorb their informational value more effectively.

    Efficient Retrofitting with Minimal Data

    Unlike DMC, which requires thousands of training steps and complex gradient-based optimization, DMS introduces no additional parameters per attention head. It reuses a small part of the attention mechanism (a single neuron) to predict eviction. This makes DMS ideal for retrofitting existing models without architectural changes.

    Empirical results show that with as few as 1K training steps, DMS can achieve 8× KV cache compression, preserving or even improving model performance across reasoning tasks.

    Benchmark Results: Scaling Performance Without Scaling Cost

    The research team tested DMS on reasoning-heavy benchmarks like:

    • AIME 2024 (advanced math)
    • MATH 500 (mathematical problem solving)
    • GPQA Diamond (hard science QA)
    • LiveCodeBench (code generation)

    Across model sizes—Qwen-R1 1.5B, 7B, and 32B—DMS improved exact-match performance by 9.1 points on AIME, 7.6 on GPQA, and 9.6 on LiveCodeBench, all under the same memory and compute budgets.

    When compared to top-performing baselines like Quest and TOVA, DMS consistently outperformed them in both KV cache read efficiency (runtime proxy) and peak memory usage, achieving better Pareto frontiers.

    General-Purpose Utility

    DMS also holds up in non-reasoning tasks. On short-context benchmarks like MMLU, GSM8K, and HellaSwag, DMS-maintained performance at compression ratios up to 4× with minimal degradation (~3.5 points). On long-context tasks like Needle-in-a-Haystack and Variable Tracking, DMS even surpassed the vanilla models, suggesting its potential to mitigate issues like information over-squashing in long sequences.

    Conclusion

    In conclusion, Dynamic Memory Sparsification (DMS) presents a practical and scalable solution for enhancing the inference-time efficiency of Transformer-based language models. By intelligently compressing the KV cache with minimal retraining, DMS enables models to reason over longer sequences or in parallel without increasing runtime or memory demands. Its consistent gains across a range of reasoning and general-purpose tasks highlight its versatility and effectiveness. As LLMs are increasingly deployed in resource-constrained environments, DMS offers a compelling path forward—balancing compression, accuracy, and ease of integration for real-world inference workloads.


    Check out the Paper. All credit for this research goes to the researchers of this project. Also, feel free to follow us on Twitter and don’t forget to join our 99k+ ML SubReddit and Subscribe to our Newsletter.

    ▶ Looking to showcase your product, webinar, or service to over 1 million AI engineers, developers, data scientists, architects, CTOs, and CIOs? Let’s explore a strategic partnership

    The post NVIDIA Researchers Introduce Dynamic Memory Sparsification (DMS) for 8× KV Cache Compression in Transformer LLMs appeared first on MarkTechPost.

    Source: Read More 

    Facebook Twitter Reddit Email Copy Link
    Previous ArticleMistral AI Releases Magistral Series: Advanced Chain-of-Thought LLMs for Enterprise and Open-Source Applications
    Next Article How Much Do Language Models Really Memorize? Meta’s New Framework Defines Model Capacity at the Bit Level

    Related Posts

    Machine Learning

    How to Evaluate Jailbreak Methods: A Case Study with the StrongREJECT Benchmark

    July 22, 2025
    Machine Learning

    Boolformer: Symbolic Regression of Logic Functions with Transformers

    July 22, 2025
    Leave A Reply Cancel Reply

    For security, use of Google's reCAPTCHA service is required which is subject to the Google Privacy Policy and Terms of Use.

    Continue Reading

    Windows 11 Notepad is turning into a full-fledged “AI writer” with Write feature

    Operating Systems

    Iran Slows Internet to Prevent Cyber Attacks Amid Escalating Regional Conflict

    Development

    CVE-2025-4567 – WordPress Post Slider Cross-Site Scripting Vulnerability

    Common Vulnerabilities and Exposures (CVEs)

    CVE-2025-1400 – “PLCTag Lib Overread Buffer Read”

    Common Vulnerabilities and Exposures (CVEs)

    Highlights

    Trusted-CGI – lightweight self-hosted lambda/applications/cgi/serverless-functions engine

    July 13, 2025

    Trusted-CGI is a lightweight runner for lambda functions/apps in CGI like mode. The post Trusted-CGI…

    With Copilot Studio’s new skill, your AI agent can use websites and apps just like you do

    April 16, 2025

    Bare – QuikJS Based JavaScript Runtime

    April 4, 2025

    CVE-2025-5959 – Google Chrome V8 Type Confusion Arbitrary Code Execution Vulnerability

    June 10, 2025
    © DevStackTips 2025. All rights reserved.
    • Contact
    • Privacy Policy

    Type above and press Enter to search. Press Esc to cancel.