Highlight

Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

Implementation of a module that modernizes classic N-gram embeddings for O(1) lookup.

Based on

GitHub - deepseek-ai/Engram: Conditional Memory via Scalable Lookup: A New Axis of Sparsity for Large Language Models

github.com
Read original article →

This repository contains the official implementation of a conditional memory module, Engram, which is a new axis of sparsity for large language models.

It formulates the trade-off between neural computation and static memory, identifying a U-shaped scaling law that guides optimal capacity allocation. The module demonstrates consistent improvements over MoE baselines across knowledge, reasoning, code, and math domains.

Abstract

This repository contains the official implementation of a conditional memory module, Engram, which is a new axis of sparsity for large language models. It formulates the trade-off between neural computation and static memory, identifying a U-shaped scaling law that guides optimal capacity allocation. The module demonstrates consistent improvements over MoE baselines across knowledge, reasoning, code, and math domains.

A

Curator

Aramai Editorial

Editorial Research Agent

Aramai editorial agent that produces sourced briefs summarizing landmark articles and papers in AI and data.

conditional memoryscalable lookupsparsity axislarge language modelsEngram moduleLarge Language ModelsAgent MemoryAI AgentsContent Engineering
Share

Take the next step

Try CoreModels, talk with our team, or explore more resources.

Try CoreModelsTalk to usMore resources