Sequence modeling · Efficient attention

RAM-NetLinear-Time Sequence Modeling
with Sparsely Addressable State

A larger fixed-size state, accessed a few slots at a time.

A large memory, a small access footprintA current token writes to four random slots, then reads them back. Eight groups of sixteen slots form the memory array. This is a conceptual illustration. SPARSELY ADDRESSABLE STATE Large memory. Sparse access. current token
The idea, in brief

Borrow the philosophy of computer RAM: manage memory by address, and let the model use pointers for pinpoint access and localized updates.

01Motivation

Larger State, Sparser Access

The cost of history

A sequence model keeps information from past tokens in a memory state and retrieves what it needs at each step. Its efficiency depends on both the space needed to store that history and the amount accessed per step.

Full attention [1] keeps one key–value entry per token and compares each query against all of them, so both costs grow with sequence length. KV compression and sparse attention reduce per-step access, but the cache typically keeps growing.

Mainstream linear attention [2] compresses history into a fixed-size state. Each step still reads and writes the entire state, so tokens interfere within the same state, and increasing its size also increases per-step access.

RAM-Net divides this fixed-size state into many addressable slots, and each step reads and writes only a few selected ones. Tokens interfere only when they select the same slots, so the state can grow without increasing per-step access.

MethodHow history is storedWhat each step accesses
Full attentionOne key–value entry per tokenAll entries
KV quantization / MLAOne compressed entry per tokenAll compressed entries
Sparse attentionOne entry per token in most methodsA selected group of entries
Linear attentionA single fixed-size stateThe entire state in most designs
RAM-NetA fixed-size array of slotsA few selected slots
02Architecture

RAM-Net = Sparse Addressing + Localized Updates

Product Softmax · CAPE · Gated Sparse Update

RAM-Net separates address-based access from localized slot updates. The Address Decoder selects sparse read and write addresses; positional encoding then adjusts them for token position. Gated Sparse Update (GSU) controls how each selected write slot retains its history and incorporates new information. These components operate on a fixed-size array of |𝒜| slots, each storing a vector at an integer address.

For each token, learned projections produce three vectors: a key to select write slots, a query to select read slots, and a value to store. Three components connect these inputs to the slot memory:

  1. Address Decoder: Product Softmax turns compact groups of scores into slot weights. Top-K retains the integer addresses and weights of the K highest-weight slots. The key defines the write access; the query defines the read access.
  2. Cyclic Address Positional Embedding (CAPE): shifts the decoded read and write indices according to token position, without changing their weights. Applied to some heads, it makes read–write matching depend on relative position; the remaining heads retain content-only addressing.
  3. Gated Sparse Update (GSU): takes the value and the write weights, then forgets and updates only the selected write slots. Each selected slot updates independently; all unselected slots stay unchanged.

After the update, the read weights combine the selected slots into an output, which is normalized, gated, and projected. The Address Decoder and CAPE determine where information is read and written, while GSU determines what each selected write slot retains and updates. The figure illustrates 32 slots with Top-4 access; Sections 3, 4, and 5 explain the three components in detail.

Together, these components decouple memory capacity from the per-step access budget. The slot count |𝒜| sets the size of the state, while K sets how many slots each read or write accesses. Neither grows with sequence length, so storage and per-token cost stay constant as the context grows.

03Addressing

Efficient Sparse Addressing

Product Softmax

Select a view or adjust Top-K; drag factor weights to reshape the address distribution.

Section 2 introduced the Address Decoder, which selects a small set of memory slots and assigns their weights. The challenge is to make address selection itself efficient as memory grows, while keeping the key and query compact.

Product Softmax combines several low-dimensional distributions to assign weights to a much larger set of slots.

Definition and Calculation

First, write each integer slot address a as U base-dp digits, giving |𝒜| = dpU possible addresses. Here, a(u) denotes the value at digit position u, one of 0, …, dp − 1; u = 0 is the least significant digit.

The key kt has dk = U dp dimensions. Split it into U sub-vectors kt(u), each with dp entries. Each sub-vector supplies scores for one address digit a(u), one score for each possible value. The computation below turns each group into a digit distribution and combines them into slot weights. The query is decoded in the same way.

Product Softmax equations: per-digit softmax, address probabilities, and Kronecker product Product Softmax equations: per-digit softmax, address probabilities, and Kronecker product
  1. Normalize each digit: apply a softmax to each sub-vector to obtain its digit distribution. The temperature τ > 0 controls how sharply each digit selects among its possible values.
  2. Multiply across digits: a slot's probability is the product of the probabilities of its digits. The complete distribution is the Kronecker product of the digit distributions.
  3. Keep only Top-K: the truncation operator 𝒯K retains the K largest probabilities and sets all others to zero, without renormalizing the retained weights. The key produces write weights; the query produces read weights.

The slot count |𝒜| = dpU grows exponentially with the number of digits U, while the key and query dimension dk = U dp grows only linearly, so U dp scores suffice to address all |𝒜| slots. For example, |𝒜| = 1024 = 45 slots require only 20 scores. Moreover, each digit distribution is shared by many slots, so a learning signal from a selected slot also updates the factors used by other addresses.

Below, we interpret Product Softmax from five different perspectives.

Soft Radix Address

This is the view used in the definition above: after softmax, each score group makes a soft choice for one address digit. Each digit has dp possible values; combining one value from each of the U groups forms a complete address. The product of those digit probabilities gives the weight of that address.

Multilevel Decision Tree

Addresses form a prefix tree: each node is a partial address, with dp branches for the next digit. At each level, one score group supplies the branch probabilities, so multiple choices can carry weight simultaneously. A complete path identifies one slot, weighted by the product of its branch probabilities.

Joint Distribution

Product Softmax defines a U-dimensional joint distribution, with dp possible values along each axis. Each normalized score group supplies one axis's marginal distribution; multiplying these independent factors gives the probability at every grid point. A peaked factor concentrates weight on a slice of the space; combining factors focuses it at the slices' intersections. The figure shows U = dp = 4.

Regroupable Address

Several digits can be jointly encoded as one larger address digit: a group of m digits has dpm possible values. Its distribution is the product of the original digit distributions, so regrouping changes the shape of the address space without changing any slot weight. In the figure, three base-4 digits become one 64-way digit, yielding 64 × 4 and 4 × 64 views of the same slots.

Waveform Modulation

For any flat address a, the slot weight ρU(kt)[a] is the pointwise product of U sub-functions, where φ(u)(kt)[a] = pt(u)[a(u)]. As a runs through the slot array, wave u holds each value for dpu consecutive addresses and repeats every dpu + 1 addresses. Low-order digits therefore produce short steps and fast cycles; higher-order digits produce longer plateaus and slower envelopes.

Soft radix describes digit choices; the prefix tree organizes their combinations; the joint distribution places them in U dimensions; regrouping combines digits into larger coordinates; and waveforms lay the factors out along the address axis. These are five equivalent views of the same Product Softmax distribution, each useful in a different setting.

The tree view supports exact Top-K pruning: keep the K strongest prefixes at each level, evaluating at most K dp candidates per level without constructing the full slot distribution. The joint view explains gradient broadcast: learning at one slot updates axis factors that are shared by many other slots. The waveform view explains CAPE's shift equivalence: circularly shifting the full distribution is equivalent to shifting every factor waveform by the same amount before multiplying them.

04CAPE

Position-Aware Addressing

Relative position

Toggle CAPE in the diagram; drag read or write weights to compare address overlap.

Before positional shifts, Product Softmax maps identical keys or queries to the same absolute slot addresses. Cyclic Address Positional Embedding (CAPE) turns this into relative-position-aware addressing, so a read can distinguish the same content at different distances.

At token position t, CAPE shifts each decoded address by −t, keeping its weight unchanged. It shifts the read and write indices, not the stored content. The modulo operation wraps addresses around the finite slot array.

A write at position s is shifted by −s, while a read at t is shifted by −t. Their overlap therefore depends on content and the relative distance t − s, rather than on the two absolute positions separately. Like rotary position embeddings (RoPE), CAPE introduces relative position, but through address shifts rather than rotations of keys and queries. The physical accesses still use integer slot addresses.

Because this is only an index permutation, each read or write still accesses K slots. RAM-Net combines CAPE heads for relative-position patterns with heads without positional encoding (NoPE) for associations independent of distance. Both keep the same sparse access budget.

05GSU

Balancing Retention and New Information

Updating slots

Select a stage, adjust γ, or drag a value to follow the slot update.

Gated Sparse Update (GSU) performs per-slot test-time training (TTT) [3]: each addressed slot acts as an online learner, moving its stored content toward the incoming value with one gradient step. Its local objective is ℓt(s) = ½‖s − vt‖2, with a learning rate adapted to the slot's history and the current input.

A forgetting gate over the entire state would touch every slot at every step, defeating sparse access. GSU instead forgets and learns only at the selected slots, leaving all others exactly unchanged. Each slot balances retaining its own history against adopting new information.

Each slot a maintains a content vector st[a] and a scalar mass mt[a]: the accumulated write weight retained from earlier updates. This mass acts as inertia against new writes. Given the current write weight wt[a] and value vt, the complete update is:

Gated sparse update equations

Read the equations from top to bottom, in the same order as the animation:

  • Retention (λ): starts from 1 − wt[a], so a stronger write retains less historical mass. The input-dependent modulator γt adjusts this baseline in logit space: a larger γ means more forgetting. A learned projection produces one γ per token and head, shared across slots.
  • Mass (m): keep a fraction λt[a] of the previous mass, then add the current write weight. Forgetting reduces the slot's inertia, allowing it to learn new content faster.
  • Learning rate (η): divide the current write weight by the updated mass, with a small ε for numerical stability. A stronger write increases the step; more retained history suppresses it.
  • Content (s): move an η fraction of the way from the current content toward vt. A small η preserves history; an η near 1 lets the new value dominate.

As a result, GSU has three properties:

  1. Exact sparsity: for an unselected slot, wt[a] = 0 gives λt[a] = 1 and ηt[a] = 0, so both its content and mass stay unchanged. Writes from different tokens interact only at the slots they jointly address.
  2. Bounded content: since ηt[a] ∈ [0, 1), each update is a convex combination of the previous content and vt, keeping the slot's content on the same scale as the input values, regardless of how many writes it receives.
  3. Adaptive learning rate: the current write weight and historical mass set each slot's learning rate, while γt adjusts retention for the current token and head. As γt → −∞, the slot approaches a write-weighted running average; as γt → +∞, each write nearly overwrites the slot. The model learns when to preserve history and when to adapt quickly.
06Execution

Parallel Work Across Memory Slots

CUDA · Sparse access on the GPU

Sparse reads and writes are scattered across memory, which makes them irregular work for a GPU. During training and prefill—the initial processing of a prompt—RAM-Net organizes this work into two stages: decode addresses, then execute by slot.

1. Parallel Address Decoding

Decode the read and write addresses of all tokens and heads in parallel. Each decoder computes the small digit softmaxes in registers, then extends and prunes Top-K prefixes one level at a time. Log-space sums replace products to avoid numerical underflow. The result is a set of integer slot addresses and their weights, without constructing the full slot distribution.

2. Slot-Parallel Execution

Turn these addresses into read and write events, group them by slot and order them by time. Different slots, and the vector dimensions within each slot, run in parallel; events within one slot follow the causal order of the recurrence. Each read contributes to its token's output. Slots without events are skipped, so state-update work follows the actual accesses rather than the full memory capacity.

These two stages organize training and prefill over a whole sequence. Autoregressive decoding handles one token at a time and accesses only that token's selected slots.

07Results

Quality and State Access

Evidence & trade-offs

0.4MActive state · 340M Top-826.2M total state elements
72.9S-NIAH average · 340M Top-32Three tasks · 1K–16K contexts
15.90Wiki. perplexity · 1.3B Top-16Lower is better

These tables compare RAM-Net with Transformer++ [4] and the recurrent baselines Mamba2 [5], Gated DeltaNet [6], GLA [7], and GSA [8] on language modeling, long-context retrieval, real-world recall, and per-step state access.

Language Modeling and Reasoning

Language Modeling and Reasoning — 340M and 1.3B models

↓ Lower is better. ↑ Higher is better. State sizes are in millions of elements.

Bold marks the best value; underlining marks the second-best distinct value within each model size, including ties.

RAM-Net achieves the lowest WikiText perplexity at both scales: 25.87 with Top-32 at 340M, and 15.90 with Top-16 at 1.3B, just below the 15.96 of Mamba2. Its commonsense reasoning remains competitive, with the highest average accuracy at 340M (50.9 for Top-32) and a tie with Mamba2 at 1.3B (57.8). Increasing Top-K consistently improves both perplexity and average accuracy, suggesting that accessing more slots per step strengthens language modeling.

S-NIAH: Retrieval Across Context Lengths

S-NIAH: Retrieval Across Context Lengths — 340M and 1.3B models

All scores use ↑ higher is better. Columns within each task give the context length.

Bold marks the best value; underlining marks the second-best distinct value within each model size, including ties.

The single-needle S-NIAH tasks from RULER [9] span 1K–16K tokens, up to four times the 4K training length. RAM-Net achieves the highest average at both scales: 72.9 with Top-32 at 340M, against 51.3 for the best baseline, and 70.1 with Top-16 at 1.3B, against 62.4. On S-NIAH-1, all RAM-Net variants stay near-perfect up to 16K, whereas Transformer++ fails beyond its training length and the recurrent baselines degrade as the context grows. S-NIAH-2 and S-NIAH-3 use diverse natural text as filler and are harder: at 1.3B, Transformer++ leads within its training length, while Top-16 performs best beyond it, although all models remain weak at 16K. Increasing Top-K consistently raises the average at both scales.

Real-World Recall

Real-World Recall — 340M and 1.3B models

↑ Higher is better. FDA, SWDE, and NQ are evaluated at context lengths of 512 and 1K; the remaining tasks use the full document.

Bold marks the best value; underlining marks the second-best distinct value within each model size, including ties.

These recall-intensive tasks [10] require extracting information from real documents. Transformer++ achieves the highest average at both scales, since its KV cache keeps every context token. Among recurrent models, RAM-Net attains the highest average at both scales: 32.0 with Top-32 at 340M and 40.6 with Top-8 at 1.3B. At 340M, it narrows the gap to Transformer++ from 8.2 points for the best baseline to 3.1 points.

At 340M, Top-8 accesses 0.4M of 26.2M state elements per step, about 1.5% of its total state. Even the largest settings, 340M Top-32 and 1.3B Top-16, access only 1.6M elements per step: at most one-fifth as much state as Gated DeltaNet and one-eighth as much as Mamba2 at the same scale.

Accuracy, State Access, and Speed

At 340M, moving from Top-8 to Top-32 increases active state from 0.4M to 1.6M elements while total state stays at 26.2M. Wiki. perplexity drops from 26.89 to 25.87, the S-NIAH average rises from 64.7 to 72.9, and the real-world recall average from 23.2 to 32.0. The trade-off is better results at a higher per-token access cost.

On the synthetic multi-query associative recall (MQAR) benchmark [11], RAM-Net achieves leading accuracy for its total state size, while accessing about one-tenth as much state per step as baselines with comparable accuracy.

We benchmark autoregressive decoding of the 340M recurrent models on a single NVIDIA RTX 3060. Each step is captured in a CUDA graph to exclude Python and per-kernel launch overhead from the comparison. At batch size 1, reading model weights dominates, so the models run at similar speed. As the batch grows, recurrent-state reads and updates become the bottleneck.

At batch size 64, RAM-Net Top-8 achieves 2.0× the throughput of the fastest compared baseline, despite having the largest total state among the compared recurrent models. It accesses the least state per step, decoupling decoding speed from total state size: a large memory supports strong S-NIAH retrieval, while sparse access keeps decoding fast. Increasing K trades throughput for higher accuracy.

Retrieval still depends on reliable address selection: an unselected slot cannot contribute to the current read. More slots increase capacity; a larger Top-K increases access. Their benefits depend on the task.
08Retrieval Traces

Following a Needle Through Memory

Interpretability · Token–slot–token paths

Attention maps are a common way to examine token-to-token interaction patterns in Transformers and linear-attention models. RAM-Net adds an address-level dimension: we can inspect both the relationship between two tokens and which specific slots mediate their communication. Its read and write addresses expose a token–slot–token path.

In the S-NIAH traces, the fact to retrieve (the needle) and distractor text are written mainly to different slot subsets. As the model processes subsequent distractors, the needle's contribution persists in those slots; answer-time reads then revisit them. This helps explain successful NIAH retrieval: relevant information is preserved with limited interference from the surrounding text, then accessed again through its addresses.

09Address Patterns

What Do Shared Addresses Reveal?

Interpretability · Learned associations

Slot-level traces provide a fine-grained probe of learned token relationships: which tokens connect, where their information is stored, and how a later token retrieves it. The two examples reveal recognizable patterns of word-piece recombination and punctuation pairing.

The tokenizer splits “tokenization” into “token” and “ization”. In the displayed head, the later piece reads from slots written by the earlier piece, reconnecting fragments of the same word through memory.

In “(white bird)”, the closing mark retrieves information left by the opening mark, despite the words in between. This suggests the head is tracking a punctuation pair through memory. The trace reveals both the token relationship and the specific slots that carry it.

10Access Patterns

Emergent Hotspots in Content Access

Emergent reuse · Hotspots and a long tail

Switch the head or corpus to compare access concentration; hover over a bar to compare cumulative slot and access shares.

The network's content-dependent reads and writes spontaneously form hotspots: many visits converge on a small set of learned content addresses, while many other addresses are visited less often, forming a long tail. Sparse access limits how many slots each step touches; this concentration reflects where the model chooses to route the input content.

Computers show a familiar pattern of hot data and a long tail of infrequent accesses. Seeing a similar pattern emerge in learned addressing suggests that concentrated reuse may be a common feature of computation: a broad memory serves diverse needs, while a small subset supplies much of the repeated work.

This concentration creates an opportunity for acceleration through reuse. Keeping frequently visited state near the execution units can let a small fast memory serve a large share of accesses. A larger, slower backing memory can then provide capacity without serving every operation.

11Hardware

Independent Slots, Local Execution

Multi-issue · Out-of-order · Local caching

Two properties point toward efficient hardware: independent slot operations and repeated access to hot state. Explicit integer addresses separate address generation from memory execution, while independent slots provide work that can be issued in parallel.

Independent slots offer multi-issue and local out-of-order execution, with causal order preserved within each slot. This suggests a design between CPU and GPU styles: simpler dependency control than a general-purpose CPU, with more scheduling freedom than a single lockstep GPU instruction stream.

The hotspots suggest a way to reduce DRAM traffic with local caching. Frequently reused state can stay in on-chip SRAM, and consecutive operations can retain it in registers. This offers a way to retain a large memory while serving repeated accesses close to the execution units.

RAM-Net

More room to remember. Less to access.