RAM-NetLinear-Time Sequence Modeling
with Sparsely Addressable State

A sequence model stores information from past tokens and retrieves the relevant part at every step. Its efficiency therefore depends on two costs, the amount of history it stores and the amount it accesses per step.

Full attention keeps one key–value entry per token and compares each query against all of them, so both costs grow with sequence length. To mitigate this, KV compression and sparse attention reduce per-step access, but the cache typically keeps growing.

Instead, mainstream linear attention compresses history into a fixed-size state. However, every step reads and writes the entire state. Consequently, all tokens interfere within the same state, and any increase in state size directly increases per-step access.

To address both issues, RAM-Net divides this fixed-size state into many addressable slots, and each step reads and writes only a few selected ones. Therefore, tokens interfere only when they select the same slots, and the state can grow without increasing per-step access.

Each token produces a key to select write slots, a query to select read slots, and a value to store.

RAM-Net processes them in three stages. First, the Address Decoder maps the key and the query to sparse write and read addresses, which select a few slots and weight their contributions. Next, Cyclic Address Positional Embedding (CAPE) can shift these addresses according to token position, making them position-aware. Finally, Gated Sparse Update (GSU) writes the value into the selected write slots and leaves all other slots unchanged. The selected read slots are then combined by their weights, normalized, gated, and projected to produce the output.

Since each step accesses only the selected slots, the size of the state and the per-step access are set separately. The slot count M sets the size of the state, while K sets how many slots each read or write accesses. Neither grows with sequence length, so storage and per-token cost stay constant as the context grows.

Selecting K slots naively would score all M of them for every token, reintroducing a cost that grows with the state size. Instead, Product Softmax avoids this by composing the large address distribution from a few small ones.

Specifically, Product Softmax factors the slot count as M = d_p^U, so that each slot index can be written as U digits with d_p values each. The key or query is split into U parts, and a softmax over each part gives a distribution over the d_p values of one digit. The weight of a slot is then the product of the probabilities assigned to its digits. As a result, U d_p scores suffice to address all M slots. For example, RAM-Net uses M = 1024 = 4⁵ with only 20 scores.

The figure above shows how these digit distributions combine into a distribution over slots, from five equivalent perspectives.

Soft Radix Address

In base d_p, each slot index is a sequence of U digits. Instead of fixing every digit to select a single slot, Product Softmax gives each digit a distribution over its d_p values. Every slot is then weighted by the product of its digit probabilities.

Multilevel Decision Tree

Each digit forms one level of a tree, so every root-to-leaf path corresponds to one slot. A slot's weight is the product of the branch probabilities along its path. Since all nodes at a level share the same digit distribution, the most probable slots can be found level by level, at a cost that grows with the depth U rather than the number of leaves M.

Joint Distribution

Each digit corresponds to one axis of a U-dimensional grid, so each slot is a grid point weighted by the product of its probabilities along every axis. A confident digit confines the weight to a slice of the grid, whereas an uncertain digit spreads it along its axis. Thus, the highest-weight slots lie where these slices intersect, extending only along the axes of uncertain digits. The figure illustrates the case U = d_p = 4.

Regroupable Address

Since the weight is a product, any set of digits can be merged into a single coarser digit without changing any slot weight. Splitting the digits into two groups thus arranges the slots as a matrix whose weights form an outer product of two vectors. The figure shows two such groupings, 64 × 4 and 4 × 64, which trace different patterns over the same weights.

Waveform Modulation

Read along the slot index, each digit's distribution becomes a periodic step waveform. Low-order digits change at every slot and repeat quickly, whereas high-order digits change slowly and hold each value over long intervals. The address distribution is the product of these waveforms, with the slow ones setting a coarse envelope and the fast ones modulating it within.

All five views express one property, namely that a slot's weight factorizes into independent per-digit probabilities. This factorization enables an exact and efficient Top-K search. Following the tree, RAM-Net fixes the digits one level at a time and keeps only the K most probable partial addresses at each level. A dropped partial address already trails K others, and since all of them are multiplied by the same remaining digit probabilities, it can never overtake them. The search thus returns the exact Top-K while evaluating only K d_p candidates per level, rather than all M slots.

Product Softmax addresses slots by content alone, so identical content maps to the same slots at every position. Such addresses cannot tell where a token occurs relative to the current one.

Cyclic Address Positional Embedding (CAPE) adds this information by shifting each token's read and write addresses along the slot array according to its position. A read at step t then overlaps a write from step s according to their content and the distance t − s, rather than their absolute positions.

As a result, the same content written at different positions lands on different slots instead of overwriting the same ones. Since the slot array is finite, these shifts wrap around its end, which makes the embedding cyclic.

The shift only permutes slot indices, so each read and write still accesses K slots. RAM-Net applies CAPE to some heads and leaves the others content-only, without any positional shift. The CAPE heads capture relative-position patterns, whereas the content-only heads capture associations that hold regardless of distance.

Each write must balance a selected slot's history against the incoming value, while leaving all other slots unchanged. Unlike the forgetting gates of mainstream linear attention, which decay the entire state at every step, Gated Sparse Update (GSU) forgets and writes only at the selected slots.

For each slot i, GSU keeps its content S_t,i together with a write mass m_t,i, which measures how much write weight the slot has accumulated. A new write changes the content in proportion to its weight relative to this mass, so a heavily written slot changes slowly. The input-dependent scalar γ_t, computed from the current token by a learned projection, controls how much of the mass is retained, and thus how readily the slot accepts new values.

Given the write weight w_t,i and the incoming value v_t, the update proceeds in four steps.

1. Retention (α). A stronger write or a larger γ_t keeps less of the previous mass.
2. Mass (m). The new mass adds the current write weight to the retained mass.
3. Update rate (λ). The rate is the write weight relative to the new mass, with a small ε for numerical stability.
4. Content (S). The content moves toward v_t by a fraction λ of the remaining gap, so a small λ preserves history and a λ near 1 lets the new value dominate.

For an unselected slot, w_t,i = 0 gives α_t,i = 1 and λ_t,i = 0, so both its content and mass stay unchanged. For a selected slot, γ_t ranges the update between two extremes, averaging all past writes as γ_t → −∞ and overwriting with the newest value as γ_t → +∞.

Sparse access reduces memory traffic. To turn that saving into efficient GPU execution, RAM-Net organizes work around independent memory slots.

During training and prefill, the model computes Top-K addresses across tokens in parallel, then groups read and write events by slot. Each slot receives its own sequence of access events.

Different slots and their channels can run in parallel, while events within each slot preserve the causal order required by the recurrence. Slots with no events require no work.

During autoregressive decoding, the model directly accesses the slots selected by the current token. It does not scan the full recurrent state.

At 340M, Top-8 accesses 0.4M of 26.2M state elements—about 1.5% of its total state and roughly 1/32 of Mamba2’s 12.9M active state. At 1.3B, Top-8 accesses 0.8M of 52.4M elements; the same Top-K accounting gives 1.6M active elements for Top-16. These figures measure state access; decoding throughput is evaluated separately.

Where does a fact go, and what remains when the model needs it? These S-NIAH-1 and S-NIAH-3 traces follow needle information from writing to answer-time reading across 1,024 slots. Switch the head or example, or jump to Needle and Read to compare the two moments.

Colors track source contributions under the recorded GSU gains. They illustrate provenance, not the actual hidden vectors. Each head uses a fixed, reordered slot layout to expose spatial patterns. These selected examples help inspect retrieval behavior; they do not estimate its overall success rate.

Selected heads show shared write–read addresses between adjacent word pieces, or between opening and closing punctuation. The history matrix exposes these connections token by token; the slot view follows how information from the source token is retained and read.

These are exploratory patterns in selected heads and short texts. Address overlap offers a useful window into memory use, but does not by itself establish semantic matching or a causal role in the model’s output.