Why an FPGA language-model accelerator compresses its memory twice

VectorFLASH is an experimental language-model accelerator implemented on a Zynq UltraScale+ XCZU15EG FPGA. Its authors report support for contexts up to 128K tokens, using vector quantisation for both model weights and the key-value cache. The architecture tackles the storage and movement of two different kinds of data, rather than assuming that compressing the model solves the memory problem. ^[1]

Weights are set before inference, so they can be compressed in advance. A key-value cache grows as a conversation or document is processed, making its contents and its storage demand part of the running workload.

What a longer context adds to memory

In an autoregressive transformer, attention uses keys and values associated with earlier tokens. Keeping those tensors in a cache avoids calculating them again whenever another token is generated. Hugging Face’s implementation documentation describes cache tensors with dimensions for the batch, attention heads, sequence length and head dimension, repeated across the model’s layers. ^[2]

The space can become substantial without any change to the model weights. Consider an illustrative model with 32 layers, eight key-value heads per layer and 128 values per head, storing both keys and values at 16 bits. With a batch of one and 131,072 tokens, the cache occupies 16 GiB before allocation overhead. That follows from multiplying two tensors by 32 layers, eight heads, 128 values, two bytes and the sequence length.

This is a worked configuration, not the model used to benchmark VectorFLASH. Halving the stored precision would ideally halve that payload, while codebooks, scales and other metadata would add their own cost. A sliding-window architecture can also limit growth in relevant layers, so context length alone does not determine cache capacity for every model. ^[3]

Compression changes the arithmetic too

Vector quantisation represents a group of values using an index into a codebook of representative vectors. Keeping the indices can use fewer bits than storing every original value, but the accelerator must then turn those indices into useful computation. Work on CodeGEMM identifies repeated reconstruction of weights from codebook entries as a source of latency and cache pressure. A compact file format is little help if it has to be expanded into a large working representation before each operation. ^[5]

Earlier work on LUT-GEMM addresses that problem by using table-based operations for quantised matrix multiplication, avoiding a separate, expensive dequantisation step. Its reported results concern a GPU implementation and do not establish VectorFLASH’s performance. The shared architectural idea is to make the computation work with the compressed representation instead of paying to reconstruct conventional operands throughout the calculation. ^[4]

VectorFLASH’s weight path combines a codebook with Huffman coding, reporting an average 2.5-bit weight representation. Its online cache path uses a Hadamard rotation implemented with the DSP48E2’s 48-bit adder, and a lookup-based multiply-accumulate core balances work through codebook-entry replication. ^[1]

Static weights and newly generated cache entries therefore need different machinery. The former can be prepared ahead of time, while the latter must be encoded as inference proceeds. That online work uses time, logic and memory bandwidth, which have to be recovered through cheaper storage access or subsequent computation.

Software implementations show the same compromise. Hugging Face warns that cache quantisation can worsen latency when the context is short and sufficient memory is already available. That warning describes its software cache strategies, not a measurement of the FPGA, but it illustrates why compression should be judged across the whole operation rather than by bit count alone. ^[3]

The benchmark still needs its workload

The authors report 2.8–8.7 tokens per second and peak arithmetic throughput of 1.48 TOPS at 7.5 W. The public abstract does not identify the model, accuracy results or a complete mapping between context length and throughput, so the upper token rate cannot be assigned to the longest context. ^[1]

That missing mapping limits a comparison with another accelerator. Generating a token from a short prompt and generating one with a long cache place different demands on memory, even when the model is unchanged. Prompt processing also has a different workload from producing the next token, so a single generation-rate figure cannot describe both.

VectorFLASH demonstrates a hardware approach to keeping weights and context compressed during inference. Its public summary does not yet establish the accuracy, response time and total board power of a specified application running a named model.

Sources

• A-SSCC, VectorFLASH author abstract, paper 1092
• Hugging Face, key-value caching and tensor dimensions
• Hugging Face, cache strategies, sliding-window limits and quantisation overhead
• LUT-GEMM authors, quantised matrix multiplication using lookup operations, ICLR 2024 (background)
• CodeGEMM authors, codebook-based computation and dequantisation costs, December 2025 (background)

Image: Illustrative computer memory photograph, not the VectorFLASH FPGA or its cache implementation. Credit: Moe Bodden / Pexels. Image source. Pexels licence.