Ceva adds cache and model support to its next NeuPro NPUs
Ceva has added support for paged key value caches, mixed precision and mixture of experts models to its next NeuPro NPU portfolio. Those features address workloads in which keeping arithmetic units supplied with data can be as demanding as performing the calculations.
The Nano, Prime, Max and Ultra range shares an architecture and software environment, extending from embedded applications into AI infrastructure. Selected customers have early access, with general availability planned for the second quarter of 2027.
Ceva supplies licensable IP, so the memory system and other choices made by the chip developer will help determine the eventual performance. A NeuPro configuration integrated into one chip need not behave like the same compute capability behind a different memory interface.
The cache grows with the conversation
In transformer inference, a key value cache retains information from earlier tokens, avoiding repeated computation as more tokens are processed. Longer context increases the data that must be stored and accessed, which can leave an accelerator waiting for memory even when it has spare arithmetic capacity.
Paged handling gives the implementation a way to allocate that storage without reserving one large fixed block for every sequence. It still has to retrieve the data efficiently. The access pattern and scheduling determine how much of the memory bandwidth becomes productive work.
Mixture of experts models create a different kind of unevenness. Selecting only some expert networks reduces the arithmetic relative to activating them all, but the chosen weights must still reach the compute units in time. Changing the selected experts can make that traffic less regular than a dense layer, so a reduction in operations need not produce the same reduction in memory transfers.
Ceva’s FP4 and FP8 support can reduce the bits that have to move, where the model retains sufficient accuracy at those precisions. Mixed precision allows less tolerant operations to use another format rather than imposing one compromise across the whole model.
The compiler has to keep the hardware busy
Mapping operations and data formats onto each configuration falls to the software environment. An unsupported operation that sends work elsewhere, or a poor arrangement of intermediate data, can absorb a gain expected from the NPU. Two implementations with similar peak throughput can consequently take very different times to run the same model.
The common software environment could make that easier to manage across the range. A smaller configuration may favour a tightly quantised working set held nearby, while a larger one has enough bandwidth to sustain more parallel work. Supporting those choices without a separate model conversion project for every device would save integration work as well as making the hardware easier to compare.
