huggingface
A memory layer that stores and reloads language-model KV states on an encrypted local disk
SITE CAPTUREBased on information published on the service's website. Selection and review policy
WHAT IT DOES
What does it do?
The provided material is a Hugging Face paper page describing experiments with the galahad-kv memory layer. It stores each block's KV state on an encrypted local NVMe disk and later restores it unchanged. The paper says it was tested on 50,000,000 tokens of public text using vLLM and a single NVIDIA H100.
KEY FEATURES
Key features
- Store KV states in blocks of about 16,000 tokens on an encrypted local NVMe disk
- Reload stored blocks without recomputing their KV states
- Tested on 50,000,000 tokens of public text with vLLM