jan/src-tauri/plugins/tauri-plugin-llamacpp
Akarshan 34b254e2d8
fix: Improve KV cache estimation robustness
The KV cache size calculation in estimate_kv_cache_internal now includes a fallback mechanism for models that do not explicitly define key_length and value_length in the GGUF metadata.

If these attention keys are missing, the head dimension (and thus key/value length) is calculated using the formula embedding_length / total_heads. This improves robustness and compatibility with GGUF models that don't have the proper keys in metadata.

Also adds logging of the full model metadata for easier debugging of the estimation process.
2025-09-30 11:14:18 +05:30
..