* feat: Improve llama.cpp argument handling and add device parsing tests
This commit refactors how arguments are passed to llama.cpp,
specifically by only adding arguments when their values differ from
their defaults. This reduces the verbosity of the command and prevents
potential conflicts or errors when llama.cpp's default behavior aligns
with the desired setting.
Additionally, new tests have been added for parsing device output from
llama.cpp, ensuring the accurate extraction of GPU information (ID,
name, total memory, and free memory). This improves the robustness of
device detection.
The following changes were made:
* **Remove redundant `--ctx-size` argument:** The `--ctx-size`
argument is now only explicitly added if `cfg.ctx_size` is greater
than 0.
* **Conditional argument adding for default values:**
* `--split-mode` is only added if `cfg.split_mode` is not empty
and not 'layer'.
* `--main-gpu` is only added if `cfg.main_gpu` is not undefined
and not 0.
* `--cache-type-k` is only added if `cfg.cache_type_k` is not 'f16'.
* `--cache-type-v` is only added if `cfg.cache_type_v` is not 'f16'
(when `flash_attn` is enabled) or not 'f32' (otherwise). This
also corrects the `flash_attn` condition.
* `--defrag-thold` is only added if `cfg.defrag_thold` is not 0.1.
* `--rope-scaling` is only added if `cfg.rope_scaling` is not
'none'.
* `--rope-scale` is only added if `cfg.rope_scale` is not 1.
* `--rope-freq-base` is only added if `cfg.rope_freq_base` is not 0.
* `--rope-freq-scale` is only added if `cfg.rope_freq_scale` is
not 1.
* **Add `parse_device_output` tests:** Comprehensive unit tests were
added to `src-tauri/src/core/utils/extensions/inference_llamacpp_extension/server.rs`
to validate the parsing of llama.cpp device output under various
scenarios, including multiple devices, single devices, different
backends (CUDA, Vulkan, SYCL), complex GPU names, and error
conditions.
* fixup cache_type_v comparision
* Fix: Llama.cpp server hangs on model load
Resolves an issue where the llama.cpp server would hang indefinitely when loading certain models, as described in the attached ticket. The server's readiness message was not being correctly detected, causing the application to stall.
The previous implementation used a line-buffered reader (BufReader::lines()) to process the stderr stream. This method proved to be unreliable for the specific output of the llama.cpp server.
This commit refactors the stderr handling logic to use a more robust, chunk-based approach (read_until(b'\n', ...)). This ensures that the output is processed as it arrives, reliably capturing critical status messages and preventing the application from hanging during model initialization.
Fixes: #6021
* Handle error gracefully with ServerError
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* Revert "Handle error gracefully with ServerError"
This reverts commit 267a8a8a3262fbe36a445a30b8b3ba9a39697643.
* Revert "Fix: Llama.cpp server hangs on model load"
This reverts commit 44e5447f82f0ae32b6db7ffb213025f130d655c4.
* Add more guards, refactor and fix error sending to FE
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
This commit introduces a significant restructuring of the documentation deployment and content strategy to support a gradual migration from Nextra to Astro.
- **New Astro Workflow (`jan-astro-docs.yml`)**: Implemented a new, separate GitHub Actions workflow to build and deploy the Astro site from the `/website` directory to a new subdomain (`v2.jan.ai`). This isolates the new site from the existing one, allowing for independent development and testing.
- **Removed Combined Workflow**: Deleted the previous, more complex combined workflow (`jan-combined-docs.yml`) and its associated test scripts to simplify the deployment process and eliminate routing conflicts.
- **Astro Config Update**: Simplified the Astro configuration (`astro.config.mjs`) by removing the conditional `base` path. The Astro site is now configured to deploy to the root of its own subdomain.
- **Mirrored Content**: Recreated the entire `/products` section from the Astro site within the Nextra site at `/docs/src/pages/products`. This provides content parity and a consistent user experience on both platforms during the transition period.
- **File Structure**: Established a clear, organized structure for platforms, models, and tools within the Nextra `products` directory.
- **Nextra Sidebar Fix**: Implemented the correct `_meta.json` structure for the new products section. Created nested meta files to build a collapsible sidebar, fixing the UI bug that caused duplicated navigation items.
- **"Coming Soon" Pages**: Added clear, concise "Coming Soon" and "In Development" banners and content for upcoming products like Jan V1, Mobile, Server, and native Tools, ensuring consistent messaging across both sites.
- **.gitignore**: Updated the root `.gitignore` to properly exclude build artifacts, caches, and environment files for both the Nextra (`/docs`) and Astro (`/website`) projects.
- **Repository Cleanup**: Removed temporary and unused files related to the previous combined deployment attempt.
This new architecture provides a stable, predictable, and low-risk path for migrating our documentation to Astro while ensuring the current production site remains unaffected.
