This change improves the robustness of the llama.cpp extension's server port selection.
Previously, the `getRandomPort()` method only checked for ports already in use by active sessions, which could lead to model load failures if the chosen port was occupied by another external process.
This change introduces a new Tauri command, `is_port_available`, which performs a system-level check to ensure the randomly selected port is truly free before attempting to start the llama-server. It also adds a retry mechanism with a maximum number of attempts (20,000) to find an available port, throwing an error if no suitable port is found within the specified range after all attempts.
This enhancement prevents port conflicts and improves the reliability and user experience of the llama.cpp extension within Jan.
Closes#5965
- Added rich Products section with detailed platform coverage
- Enhanced all documentation sections with improved formatting
- Added new images and visual content throughout
- Reorganized Local Server docs into main docs flow
- Removed .vscode settings and added api-server-ui.png asset
* fix: assistant with last used and fix metadata
* chore: revert instruction and desc
* chore: fix current assistant state
* chore: updae metadata message assistant
* chore: update test case
Previously, the `autoUnload` flag was not being updated when set via config,
causing models to be auto-unloaded regardless of the intended behavior.
This patch ensures the setting is respected at runtime.
- Removed docs/src/pages/products/ directory and all contents
- Removed products entry from docs/src/pages/_meta.json
- Reverted docs/package.json, docs/bun.lock, docs/yarn.lock to dev state
This creates a cleaner separation between:
- Current branch: Astro site changes only
- rp/nextra-product-section branch: Nextra product section changes only
This commit introduces a significant restructuring of the documentation deployment and content strategy to support a gradual migration from Nextra to Astro.
- **New Astro Workflow (`jan-astro-docs.yml`)**: Implemented a new, separate GitHub Actions workflow to build and deploy the Astro site from the `/website` directory to a new subdomain (`v2.jan.ai`). This isolates the new site from the existing one, allowing for independent development and testing.
- **Removed Combined Workflow**: Deleted the previous, more complex combined workflow (`jan-combined-docs.yml`) and its associated test scripts to simplify the deployment process and eliminate routing conflicts.
- **Astro Config Update**: Simplified the Astro configuration (`astro.config.mjs`) by removing the conditional `base` path. The Astro site is now configured to deploy to the root of its own subdomain.
- **Mirrored Content**: Recreated the entire `/products` section from the Astro site within the Nextra site at `/docs/src/pages/products`. This provides content parity and a consistent user experience on both platforms during the transition period.
- **File Structure**: Established a clear, organized structure for platforms, models, and tools within the Nextra `products` directory.
- **Nextra Sidebar Fix**: Implemented the correct `_meta.json` structure for the new products section. Created nested meta files to build a collapsible sidebar, fixing the UI bug that caused duplicated navigation items.
- **"Coming Soon" Pages**: Added clear, concise "Coming Soon" and "In Development" banners and content for upcoming products like Jan V1, Mobile, Server, and native Tools, ensuring consistent messaging across both sites.
- **.gitignore**: Updated the root `.gitignore` to properly exclude build artifacts, caches, and environment files for both the Nextra (`/docs`) and Astro (`/website`) projects.
- **Repository Cleanup**: Removed temporary and unused files related to the previous combined deployment attempt.
This new architecture provides a stable, predictable, and low-risk path for migrating our documentation to Astro while ensuring the current production site remains unaffected.
This commit addresses a race condition where, with "Auto-Unload Old Models" enabled, rapidly attempting to load multiple models could result in more than one model being loaded simultaneously.
Previously, the unloading logic did not account for models that were still in the process of loading when a new load operation was initiated. This allowed new models to start loading before the previous ones had fully completed their unload cycle.
To resolve this:
- A `loadingModels` map has been introduced to track promises for models currently in the loading state.
- The `load` method now checks if a model is already being loaded and, if so, returns the existing promise, preventing duplicate load operations for the same model.
- The `performLoad` method (which encapsulates the actual loading logic) now ensures that when `autoUnload` is active, it waits for any *other* models that are concurrently loading to finish before proceeding to unload all currently loaded models. This guarantees that the auto-unload mechanism properly unloads all models, including those initiated in quick succession, thereby preventing the race condition.
This fixes the issue where clicking the start button very fast on multiple models would bypass the auto-unload functionality.
* fix: migrate app settings to the new version
* fix: edge cases
* fix: migrate HF import model on Windows
* fix hardware page broken after downgraded
* test: correct test
* fix: backward compatible hardware info
This commit addresses a potential race condition that could lead to "connection errors" when unloading a llamacpp model.
The issue arose because the `activeSessions` map still has the session info of the model during unload. This could lead to "connection errors" when the backend is taking time to unload while there is an ongoing request to the model.
The fix involves:
1. **Deleting the `pid` from `activeSessions` before calling backend's unload:** This ensures that the model is cleared from the map before we start unloading.
2. **Failure handling**: If somehow the backend fails to unload, the session info for that model is added back to prevent any race conditions.
This commit improves the robustness and reliability of the unloading process by preventing potential conflicts.
* fix: Allow N-GPU Layers (NGL) to be set to 0 in llama.cpp
The `n_gpu_layers` (NGL) setting in the llama.cpp extension was incorrectly preventing users from disabling GPU layers by automatically defaulting to 100 when set to 0.
This was caused by a condition that only pushed `cfg.n_gpu_layers` if it was greater than 0 (`cfg.n_gpu_layers > 0`).
This commit updates the condition to `cfg.n_gpu_layers >= 0`, allowing 0 to be a valid and accepted value for NGL. This ensures that users can effectively disable GPU offloading when desired.
* fix: default ngl
---------
Co-authored-by: Louis <louis@jan.ai>
The 'Auto-Unload Old Models' setting in the llama.cpp extension failed to persist due to a typo in its key name within `settings.json`. The key was incorrectly `auto_unload_models` instead of `auto_unload`.
This commit corrects the key name to `auto_unload`, ensuring that user-configured changes to this setting are properly saved, retrieved, and persist across application restarts.
This resolves the issue where the setting would change and remain to its previous value after being changed.
* feat: Enhance Llama.cpp backend management with persistence
This commit introduces significant improvements to how the Llama.cpp extension manages and updates its backend installations, focusing on user preference persistence and smarter auto-updates.
Key changes include:
* **Persistent Backend Type Preference:** The extension now stores the user's preferred backend type (e.g., `cuda`, `cpu`, `metal`) in `localStorage`. This ensures that even after updates or restarts, the system attempts to use the user's previously selected backend type, if available.
* **Intelligent Auto-Update:** The auto-update mechanism has been refined to prioritize updating to the **latest version of the *currently selected backend type*** rather than always defaulting to the "best available" backend (which might change). This respects user choice while keeping the chosen backend type up-to-date.
* **Improved Initial Installation/Configuration:** For fresh installations or cases where the `version_backend` setting is invalid, the system now intelligently determines and installs the best available backend, then persists its type.
* **Refined Old Backend Cleanup:** The `removeOldBackends` function has been renamed to `removeOldBackend` and modified to specifically clean up *older versions of the currently selected backend type*, preventing the accumulation of unnecessary files while preserving other backend types the user might switch to.
* **Robust Local Storage Handling:** New private methods (`getStoredBackendType`, `setStoredBackendType`, `clearStoredBackendType`) are introduced to safely interact with `localStorage`, including error handling for potential `localStorage` access issues.
* **Version Filtering Utility:** A new utility `findLatestVersionForBackend` helps in identifying the latest available version for a specific backend type from a list of supported backends.
These changes provide a more stable, user-friendly, and maintainable backend management experience for the Llama.cpp extension.
Fixes: #5883
* fix: cortex models migration should be done once
* feat: Optimize Llama.cpp backend preference storage and UI updates
This commit refines the Llama.cpp extension's backend management by:
* **Optimizing `localStorage` Writes:** The system now only writes the backend type preference to `localStorage` if the new value is different from the currently stored one. This reduces unnecessary `localStorage` operations.
* **Ensuring UI Consistency on Initial Setup:** When a fresh installation or an invalid backend configuration is detected, the UI settings are now explicitly updated to reflect the newly determined `effectiveBackendString`, ensuring the displayed setting matches the active configuration.
These changes improve performance by reducing redundant storage operations and enhance user experience by maintaining UI synchronization with the backend state.
* Revert "fix: provider settings should be refreshed on page load (#5887)"
This reverts commit ce6af62c7df4a7e7ea8c0896f307309d6bf38771.
* fix: add loader version backend llamacpp
* fix: wrong key name
* fix: model setting issues
* fix: virtual dom hub
* chore: cleanup
* chore: hide device ofload setting
---------
Co-authored-by: Louis <louis@jan.ai>
Co-authored-by: Faisal Amir <urmauur@gmail.com>
* feat: add support for querying available backend devices
This change introduces a new `get_devices` method to the `llamacpp_extension` engine that allows the frontend to query and display a list of available devices (e.g., Vulkan, CUDA, SYCL) from the compiled `llama-server` binary.
* Added `DeviceList` interface to represent GPU/device metadata.
* Implemented `getDevices(): Promise<DeviceList[]>` method.
* Splits `version/backend`, ensures backend is ready.
* Invokes the new Tauri command `get_devices`.
* Introduced a new `get_devices` Tauri command.
* Parses `llama-server --list-devices` output to extract available devices with memory info.
* Introduced `DeviceInfo` struct (`id`, `name`, `mem`, `free`) and exposed it via serialization.
* Robust parsing logic using string processing (non-regex) to locate memory stats.
* Registered the new command in the `tauri::Builder` in `lib.rs`.
* Fixed logic to correctly parse multiple devices from the llama-server output.
* Handles common failure modes: binary not found, malformed memory info, etc.
This sets the foundation for device selection, memory-aware model loading, and improved diagnostics in Jan AI engine setup flows.
* Update extensions/llamacpp-extension/src/index.ts
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
---------
Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
* ci: update artifact name for Linux and Windows build
* ci: enhance logic for naming convention for mac, linux and windows builds
* fix: resolve nested template expression in artifact names
