Addedd changelog, reorganized docs, added MCP examples, and updated different pages.
This commit is contained in:
Ramon Perez
parent
44082846d5
commit
2f77c6ba9c
BIN
docs/public/assets/images/changelog/release_v0_6_5.gif
Normal file
BIN
docs/public/assets/images/changelog/release_v0_6_5.gif
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 14 MiB |
26
docs/src/pages/changelog/2025-07-17-responsive-ui.mdx
Normal file
26
docs/src/pages/changelog/2025-07-17-responsive-ui.mdx
Normal file
@ -0,0 +1,26 @@
|
||||
---
|
||||
title: "Jan v0.6.5 brings responsive UI and MCP examples!"
|
||||
version: 0.6.5
|
||||
description: "New MCP examples, updated pages, and bug fixes!"
|
||||
date: 2025-07-17
|
||||
ogImage: "/assets/images/changelog/release_v0_6_5.gif"
|
||||
---
|
||||
|
||||
import ChangelogHeader from "@/components/Changelog/ChangelogHeader"
|
||||
|
||||
<ChangelogHeader title="Jan v0.6.5 brings responsive UI and MCP examples!" date="2025-07-17" ogImage="/assets/images/changelog/release_v0_6_5.gif" />
|
||||
|
||||
## Highlights 🎉
|
||||
|
||||
Jan v0.6.5 brings responsive UI improvements, enhanced model provider management, and better Linux compatibility alongside
|
||||
new MCP examples.
|
||||
|
||||
- Support responsive UI on Jan
|
||||
- Rework of Model Providers UI
|
||||
- Bump version of llama.cpp
|
||||
- Fix the bug where fetching models from custom provider can causes app to crash
|
||||
- AppImage can now render on wayland + mesa
|
||||
|
||||
Update your Jan or [download the latest](https://jan.ai/).
|
||||
|
||||
For more details, see the [GitHub release notes](https://github.com/menloresearch/jan/releases/tag/v0.6.5).
|
||||
BIN
docs/src/pages/docs/_assets/e2b-key.png
Normal file
BIN
docs/src/pages/docs/_assets/e2b-key.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 266 KiB |
BIN
docs/src/pages/docs/_assets/e2b-key1.png
Normal file
BIN
docs/src/pages/docs/_assets/e2b-key1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 247 KiB |
BIN
docs/src/pages/docs/_assets/e2b-key2.png
Normal file
BIN
docs/src/pages/docs/_assets/e2b-key2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 314 KiB |
BIN
docs/src/pages/docs/_assets/e2b-key3.png
Normal file
BIN
docs/src/pages/docs/_assets/e2b-key3.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 308 KiB |
BIN
docs/src/pages/docs/_assets/e2b-key4.png
Normal file
BIN
docs/src/pages/docs/_assets/e2b-key4.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 500 KiB |
BIN
docs/src/pages/docs/_assets/e2b-key5.png
Normal file
BIN
docs/src/pages/docs/_assets/e2b-key5.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 314 KiB |
BIN
docs/src/pages/docs/_assets/e2b-key6.png
Normal file
BIN
docs/src/pages/docs/_assets/e2b-key6.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 558 KiB |
BIN
docs/src/pages/docs/_assets/e2b-key7.png
Normal file
BIN
docs/src/pages/docs/_assets/e2b-key7.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 649 KiB |
BIN
docs/src/pages/docs/_assets/exa.png
Normal file
BIN
docs/src/pages/docs/_assets/exa.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 56 KiB |
BIN
docs/src/pages/docs/_assets/exa1.png
Normal file
BIN
docs/src/pages/docs/_assets/exa1.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 340 KiB |
BIN
docs/src/pages/docs/_assets/exa2.png
Normal file
BIN
docs/src/pages/docs/_assets/exa2.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 507 KiB |
BIN
docs/src/pages/docs/_assets/exa3.png
Normal file
BIN
docs/src/pages/docs/_assets/exa3.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 468 KiB |
BIN
docs/src/pages/docs/_assets/exa4.png
Normal file
BIN
docs/src/pages/docs/_assets/exa4.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 662 KiB |
BIN
docs/src/pages/docs/_assets/mcp-server.png
Normal file
BIN
docs/src/pages/docs/_assets/mcp-server.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 88 KiB |
@ -24,6 +24,8 @@
|
||||
"type": "separator"
|
||||
},
|
||||
"llama-cpp": "Local AI Engine",
|
||||
"model-parameters": "Model Parameters",
|
||||
|
||||
"privacy-policy": {
|
||||
"type": "page",
|
||||
"display": "hidden",
|
||||
@ -45,6 +47,5 @@
|
||||
"settings": "Settings",
|
||||
"data-folder": "Jan Data Folder",
|
||||
"troubleshooting": "Troubleshooting",
|
||||
"model-parameters": "Model Parameters",
|
||||
"privacy": "Privacy"
|
||||
}
|
||||
|
||||
@ -10,9 +10,5 @@
|
||||
"linux": {
|
||||
"title": "Linux",
|
||||
"href": "/docs/desktop/linux"
|
||||
},
|
||||
"beta": {
|
||||
"title": "BETA apps",
|
||||
"href": "/docs/desktop/beta"
|
||||
}
|
||||
}
|
||||
|
||||
@ -1,81 +0,0 @@
|
||||
---
|
||||
title: Beta Builds
|
||||
description: Try out the latest features of Jan through our beta builds. Get early access to new features and help shape the future of Jan.
|
||||
keywords:
|
||||
[
|
||||
Jan,
|
||||
beta,
|
||||
preview,
|
||||
early-access,
|
||||
testing,
|
||||
development,
|
||||
experimental-features,
|
||||
feedback,
|
||||
support
|
||||
]
|
||||
---
|
||||
|
||||
import { Tabs, Callout } from 'nextra/components'
|
||||
|
||||
# Jan Beta Builds
|
||||
|
||||
Welcome to Jan's beta program! Beta builds give you early access to new features and improvements before they reach the stable release. By participating in the beta program, you can help shape the future of Jan by providing feedback and reporting issues.
|
||||
|
||||
<Callout type="warning">
|
||||
Beta builds may contain bugs and unstable features. We recommend using the stable release for production environments.
|
||||
</Callout>
|
||||
|
||||
## Download Beta Builds
|
||||
|
||||
Get the latest beta builds for your platform:
|
||||
|
||||
<Tabs items={['macOS', 'Windows', 'Linux']}>
|
||||
<Tabs.Tab>
|
||||
- [MacOS Universal](https://app.jan.ai/download/beta/mac-universal)
|
||||
</Tabs.Tab>
|
||||
|
||||
<Tabs.Tab>
|
||||
- [Windows Installer](https://app.jan.ai/download/beta/win-x64)
|
||||
</Tabs.Tab>
|
||||
|
||||
<Tabs.Tab>
|
||||
- [jan.deb](https://app.jan.ai/download/beta/linux-amd64-deb)
|
||||
- [jan.AppImage](https://app.jan.ai/download/beta/linux-amd64-appimage)
|
||||
</Tabs.Tab>
|
||||
</Tabs>
|
||||
|
||||
## Getting Help & Support
|
||||
|
||||
### Discord Community
|
||||
Join our Discord community to:
|
||||
- Get help with beta builds
|
||||
- Discuss new features
|
||||
- Connect with other beta testers
|
||||
- Share feedback directly with the team
|
||||
|
||||
[Join our Discord Server](https://discord.com/channels/1107178041848909847/1149558035971321886) or report directly to [#🆘|jan-help](https://discord.com/channels/1107178041848909847/1192090449725358130)
|
||||
|
||||
### GitHub Issues
|
||||
Found a bug or have a feature request? Let us know on GitHub:
|
||||
|
||||
1. Check if the issue has already been reported
|
||||
2. Create a new issue with:
|
||||
- Clear description of the problem
|
||||
- Steps to reproduce
|
||||
- Expected vs actual behavior
|
||||
- System information
|
||||
- Screenshots if applicable
|
||||
|
||||
[File an Issue on GitHub](https://github.com/menloresearch/jan/issues)
|
||||
|
||||
## Beta Program Benefits
|
||||
|
||||
- Early access to new features
|
||||
- Direct influence on product development
|
||||
- Opportunity to shape Jan's future
|
||||
- Access to beta-specific support channels
|
||||
- Regular updates with the latest improvements
|
||||
|
||||
<Callout type="info">
|
||||
Your feedback is invaluable in helping us improve Jan. Thank you for being part of our beta testing community!
|
||||
</Callout>
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: llama.cpp
|
||||
description: A step-by-step guide on how to customize the llama.cpp engine.
|
||||
title: llama.cpp Engine
|
||||
description: Understand and configure Jan's local AI engine for running models on your hardware.
|
||||
keywords:
|
||||
[
|
||||
Jan,
|
||||
@ -27,139 +27,181 @@ import { Tabs } from 'nextra/components'
|
||||
import { Callout, Steps } from 'nextra/components'
|
||||
import { Settings, EllipsisVertical, Plus, FolderOpen, Pencil } from 'lucide-react'
|
||||
|
||||
# Local Model Management
|
||||
# Local AI Engine (llama.cpp)
|
||||
|
||||
## Overview
|
||||
Jan uses **llama.cpp** for running local AI models. You can find its settings in **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Local Engine** > **llama.cpp**:
|
||||
## What is llama.cpp?
|
||||
|
||||
llama.cpp is the engine that runs AI models locally on your computer. Think of it as the software that takes an AI model file and makes it actually work on your hardware - whether that's your CPU, graphics card, or Apple's M-series chips.
|
||||
|
||||
Originally created by Georgi Gerganov, llama.cpp is designed to run large language models efficiently on consumer hardware without requiring specialized AI accelerators or cloud connections.
|
||||
|
||||
## Why This Matters
|
||||
|
||||
**Privacy**: Your conversations never leave your computer
|
||||
**Cost**: No monthly subscription fees or API costs
|
||||
**Speed**: No internet required once models are downloaded
|
||||
**Control**: Choose exactly which models to run and how they behave
|
||||
|
||||
## Accessing Engine Settings
|
||||
|
||||
Find llama.cpp settings at **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Local Engine** > **llama.cpp**:
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
These settings are for advanced users, you would want to check these settings when:
|
||||
- Your AI models are running slowly or not working
|
||||
- You've installed new hardware (like a graphics card)
|
||||
- You want to tinker & test performance with different [backends](/docs/local-engines/llama-cpp#available-backends)
|
||||
<Callout type="info">
|
||||
These are advanced settings. You typically only need to adjust them if models aren't working properly or you want to optimize performance for your specific hardware.
|
||||
</Callout>
|
||||
|
||||
## Engine Version and Updates
|
||||
- **Engine Version**: View current version of llama.cpp engine
|
||||
- **Check Updates**: Verify if a newer version is available & install available updates when it's available
|
||||
## Engine Management
|
||||
|
||||
| Feature | What It Does | When You Need It |
|
||||
|---------|-------------|------------------|
|
||||
| **Engine Version** | Shows which version of llama.cpp you're running | Check compatibility with newer models |
|
||||
| **Check Updates** | Downloads newer engine versions | When new models require updated engine |
|
||||
| **Backend Selection** | Choose the version optimized for your hardware | After installing new graphics cards or when performance is poor |
|
||||
|
||||
## Available Backends
|
||||
## Hardware Backends
|
||||
|
||||
Jan offers different backend variants for **llama.cpp** based on your operating system, you can:
|
||||
- Download different backends as needed
|
||||
- Switch between backends for different hardware configurations
|
||||
- View currently installed backends in the list
|
||||
Jan offers different backend versions optimized for your specific hardware. Think of these as different "drivers" - each one is tuned for particular processors or graphics cards.
|
||||
|
||||
<Callout type="warning">
|
||||
Choose the backend that matches your hardware. Using the wrong variant may cause performance issues or prevent models from loading.
|
||||
Using the wrong backend can make models run slowly or fail to load. Pick the one that matches your hardware.
|
||||
</Callout>
|
||||
|
||||
<Tabs items={['Windows', 'Linux', 'macOS']}>
|
||||
|
||||
<Tabs.Tab>
|
||||
### CUDA Support (NVIDIA GPUs)
|
||||
- `llama.cpp-avx-cuda-11-7`
|
||||
- `llama.cpp-avx-cuda-12-0`
|
||||
- `llama.cpp-avx2-cuda-11-7`
|
||||
- `llama.cpp-avx2-cuda-12-0`
|
||||
- `llama.cpp-avx512-cuda-11-7`
|
||||
- `llama.cpp-avx512-cuda-12-0`
|
||||
- `llama.cpp-noavx-cuda-11-7`
|
||||
- `llama.cpp-noavx-cuda-12-0`
|
||||
|
||||
### CPU Only
|
||||
- `llama.cpp-avx`
|
||||
- `llama.cpp-avx2`
|
||||
- `llama.cpp-avx512`
|
||||
- `llama.cpp-noavx`
|
||||
### NVIDIA Graphics Cards (Recommended for Speed)
|
||||
Choose based on your CUDA version (check NVIDIA Control Panel):
|
||||
|
||||
### Other Accelerators
|
||||
- `llama.cpp-vulkan`
|
||||
**For CUDA 12.0:**
|
||||
- `llama.cpp-avx2-cuda-12-0` (most common)
|
||||
- `llama.cpp-avx512-cuda-12-0` (newer Intel/AMD CPUs)
|
||||
|
||||
**For CUDA 11.7:**
|
||||
- `llama.cpp-avx2-cuda-11-7` (most common)
|
||||
- `llama.cpp-avx512-cuda-11-7` (newer Intel/AMD CPUs)
|
||||
|
||||
### CPU Only (No Graphics Card Acceleration)
|
||||
- `llama.cpp-avx2` (most modern CPUs)
|
||||
- `llama.cpp-avx512` (newer Intel/AMD CPUs)
|
||||
- `llama.cpp-avx` (older CPUs)
|
||||
- `llama.cpp-noavx` (very old CPUs)
|
||||
|
||||
### Other Graphics Cards
|
||||
- `llama.cpp-vulkan` (AMD, Intel Arc, some others)
|
||||
|
||||
<Callout type="info">
|
||||
- For detailed hardware compatibility, please visit our guide for [Windows](/docs/desktop/windows#compatibility).
|
||||
- AVX, AVX2, and AVX-512 are CPU instruction sets. For best performance, use the most advanced instruction set your CPU supports.
|
||||
- CUDA versions should match your installed NVIDIA drivers.
|
||||
**Quick Test**: Start with `avx2-cuda-12-0` if you have an NVIDIA card, or `avx2` for CPU-only. If it doesn't work, try the `avx` variant.
|
||||
</Callout>
|
||||
|
||||
</Tabs.Tab>
|
||||
|
||||
<Tabs.Tab>
|
||||
### CUDA Support (NVIDIA GPUs)
|
||||
- `llama.cpp-avx-cuda-11-7`
|
||||
- `llama.cpp-avx-cuda-12-0`
|
||||
- `llama.cpp-avx2-cuda-11-7`
|
||||
- `llama.cpp-avx2-cuda-12-0`
|
||||
- `llama.cpp-avx512-cuda-11-7`
|
||||
- `llama.cpp-avx512-cuda-12-0`
|
||||
- `llama.cpp-noavx-cuda-11-7`
|
||||
- `llama.cpp-noavx-cuda-12-0`
|
||||
|
||||
### NVIDIA Graphics Cards
|
||||
Same CUDA options as Windows:
|
||||
- `llama.cpp-avx2-cuda-12-0` (most common)
|
||||
- `llama.cpp-avx2-cuda-11-7` (older drivers)
|
||||
|
||||
### CPU Only
|
||||
- `llama.cpp-avx`
|
||||
- `llama.cpp-avx2`
|
||||
- `llama.cpp-avx512`
|
||||
- `llama.cpp-noavx`
|
||||
- `llama.cpp-avx2` (most modern CPUs)
|
||||
- `llama.cpp-avx512` (newer Intel/AMD CPUs)
|
||||
- `llama.cpp-arm64` (ARM processors like Raspberry Pi)
|
||||
|
||||
### Other Accelerators
|
||||
- `llama.cpp-vulkan`
|
||||
- `llama.cpp-arm64`
|
||||
|
||||
<Callout type="info">
|
||||
- For detailed hardware compatibility, please visit our guide for [Linux](docs/desktop/linux).
|
||||
- AVX, AVX2, and AVX-512 are CPU instruction sets. For best performance, use the most advanced instruction set your CPU supports.
|
||||
- CUDA versions should match your installed NVIDIA drivers.
|
||||
</Callout>
|
||||
### Other Graphics Cards
|
||||
- `llama.cpp-vulkan` (AMD, Intel graphics)
|
||||
|
||||
</Tabs.Tab>
|
||||
|
||||
<Tabs.Tab>
|
||||
### Apple Silicon
|
||||
- `llama.cpp-mac-arm64`: For M1/M2/M3 Macs
|
||||
|
||||
### Intel
|
||||
- `llama.cpp-mac-amd64`: For Intel-based Macs
|
||||
### Apple Silicon (M1/M2/M3/M4)
|
||||
- `llama.cpp-mac-arm64` (recommended)
|
||||
|
||||
### Intel Macs
|
||||
- `llama.cpp-mac-amd64`
|
||||
|
||||
<Callout type="info">
|
||||
For detailed hardware compatibility, please visit our guide for [Mac](/docs/desktop/mac#compatibility).
|
||||
Apple Silicon Macs automatically use the GPU through Metal - no additional setup needed.
|
||||
</Callout>
|
||||
|
||||
|
||||
</Tabs.Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
## Performance Settings
|
||||
|
||||
These control how efficiently models run:
|
||||
|
||||
| Setting | Description | Default |
|
||||
|---------|-------------|---------|
|
||||
| **Continuous Batching** | - Enables processing multiple requests simultaneously<br></br>- Improves throughput for concurrent operations | Enabled |
|
||||
| **Parallel Operations** | - Number of prompts to run in parallel<br></br>- Affects model inference speed | 4 |
|
||||
| **CPU Threads** | - Number of CPU cores to use when running without GPU<br></br>- Higher thread counts may improve performance but increase CPU usage | Auto-detected based on your system's capabilities |
|
||||
|
||||
|
||||
| Setting | What It Does | Recommended Value | Impact |
|
||||
|---------|-------------|------------------|---------|
|
||||
| **Continuous Batching** | Process multiple requests at once | Enabled | Faster when using multiple tools or having multiple conversations |
|
||||
| **Parallel Operations** | How many requests to handle simultaneously | 4 | Higher = more multitasking, but uses more memory |
|
||||
| **CPU Threads** | How many processor cores to use | Auto-detected | More threads can speed up CPU processing |
|
||||
|
||||
## Memory Settings
|
||||
|
||||
| Setting | Description | Default |
|
||||
|---------|-------------|---------|
|
||||
| **Flash Attention** | - Optimizes attention computation<br></br>- Reduces memory usage<br></br>- Recommended for most cases | Enabled |
|
||||
| **Caching** | - Enable to store recent prompts and responses<br></br>- Improves response time for repeated prompts | Enabled |
|
||||
| **KV Cache Type** | - KV cache implementation type; controls memory usage and precision trade-off<br></br>- Options:<br></br>• f16 (most stable)<br></br>• q8_0 (balanced)<br></br>• q4_0 (lowest memory) | f16 |
|
||||
| **mmap** | - Enables memory-mapped model loading<br></br>- Reduces memory usage<br></br>- Recommended for large models | Enabled |
|
||||
| **Context Shift** | - Automatically shifts the context window when the model is unable to process the entire prompt<br/> - Ensures that the most relevant information is always included <br/> - Recommended for long conversations and multiple tool calls | Disabled |
|
||||
These control how models use your computer's memory:
|
||||
|
||||
| Setting | What It Does | Recommended Value | When to Change |
|
||||
|---------|-------------|------------------|----------------|
|
||||
| **Flash Attention** | More efficient memory usage | Enabled | Leave enabled unless you have problems |
|
||||
| **Caching** | Remember recent conversations | Enabled | Speeds up follow-up questions |
|
||||
| **KV Cache Type** | Memory precision trade-off | f16 | Change to q8_0 or q4_0 if running out of memory |
|
||||
| **mmap** | Load models more efficiently | Enabled | Helps with large models |
|
||||
| **Context Shift** | Handle very long conversations | Disabled | Enable for very long chats or multiple tool calls |
|
||||
|
||||
## Best Practices
|
||||
- Start with default settings
|
||||
- Adjust based on your hardware capabilities
|
||||
- Monitor system performance
|
||||
- Test changes with your specific use case
|
||||
### KV Cache Types Explained
|
||||
- **f16**: Most stable, uses more memory
|
||||
- **q8_0**: Balanced memory usage and quality
|
||||
- **q4_0**: Uses least memory, slight quality loss
|
||||
|
||||
## Troubleshooting Common Issues
|
||||
|
||||
**Models won't load:**
|
||||
- Try a different backend (switch from CUDA to CPU or vice versa)
|
||||
- Check if you have enough RAM/VRAM
|
||||
- Update to latest engine version
|
||||
|
||||
**Very slow performance:**
|
||||
- Make sure you're using GPU acceleration (CUDA/Metal/Vulkan backend)
|
||||
- Increase GPU Layers in model settings
|
||||
- Close other memory-intensive programs
|
||||
|
||||
**Out of memory errors:**
|
||||
- Reduce Context Size in model settings
|
||||
- Switch KV Cache Type to q8_0 or q4_0
|
||||
- Try a smaller model variant
|
||||
|
||||
**Random crashes:**
|
||||
- Switch to a more stable backend (try avx instead of avx2)
|
||||
- Disable overclocking if you have it enabled
|
||||
- Update graphics drivers
|
||||
|
||||
## Quick Setup Guide
|
||||
|
||||
**For most users:**
|
||||
1. Use the default backend that Jan installs
|
||||
2. Leave all performance settings at defaults
|
||||
3. Only adjust if you experience problems
|
||||
|
||||
**If you have an NVIDIA graphics card:**
|
||||
1. Download the appropriate CUDA backend
|
||||
2. Make sure GPU Layers is set high in model settings
|
||||
3. Enable Flash Attention
|
||||
|
||||
**If models are too slow:**
|
||||
1. Check you're using GPU acceleration
|
||||
2. Try enabling Continuous Batching
|
||||
3. Close other applications using memory
|
||||
|
||||
**If running out of memory:**
|
||||
1. Change KV Cache Type to q8_0
|
||||
2. Reduce Context Size in model settings
|
||||
3. Try a smaller model
|
||||
|
||||
<Callout type="info">
|
||||
Performance impact varies by hardware, model size, and usage patterns.
|
||||
Most users can run Jan successfully without changing any of these settings. The defaults are chosen to work well on typical hardware.
|
||||
</Callout>
|
||||
|
||||
290
docs/src/pages/docs/mcp-examples/data-analysis/e2b.mdx
Normal file
290
docs/src/pages/docs/mcp-examples/data-analysis/e2b.mdx
Normal file
@ -0,0 +1,290 @@
|
||||
---
|
||||
title: E2B Code Sandbox
|
||||
description: Execute Python code securely in isolated sandbox environments with E2B.
|
||||
keywords:
|
||||
[
|
||||
Jan,
|
||||
MCP,
|
||||
Model Context Protocol,
|
||||
E2B,
|
||||
code execution,
|
||||
sandbox,
|
||||
data analysis,
|
||||
Python,
|
||||
secure computing,
|
||||
tool calling,
|
||||
]
|
||||
---
|
||||
|
||||
import { Callout, Steps } from 'nextra/components'
|
||||
|
||||
# E2B Code Sandbox MCP
|
||||
|
||||
E2B MCP provides isolated Python execution environments. Your AI can run actual code instead of just describing what code might do.
|
||||
|
||||
The real value emerges when you combine secure remote execution with Jan's flexible model selection. You can use
|
||||
local models for conversation and reasoning while offloading actual computation to E2B's sandboxes. This means you
|
||||
get the privacy and control of local models plus the computational power of cloud infrastructure, without the
|
||||
complexity of managing Python environments or dependencies locally.
|
||||
|
||||
## Setup
|
||||
|
||||
### Prerequisites
|
||||
|
||||
- Jan with MCP enabled
|
||||
- E2B API key from [e2b.dev](https://e2b.dev/)
|
||||
- Node.js installed
|
||||
- Model with tool calling support
|
||||
|
||||
### Configuration
|
||||
|
||||
1. **Enable MCP**: Go to **Settings** > **MCP Servers**, toggle **Allow All MCP Tool Permission** ON
|
||||
|
||||
|
||||
|
||||
<Callout type="info">
|
||||
Don't forget that MCP gets enabled once you turn on Experimental Features in Jan's General settings.
|
||||
</Callout>
|
||||
|
||||
2. **Get API Key**: Sign up at [e2b.dev](https://e2b.dev/), generate an API key
|
||||
|
||||
|
||||
|
||||
Add a meaningful name to your key.
|
||||
|
||||
|
||||
|
||||
3. **Add MCP Server**: Click `+` in MCP Servers section
|
||||
|
||||
Configure:
|
||||
- **Server Name**: `e2b-server`
|
||||
- **Command**: `npx`
|
||||
- **Arguments**: `@e2b/mcp-server`
|
||||
- **Environment Variables**:
|
||||
- Key: `E2B_API_KEY`
|
||||
- Value: `your-api-key`
|
||||
|
||||
|
||||
|
||||
4. **Verify**: Check server shows as active
|
||||
|
||||
|
||||
|
||||
|
||||
## Pre-installed Libraries
|
||||
|
||||
The sandbox includes these packages by default:
|
||||
|
||||
**Data Analysis & Science:**
|
||||
- `pandas` (1.5.3) - Data manipulation
|
||||
- `numpy` (1.26.4) - Numerical computing
|
||||
- `scipy` (1.12.0) - Scientific computing
|
||||
- `scikit-learn` (1.4.1) - Machine learning
|
||||
- `sympy` (1.12) - Symbolic mathematics
|
||||
|
||||
**Visualization:**
|
||||
- `matplotlib` (3.8.3) - Static plots
|
||||
- `seaborn` (0.13.2) - Statistical visualization
|
||||
- `plotly` (5.19.0) - Interactive charts
|
||||
- `bokeh` (3.3.4) - Web-ready visualizations
|
||||
|
||||
**Data Processing:**
|
||||
- `requests` (2.26.0) - HTTP requests
|
||||
- `beautifulsoup4` (4.12.3) - HTML/XML parsing
|
||||
- `openpyxl` (3.1.2) - Excel files
|
||||
- `python-docx` (1.1.0) - Word documents
|
||||
|
||||
**Text & NLP:**
|
||||
- `nltk` (3.8.1) - Natural language processing
|
||||
- `spacy` (3.7.4) - Advanced NLP
|
||||
- `textblob` (0.18.0) - Text processing
|
||||
- `gensim` (4.3.2) - Topic modeling
|
||||
|
||||
**Image & Audio:**
|
||||
- `opencv-python` (4.9.0) - Computer vision
|
||||
- `scikit-image` (0.22.0) - Image processing
|
||||
- `imageio` (2.34.0) - Image I/O
|
||||
- `librosa` (0.10.1) - Audio analysis
|
||||
|
||||
Additional packages can be installed as needed.
|
||||
|
||||
## Examples
|
||||
|
||||
|
||||
For the following examples, we'll use Claude 4 Sonnet but you can use any local or remote
|
||||
model with tool calling capabilities you'd like.
|
||||
|
||||
<Callout type="info">
|
||||
Make sure you activate Tools on the model you're using.
|
||||
</Callout>
|
||||
|
||||
|
||||
|
||||
### Basic Data Analysis
|
||||
|
||||
Start small. Open a new chat, confirm that the model has tools enabled and ask it to create a small dataset of 100 students with grades and study hours.
|
||||
|
||||
|
||||
|
||||
|
||||
```
|
||||
Create a small dataset of 100 students with grades and study hours.
|
||||
Calculate the correlation and create a scatter plot.
|
||||
```
|
||||
|
||||
The model will:
|
||||
1. Generate data with pandas (100 rows)
|
||||
2. Calculate correlation coefficient
|
||||
3. Create a matplotlib scatter plot
|
||||
4. Add trend line
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<Callout type="warning">
|
||||
As of `v0.6.5`, Jan won't be able to display visualizations in the chat but we're to get this fixed for the next release.
|
||||
</Callout>
|
||||
|
||||
|
||||
### Statistical Computing
|
||||
|
||||
```
|
||||
Run a Monte Carlo simulation with 10,000 iterations to estimate π.
|
||||
```
|
||||
|
||||
Expected output:
|
||||
- Numerical computation with numpy
|
||||
- Convergence plot showing estimate improvement
|
||||
- Final π estimate
|
||||
|
||||
|
||||
For more intensive simulations, increase iterations gradually and monitor performance.
|
||||
|
||||
### Machine Learning
|
||||
|
||||
```
|
||||
Create a simple 2-class dataset with 200 samples. Train a logistic regression
|
||||
model and visualize the decision boundary.
|
||||
```
|
||||
|
||||
The model will:
|
||||
- Generate synthetic 2D classification data
|
||||
- Train a single scikit-learn model
|
||||
- Plot data points and decision boundary
|
||||
|
||||
|
||||
### Time Series Analysis
|
||||
|
||||
```
|
||||
Generate daily temperature data for one year. Calculate moving averages
|
||||
and identify seasonal patterns.
|
||||
```
|
||||
|
||||
Output includes:
|
||||
- Line plot of temperature data
|
||||
- Moving average overlay
|
||||
- Simple seasonal decomposition
|
||||
|
||||
|
||||
### Scaling Up
|
||||
|
||||
Once basic examples work, you can increase complexity:
|
||||
- Larger datasets (1000+ samples)
|
||||
- Multiple models for comparison
|
||||
- Complex visualizations with subplots
|
||||
- Advanced statistical tests
|
||||
|
||||
The sandbox handles moderate computational loads well. For very large datasets or intensive ML training, consider breaking work into smaller chunks.
|
||||
|
||||
## Chart Generation
|
||||
|
||||
E2B automatically detects and extracts charts from matplotlib code. Charts are returned as base64-encoded images and downloadable files.
|
||||
|
||||
### Static Charts
|
||||
|
||||
```python
|
||||
import matplotlib.pyplot as plt
|
||||
import numpy as np
|
||||
|
||||
x = np.linspace(0, 10, 100)
|
||||
y = np.sin(x)
|
||||
|
||||
plt.figure(figsize=(10, 6))
|
||||
plt.plot(x, y)
|
||||
plt.title('Sine Wave')
|
||||
plt.xlabel('x')
|
||||
plt.ylabel('sin(x)')
|
||||
plt.show()
|
||||
```
|
||||
|
||||
E2B captures the plot and makes it available for download.
|
||||
|
||||
### Interactive Charts
|
||||
|
||||
The system extracts chart data for frontend visualization:
|
||||
|
||||
```python
|
||||
plt.bar(['A', 'B', 'C'], [10, 20, 15])
|
||||
plt.title('Sample Bar Chart')
|
||||
plt.show()
|
||||
```
|
||||
|
||||
Returns structured data:
|
||||
```json
|
||||
{
|
||||
"type": "bar",
|
||||
"title": "Sample Bar Chart",
|
||||
"elements": [
|
||||
{"label": "A", "value": 10},
|
||||
{"label": "B", "value": 20},
|
||||
{"label": "C", "value": 15}
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
Supported chart types: line, bar, scatter, pie, box plots.
|
||||
|
||||
## Available Tools
|
||||
|
||||
- **run_code**: Execute Python code
|
||||
- **install_package**: Add Python packages
|
||||
- **create_file**: Save files to sandbox
|
||||
- **read_file**: Access sandbox files
|
||||
- **list_files**: Browse sandbox contents
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Connection Issues:**
|
||||
- Verify API key is correct
|
||||
- Check Node.js installation
|
||||
- Restart Jan if server won't start
|
||||
|
||||
**Execution Problems:**
|
||||
- Free sandboxes have 2 cores and 1GB RAM - start with small datasets
|
||||
- Large computations may time out or run out of memory
|
||||
- Scale up complexity gradually after testing basic examples
|
||||
- Some packages may require explicit installation
|
||||
|
||||
**Package Installation:**
|
||||
- Most data science packages install successfully
|
||||
- System dependencies may cause failures for some packages
|
||||
- Try alternative packages if installation fails
|
||||
|
||||
<Callout type="warning">
|
||||
E2B has computational and memory limits. Break large operations into smaller chunks if needed.
|
||||
</Callout>
|
||||
|
||||
## Use Cases
|
||||
|
||||
E2B is useful for:
|
||||
|
||||
- **Academic Research**: Statistical analysis, data visualization, hypothesis testing
|
||||
- **Data Science**: Exploratory data analysis, model prototyping, result validation
|
||||
- **Financial Analysis**: Portfolio optimization, risk calculations, market simulations
|
||||
- **Scientific Computing**: Numerical simulations, mathematical modeling, algorithm testing
|
||||
- **Prototyping**: Quick algorithm validation, proof-of-concept development
|
||||
|
||||
The sandbox provides isolated execution without local environment setup or dependency management.
|
||||
@ -17,184 +17,200 @@ keywords:
|
||||
|
||||
import { Callout, Steps } from 'nextra/components'
|
||||
|
||||
# Using Exa Search MCP with Jan
|
||||
# Exa Search MCP
|
||||
|
||||
Exa MCP turns your AI into a research powerhouse. Instead of hallucinating facts or working with stale training data, your model gets real-time access to the web through Exa's AI-powered search engine.
|
||||
[Exa MCP](https://docs.exa.ai/examples/exa-mcp) provides real-time web search capabilities for AI models. Instead of relying on training data,
|
||||
models can access current web content through Exa's search API.
|
||||
|
||||
## What You Get
|
||||
## Available Tools
|
||||
|
||||
Exa MCP provides eight specialized search tools that actually work:
|
||||
Exa MCP includes eight search functions:
|
||||
|
||||
- **web_search_exa**: General web search with content extraction
|
||||
- **research_paper_search**: Academic papers and research content
|
||||
- **company_research**: Deep company analysis and intelligence
|
||||
- **crawling**: Extract content from specific URLs
|
||||
- **competitor_finder**: Find business competitors
|
||||
- **linkedin_search**: Search LinkedIn profiles and companies
|
||||
- **wikipedia_search_exa**: Wikipedia content retrieval
|
||||
- **github_search**: Repository and code search
|
||||
|
||||
Think of it as giving your AI a PhD in Google-fu.
|
||||
- `web_search_exa`: General web search with content extraction
|
||||
- `research_paper_search`: Academic papers and research content
|
||||
- `company_research`: Company analysis and business intelligence
|
||||
- `crawling`: Extract content from specific URLs
|
||||
- `competitor_finder`: Find business competitors
|
||||
- `linkedin_search`: Search LinkedIn profiles and companies
|
||||
- `wikipedia_search_exa`: Wikipedia content retrieval
|
||||
- `github_search`: Repository and code search
|
||||
|
||||
## Prerequisites
|
||||
|
||||
You'll need:
|
||||
- Jan installed and running
|
||||
- An Exa API key ([get yours here](https://dashboard.exa.ai/api-keys))
|
||||
- A model with tool calling capabilities (Claude, GPT-4, or compatible local models)
|
||||
- Node.js installed on your machine
|
||||
- Jan with MCP enabled
|
||||
- Exa API key from [dashboard.exa.ai](https://dashboard.exa.ai/api-keys)
|
||||
- Model with tool calling support
|
||||
- Node.js installed
|
||||
|
||||
<Callout type="info">
|
||||
**Model Compatibility**: Not all models play nice with tools. Claude models work exceptionally well. For local models, ensure tool calling is enabled in Jan's model parameters.
|
||||
Tool calling support varies by model. Jan Nano 32k and 128k, Claude, Gemini, GPT-4o and above models work reliably. For both local and remote models,
|
||||
verify tool calling is enabled in model parameters.
|
||||
</Callout>
|
||||
|
||||
## Setup
|
||||
|
||||
### Step 1: Enable MCP in Jan
|
||||
### Enable MCP
|
||||
|
||||
Navigate to **Settings** > **MCP Servers** and toggle **Allow All MCP Tool Permission** ON.
|
||||
1. Go to **Settings** > **MCP Servers**
|
||||
2. Toggle **Allow All MCP Tool Permission** ON
|
||||
|
||||
*[Screenshot placeholder: MCP settings page with toggle enabled]*
|
||||
|
||||
|
||||
### Step 2: Get Your Exa API Key
|
||||
### Get API Key
|
||||
|
||||
1. Go to [dashboard.exa.ai/api-keys](https://dashboard.exa.ai/api-keys)
|
||||
2. Sign up or log in
|
||||
3. Generate a new API key
|
||||
4. Copy it somewhere safe
|
||||
1. Visit [dashboard.exa.ai/api-keys](https://dashboard.exa.ai/api-keys)
|
||||
2. Create account or sign in
|
||||
3. Generate API key
|
||||
4. Save the key
|
||||
|
||||
*[Screenshot placeholder: Exa dashboard showing API key generation]*
|
||||
|
||||
|
||||
### Step 3: Add Exa MCP Server
|
||||
### Configure MCP Server
|
||||
|
||||
Click the `+` button in the MCP Servers section.
|
||||
Click `+` in MCP Servers section:
|
||||
|
||||
*[Screenshot placeholder: MCP server addition dialog]*
|
||||
|
||||
Enter the following configuration:
|
||||
|
||||
**Configuration:**
|
||||
- **Server Name**: `exa`
|
||||
- **Command**: `npx`
|
||||
- **Arguments**: `-y exa-mcp-server`
|
||||
- **Environment Variables**:
|
||||
- Key: `EXA_API_KEY`
|
||||
- Value: `your-actual-api-key-here`
|
||||
- Value: `your-api-key`
|
||||
|
||||
*[Screenshot placeholder: Filled MCP server configuration form]*
|
||||
|
||||
|
||||
### Step 4: Verify Connection
|
||||
### Verify Setup
|
||||
|
||||
Check that the server shows as active in your MCP Servers list.
|
||||
Check server status in the MCP Servers list.
|
||||
|
||||
*[Screenshot placeholder: Active Exa MCP server in the list]*
|
||||
|
||||
|
||||
### Step 5: Configure Your Model
|
||||
### Model Configuration
|
||||
|
||||
Use a model provider that supports tool calling. For cloud models:
|
||||
Use a compatible model provider:
|
||||
|
||||
- **Anthropic**: Ensure tool calling is enabled in Model Providers settings
|
||||
- **OpenAI**: Tool calling should work by default
|
||||
- **OpenRouter**: Works with tool-capable models like Claude
|
||||
- **Jan Nano 32k**
|
||||
- **Anthropic**
|
||||
- **OpenAI**
|
||||
- **OpenRouter**
|
||||
|
||||
*[Screenshot placeholder: Model provider settings with tool calling enabled]*
|
||||
|
||||
|
||||
For local models, enable tool calling in **Model Parameters** > **Model Capabilities**.
|
||||
## Usage
|
||||
|
||||
## Using Exa MCP
|
||||
Start a new chat with a tool-enabled model. Exa tools will appear in the available tools list.
|
||||
|
||||
Once configured, start a new chat with your tool-enabled model. The available Exa tools will appear in the tools section.
|
||||
|
||||
*[Screenshot placeholder: Chat interface showing Exa MCP tools available]*
|
||||
|
||||
|
||||
### Example Queries
|
||||
|
||||
Try these to see Exa MCP in action:
|
||||
|
||||
**Research Query:**
|
||||
**Current Events & Activities:**
|
||||
```
|
||||
Find recent research papers about transformer model optimization published in 2024
|
||||
What is happening this week, mid July 2025, in Sydney, Australia?
|
||||
```
|
||||
|
||||
**Company Intelligence:**
|
||||
|
||||
|
||||
**Investment Research:**
|
||||
```
|
||||
Research the company Anthropic - their business model, recent funding, and key competitors
|
||||
Find recent research papers about quantum computing startups that received Series A funding in 2024-2025
|
||||
```
|
||||
|
||||
**Technical Search:**
|
||||
**Tech Discovery:**
|
||||
```
|
||||
Find GitHub repositories for modern React state management libraries with good TypeScript support
|
||||
Find GitHub repositories for WebAssembly runtime engines written in Rust with active development
|
||||
```
|
||||
|
||||
**Content Analysis:**
|
||||
**Career Intelligence:**
|
||||
```
|
||||
Extract and summarize the content from this article: [URL]
|
||||
Search LinkedIn for AI safety researchers at major tech companies who published papers in the last 6 months
|
||||
```
|
||||
|
||||
*[Screenshot placeholder: Example of Exa MCP performing a search with results]*
|
||||
**Competitive Analysis:**
|
||||
```
|
||||
Research emerging competitors to OpenAI in the large language model space, focusing on companies founded after 2023
|
||||
```
|
||||
|
||||
## Advanced Configuration
|
||||
**Travel & Local Research:**
|
||||
```
|
||||
Find authentic local food experiences in Tokyo that aren't in typical tourist guides, mentioned in recent travel blogs
|
||||
```
|
||||
|
||||
### Selective Tool Enabling
|
||||
**Academic Research:**
|
||||
```
|
||||
Find recent papers about carbon capture technology breakthroughs published in Nature or Science during 2025
|
||||
```
|
||||
|
||||
If you don't want all eight tools (and who needs that much power?), you can enable specific ones:
|
||||
**Creator Economy:**
|
||||
```
|
||||
Research successful creators who transitioned from TikTok to longer-form content platforms in 2024-2025
|
||||
```
|
||||
|
||||
- **Command**: `npx`
|
||||
- **Arguments**: `-y exa-mcp-server --tools=web_search_exa,research_paper_search,company_research`
|
||||
**Emerging Tech Trends:**
|
||||
```
|
||||
Find startups working on brain-computer interfaces that have raised funding in the past 12 months
|
||||
```
|
||||
|
||||
### Remote MCP Option
|
||||
**Health & Wellness:**
|
||||
```
|
||||
Extract information about the latest longevity research findings from Peter Attia's recent podcast episodes
|
||||
```
|
||||
|
||||
For the truly lazy, use Exa's hosted MCP server instead of running it locally:
|
||||
**Regulatory Intelligence:**
|
||||
```
|
||||
Find recent AI regulation developments in the EU that could impact US companies, focusing on July 2025 updates
|
||||
```
|
||||
|
||||
- **Server Name**: `exa-remote`
|
||||
- **Command**: `npx`
|
||||
- **Arguments**: `-y mcp-remote https://mcp.exa.ai/mcp?exaApiKey=your-api-key`
|
||||
**Supply Chain Research:**
|
||||
```
|
||||
Research companies developing sustainable packaging alternatives that have partnerships with major retailers
|
||||
```
|
||||
|
||||
## Use Cases That Actually Matter
|
||||
## Use Cases
|
||||
|
||||
### Academic Research
|
||||
Perfect for literature reviews, finding recent papers, and staying current with research trends. No more sifting through Google Scholar's ancient interface.
|
||||
Literature reviews, finding recent papers, tracking research trends.
|
||||
|
||||
### Competitive Intelligence
|
||||
Research competitors, analyze market positioning, and gather business intelligence without manual browsing.
|
||||
### Business Intelligence
|
||||
Competitor analysis, market research, company information gathering.
|
||||
|
||||
### Technical Discovery
|
||||
Find libraries, tools, and solutions on GitHub. Better than hoping Stack Overflow has the answer.
|
||||
### Technical Research
|
||||
Finding libraries, tools, and code repositories. Documentation research.
|
||||
|
||||
### Content Curation
|
||||
Extract and analyze content from specific URLs for research or content creation.
|
||||
### Content Analysis
|
||||
Extracting and analyzing content from specific URLs for research.
|
||||
|
||||
### Professional Networking
|
||||
Search LinkedIn for potential collaborators, industry experts, or business connections.
|
||||
### Professional Search
|
||||
LinkedIn searches for industry connections and expertise.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Server won't connect?**
|
||||
- Verify your API key is correct (no extra spaces or quotes)
|
||||
- Ensure Node.js is installed
|
||||
- Restart Jan completely
|
||||
**Connection Issues:**
|
||||
- Verify API key accuracy
|
||||
- Check Node.js installation
|
||||
- Restart Jan
|
||||
- Make sure you have enough credits in your Exa account
|
||||
|
||||
**Model not using tools?**
|
||||
- Check that tool calling is enabled for your model
|
||||
- Try a different model (Claude works best)
|
||||
- Verify the MCP server is active
|
||||
**Tool Calling Problems:**
|
||||
- Confirm tool calling is enabled for your model
|
||||
- Try Jan Nano 32k, Claude, Gemini, GPT-4o and above models
|
||||
- Check MCP server status
|
||||
|
||||
**Search results seem off?**
|
||||
- Exa's AI search is context-aware - be specific in your queries
|
||||
- Use natural language rather than keyword stuffing
|
||||
**Search Quality:**
|
||||
- Use specific, descriptive queries
|
||||
- Prefer natural language over keywords
|
||||
|
||||
**API key errors?**
|
||||
- Double-check your key at [dashboard.exa.ai](https://dashboard.exa.ai)
|
||||
- Ensure you haven't hit rate limits
|
||||
- Try regenerating the key if issues persist
|
||||
**API Errors:**
|
||||
- Verify API key at [dashboard.exa.ai](https://dashboard.exa.ai)
|
||||
- Check rate limits on your plan
|
||||
- Regenerate API key if needed
|
||||
|
||||
<Callout type="warning">
|
||||
**Rate Limits**: Exa has API rate limits. If you're planning to go full researcher mode, check your plan limits to avoid surprises.
|
||||
Exa has API rate limits. Check your plan limits to avoid interruptions.
|
||||
</Callout>
|
||||
|
||||
## What's Next
|
||||
## Next Steps
|
||||
|
||||
Once you've got Exa MCP running, you'll wonder how you ever did research without it. The combination of Jan's privacy-focused approach with Exa's real-time search capabilities creates a research environment that's both powerful and under your control.
|
||||
|
||||
No more tab explosions. No more bookmark chaos. Just ask, and get answers backed by current, relevant sources.
|
||||
|
||||
*[Screenshot placeholder: Final example showing a complex research query with multiple Exa tools being used]*
|
||||
Exa MCP enables real-time web search within Jan's privacy-focused environment. Models can access current information while maintaining
|
||||
local conversation processing.
|
||||
@ -1,245 +0,0 @@
|
||||
---
|
||||
title: Perplexity MCP
|
||||
description: Connect Jan to Perplexity's AI-powered search and reasoning capabilities.
|
||||
keywords:
|
||||
[
|
||||
Jan,
|
||||
MCP,
|
||||
Model Context Protocol,
|
||||
Perplexity,
|
||||
AI search,
|
||||
web search,
|
||||
reasoning,
|
||||
Sonar,
|
||||
real-time search,
|
||||
tool calling,
|
||||
]
|
||||
---
|
||||
|
||||
import { Callout, Steps } from 'nextra/components'
|
||||
|
||||
# Using Perplexity MCP with Jan
|
||||
|
||||
Perplexity MCP brings real-time web search and specialized reasoning to your AI conversations. Instead of your model pretending it knows what happened yesterday, it can actually find out.
|
||||
|
||||
## What You Get
|
||||
|
||||
Perplexity MCP offers two flavors of intelligence:
|
||||
|
||||
### Official Go Implementation
|
||||
- **perplexity_ask**: Real-time web search using Sonar Pro
|
||||
- **perplexity_reason**: Complex reasoning with Sonar Reasoning Pro
|
||||
- Fast, minimal setup with pre-built binaries
|
||||
|
||||
### Python Implementation
|
||||
- **ask_perplexity**: Expert programming assistance and technical explanations
|
||||
- **chat_perplexity**: Maintains ongoing conversations with full history
|
||||
- **list_chats_perplexity**: Browse your conversation history
|
||||
- **read_chat_perplexity**: Retrieve complete chat histories
|
||||
- **search**: General web search with customizable detail levels
|
||||
|
||||
Both give your AI the ability to stop guessing and start knowing.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
You'll need:
|
||||
- Jan installed and running
|
||||
- A Perplexity API key ([get yours here](https://docs.perplexity.ai/docs/getting-started))
|
||||
- For Go version: Homebrew (macOS/Linux) or manual binary installation
|
||||
- For Python version: Python 3.10+ and uvx package manager
|
||||
- A model with tool calling capabilities
|
||||
|
||||
<Callout type="info">
|
||||
**API Key Required**: Unlike some search engines that pretend to be free, Perplexity requires an API key. The good news? Their pricing is actually reasonable for the quality you get.
|
||||
</Callout>
|
||||
|
||||
## Setup
|
||||
|
||||
### Step 1: Enable MCP in Jan
|
||||
|
||||
Navigate to **Settings** > **MCP Servers** and toggle **Allow All MCP Tool Permission** ON.
|
||||
|
||||
*[Screenshot placeholder: MCP settings page with toggle enabled]*
|
||||
|
||||
### Step 2: Get Your Perplexity API Key
|
||||
|
||||
1. Go to [docs.perplexity.ai](https://docs.perplexity.ai/docs/getting-started)
|
||||
2. Sign up or log in to get API access
|
||||
3. Generate your API key
|
||||
4. Keep it handy
|
||||
|
||||
*[Screenshot placeholder: Perplexity API key generation page]*
|
||||
|
||||
### Step 3: Choose Your Installation Method
|
||||
|
||||
You have two solid options. Pick your poison:
|
||||
|
||||
#### Option A: Go Implementation (Recommended for Speed)
|
||||
|
||||
**Using Homebrew (macOS/Linux):**
|
||||
```bash
|
||||
brew install alcova-ai/tap/perplexity-mcp
|
||||
```
|
||||
|
||||
**Manual Installation:**
|
||||
Download the appropriate binary from [Perplexity MCP releases](https://github.com/Alcova-AI/perplexity-mcp/releases) and place it in your PATH.
|
||||
|
||||
#### Option B: Python Implementation (More Features)
|
||||
|
||||
**Install uvx if you haven't already:**
|
||||
```bash
|
||||
pip install uvx
|
||||
```
|
||||
|
||||
### Step 4: Configure MCP Server
|
||||
|
||||
Click the `+` button in the MCP Servers section and choose your configuration:
|
||||
|
||||
*[Screenshot placeholder: MCP server addition dialog]*
|
||||
|
||||
#### For Go Implementation:
|
||||
|
||||
- **Server Name**: `perplexity-go`
|
||||
- **Command**: `perplexity-mcp`
|
||||
- **Arguments**: (leave empty)
|
||||
- **Environment Variables**:
|
||||
- Key: `PERPLEXITY_API_KEY`
|
||||
- Value: `your-api-key-here`
|
||||
|
||||
*[Screenshot placeholder: Go implementation MCP configuration]*
|
||||
|
||||
#### For Python Implementation:
|
||||
|
||||
- **Server Name**: `perplexity-python`
|
||||
- **Command**: `uvx`
|
||||
- **Arguments**: `mcp-perplexity`
|
||||
- **Environment Variables**:
|
||||
- Key: `PERPLEXITY_API_KEY`
|
||||
- Value: `your-api-key-here`
|
||||
- Key: `PERPLEXITY_MODEL`
|
||||
- Value: `sonar-pro` (optional, this is the default)
|
||||
|
||||
*[Screenshot placeholder: Python implementation MCP configuration]*
|
||||
|
||||
### Step 5: Verify Connection
|
||||
|
||||
Check that your chosen server shows as active in the MCP Servers list.
|
||||
|
||||
*[Screenshot placeholder: Active Perplexity MCP server in the list]*
|
||||
|
||||
### Step 6: Configure Your Model
|
||||
|
||||
Ensure your model supports tool calling. For cloud models, verify tool calling is enabled in your provider settings.
|
||||
|
||||
*[Screenshot placeholder: Model provider settings with tool calling enabled]*
|
||||
|
||||
## Using Perplexity MCP
|
||||
|
||||
Start a new chat with a tool-enabled model. The available Perplexity tools will appear in the tools section.
|
||||
|
||||
*[Screenshot placeholder: Chat interface showing Perplexity MCP tools available]*
|
||||
|
||||
### Example Queries
|
||||
|
||||
**Real-Time News:**
|
||||
```
|
||||
What are the latest developments in AI regulation in the EU this week?
|
||||
```
|
||||
|
||||
**Technical Research:**
|
||||
```
|
||||
Search for recent best practices in React server components and explain the key differences from client components
|
||||
```
|
||||
|
||||
**Ongoing Research (Python version):**
|
||||
```
|
||||
Start a research conversation about quantum computing advancements in 2024. I want to maintain this conversation for follow-up questions.
|
||||
```
|
||||
|
||||
**Code Validation:**
|
||||
```
|
||||
Check if using jQuery in 2024 is considered deprecated for modern web development
|
||||
```
|
||||
|
||||
*[Screenshot placeholder: Example of Perplexity MCP performing a search with results]*
|
||||
|
||||
## Advanced Configuration
|
||||
|
||||
### Custom Models
|
||||
|
||||
For the Go implementation, you can specify different models:
|
||||
|
||||
- **Arguments**: `--model sonar-reasoning-pro --reasoning-model sonar-pro`
|
||||
|
||||
For the Python implementation, use environment variables:
|
||||
|
||||
- **PERPLEXITY_MODEL_ASK**: `sonar-reasoning-pro`
|
||||
- **PERPLEXITY_MODEL_CHAT**: `sonar-pro`
|
||||
|
||||
### Chat History Management (Python Version)
|
||||
|
||||
The Python implementation stores chat history locally. Configure the database path:
|
||||
|
||||
- **DB_PATH**: `/path/to/your/chats.db`
|
||||
|
||||
### Web UI (Python Version)
|
||||
|
||||
Enable a web interface for chat management:
|
||||
|
||||
- **WEB_UI_ENABLED**: `true`
|
||||
- **WEB_UI_PORT**: `8050`
|
||||
- **WEB_UI_HOST**: `127.0.0.1`
|
||||
|
||||
## Use Cases That Matter
|
||||
|
||||
### Research and Fact-Checking
|
||||
Perfect for journalists, researchers, and anyone who needs current, verified information. No more "according to my training data from 2023" responses.
|
||||
|
||||
### Technical Problem-Solving
|
||||
Get current best practices, recent framework updates, and solutions to bleeding-edge problems. Stack Overflow wishes it was this fresh.
|
||||
|
||||
### Market Research
|
||||
Track industry trends, competitor analysis, and market developments in real-time. Your AI finally knows what happened after its training cutoff.
|
||||
|
||||
### Academic Research
|
||||
Access current papers, recent findings, and evolving scientific discussions. Like having a research assistant who never sleeps.
|
||||
|
||||
### Code Modernization
|
||||
Check if your dependencies are deprecated, find modern alternatives, and stay current with evolving best practices.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Server won't connect?**
|
||||
- Verify your API key is valid and properly formatted
|
||||
- For Go version: Ensure the binary is in your PATH
|
||||
- For Python version: Check that uvx and Python 3.10+ are installed
|
||||
- Restart Jan completely
|
||||
|
||||
**API errors?**
|
||||
- Check your Perplexity API usage limits
|
||||
- Verify your API key hasn't expired
|
||||
- Ensure you have sufficient credits
|
||||
|
||||
**No search results?**
|
||||
- Perplexity's search is context-aware - be specific in your queries
|
||||
- Check that your model actually supports tool calling
|
||||
- Try rephrasing your question
|
||||
|
||||
**Python version issues?**
|
||||
- Ensure uvx is properly installed: `pip install uvx`
|
||||
- Check Python version: `python --version` (should be 3.10+)
|
||||
- Verify dependencies: `uvx --help`
|
||||
|
||||
<Callout type="warning">
|
||||
**Rate Limits**: Perplexity has usage limits based on your plan. Monitor your usage to avoid hitting limits during intensive research sessions.
|
||||
</Callout>
|
||||
|
||||
## What's Next
|
||||
|
||||
With Perplexity MCP running, your AI conversations become grounded in reality. No more hallucinated facts, no more outdated information, no more "I don't have access to real-time data" excuses.
|
||||
|
||||
Whether you're researching the latest AI developments, debugging modern frameworks, or tracking current events, your AI assistant finally has the tools to give you accurate, up-to-date answers.
|
||||
|
||||
Time to retire those bookmark folders. Your AI can find what you need, when you need it.
|
||||
|
||||
*[Screenshot placeholder: Complex research query showing Perplexity MCP tools working together]*
|
||||
@ -1,6 +1,6 @@
|
||||
---
|
||||
title: Model Parameters
|
||||
description: Manage your interaction with AI locally.
|
||||
description: Customize how your AI models behave and perform.
|
||||
keywords:
|
||||
[
|
||||
Jan,
|
||||
@ -12,64 +12,99 @@ keywords:
|
||||
conversational AI,
|
||||
no-subscription fee,
|
||||
large language models,
|
||||
threads,
|
||||
chat history,
|
||||
thread history,
|
||||
model settings,
|
||||
parameters,
|
||||
]
|
||||
---
|
||||
import { Callout, Steps } from 'nextra/components'
|
||||
|
||||
# Model Parameters
|
||||
To customize model settings for a conversation or a model:
|
||||
- In **Threads**, click the **Gear icon** next to selected model
|
||||
- Or, in **Settings > Model Providers > Llama.cpp**, click the **gear icon** next to a model for advanced settings
|
||||
- Click the **edit button** next to a model to configure capabilities
|
||||
|
||||
## Inference & Engine Parameters (Gear Icon)
|
||||
These settings are available in the model settings modal:
|
||||
| Parameter | Description |
|
||||
|---------------------|-------------|
|
||||
| **Context Size** | Maximum prompt context length (how much text the model can consider at once). |
|
||||
| **GPU Layers** | Number of model layers to offload to GPU. More layers = faster, but uses more VRAM. |
|
||||
| **Temperature** | Controls response randomness. Lower = more focused, higher = more creative. |
|
||||
| **Top K** | Top-K sampling. Limits next token selection to the K most likely. |
|
||||
| **Top P** | Top-P (nucleus) sampling. Limits next token selection to a cumulative probability. |
|
||||
| **Min P** | Minimum probability for token selection. |
|
||||
| **Repeat Last N** | Number of tokens to consider for repeat penalty. |
|
||||
| **Repeat Penalty** | Penalize repeating token sequences. |
|
||||
| **Presence Penalty**| Penalize alpha presence (encourages new topics). |
|
||||
| **Frequency Penalty** | Reduces word repetition. |
|
||||
Model parameters control how your AI thinks and responds. Think of them as the AI's personality settings and performance controls.
|
||||
|
||||
## How to Access Settings
|
||||
|
||||
**For individual conversations:**
|
||||
- In **Threads**, click the **gear icon** next to your selected model
|
||||
|
||||
**For permanent model settings:**
|
||||
- Go to **Settings > Model Providers > Llama.cpp**, click the **gear icon** next to a model
|
||||
|
||||
**For model capabilities:**
|
||||
- Click the **edit button** next to a model to enable features like vision or tools
|
||||
|
||||
## Performance Settings (Gear Icon)
|
||||
|
||||
These settings control how the model thinks and performs:
|
||||
|
||||
| Setting | What It Does | Simple Explanation |
|
||||
|---------|-------------|-------------------|
|
||||
| **Context Size** | How much text the model remembers | Like the model's working memory. Larger = remembers more of your conversation, but uses more computer memory. |
|
||||
| **GPU Layers** | How much work your graphics card does | More layers on GPU = faster responses, but needs more graphics memory. Start high and reduce if you get errors. |
|
||||
| **Temperature** | How creative vs. predictable responses are | Low (0.1-0.3) = focused, consistent answers. High (0.7-1.0) = creative, varied responses. Try 0.7 for general use. |
|
||||
| **Top K** | How many word choices the model considers | Smaller numbers (20-40) = more focused. Larger numbers (80-100) = more variety. Most people don't need to change this. |
|
||||
| **Top P** | Another way to control word variety | Works with Top K. Values like 0.9 work well. Lower = more focused, higher = more creative. |
|
||||
| **Min P** | Minimum chance a word needs to be chosen | Prevents very unlikely words. Usually fine at default settings. |
|
||||
| **Repeat Last N** | How far back to check for repetition | Helps prevent the model from repeating itself. Default values usually work well. |
|
||||
| **Repeat Penalty** | How much to avoid repeating words | Higher values (1.1-1.3) reduce repetition. Too high makes responses awkward. |
|
||||
| **Presence Penalty** | Encourages talking about new topics | Higher values make the model explore new subjects instead of staying on one topic. |
|
||||
| **Frequency Penalty** | Reduces word repetition | Similar to repeat penalty but focuses on how often words are used. |
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
## Model Capabilities (Edit Button)
|
||||
These toggles are available when you click the edit button next to a model:
|
||||
- **Vision**: Enable image input/output
|
||||
- **Tools**: Enable advanced tools (web search, file ops, code)
|
||||
- **Embeddings**: Enable embedding generation
|
||||
- **Web Search**: Allow model to search the web
|
||||
- **Reasoning**: Enable advanced reasoning features
|
||||
|
||||
<br/>
|
||||
These toggle switches enable special features:
|
||||
|
||||
- **Vision**: Let the model see and analyze images you share
|
||||
- **Tools**: Enable advanced features like web search, file operations, and code execution
|
||||
- **Embeddings**: Allow the model to create numerical representations of text (for advanced users)
|
||||
- **Web Search**: Let the model search the internet for current information
|
||||
- **Reasoning**: Enable step-by-step thinking for complex problems
|
||||
|
||||
|
||||
|
||||
<br/>
|
||||
|
||||
### Model Parameters
|
||||
This setting defines and configures the model's behavior.
|
||||
| Parameter | Description |
|
||||
|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| **Prompt Template** | A structured format that guides how the model should respond. Contains **placeholders** and **instructions** that help shape the model's output in a consistent way.|
|
||||
## Hardware Settings
|
||||
|
||||
### Engine Parameters
|
||||
These settings parameters control how the model runs on your hardware.
|
||||
| Parameter | Description |
|
||||
|---------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
|
||||
| **Number of GPU Layers (ngl)** | - Controls how many layers of the model run on your GPU.<br></br>- More layers on GPU generally means faster processing, but requires more GPU memory.|
|
||||
| **Context Length** | - Controls how much text the model can consider at once.<br></br>- Longer context allows the model to handle more input but uses more memory and runs slower.<br></br>- The maximum context length varies with the model used.<br></br>|
|
||||
These control how efficiently the model runs on your computer:
|
||||
|
||||
### GPU Layers
|
||||
Think of your model as a stack of layers, like a cake. Each layer can run on either your main processor (CPU) or graphics card (GPU). Your graphics card is usually much faster.
|
||||
|
||||
- **More GPU layers** = Faster responses, but uses more graphics memory
|
||||
- **Fewer GPU layers** = Slower responses, but uses less graphics memory
|
||||
|
||||
Start with the maximum number and reduce if you get out-of-memory errors.
|
||||
|
||||
### Context Length
|
||||
This is like the model's short-term memory - how much of your conversation it can remember at once.
|
||||
|
||||
- **Longer context** = Remembers more of your conversation, better for long discussions
|
||||
- **Shorter context** = Uses less memory, runs faster, but might "forget" earlier parts of long conversations
|
||||
|
||||
<Callout type="info">
|
||||
By default, Jan defaults to the minimum between **8192** and the model's maximum context length, you can adjust this based on your needs.
|
||||
Jan defaults to 8192 tokens (roughly 6000 words) or your model's maximum, whichever is smaller. This handles most conversations well.
|
||||
</Callout>
|
||||
|
||||
## Quick Setup Guide
|
||||
|
||||
**For most users:**
|
||||
1. Enable **Tools** if you want web search and code execution
|
||||
2. Set **Temperature** to 0.7 for balanced creativity
|
||||
3. Max out **GPU Layers** (reduce only if you get memory errors)
|
||||
4. Leave other settings at defaults
|
||||
|
||||
**For creative writing:**
|
||||
- Increase **Temperature** to 0.8-1.0
|
||||
- Increase **Top P** to 0.95
|
||||
|
||||
**For factual/technical work:**
|
||||
- Decrease **Temperature** to 0.1-0.3
|
||||
- Enable **Tools** for web search and calculations
|
||||
|
||||
**Troubleshooting:**
|
||||
- **Responses too repetitive?** Increase Temperature or Repeat Penalty
|
||||
- **Out of memory errors?** Reduce GPU Layers or Context Size
|
||||
- **Responses too random?** Decrease Temperature
|
||||
- **Model running slowly?** Increase GPU Layers (if you have VRAM) or reduce Context Size
|
||||
|
||||
BIN
docs/src/pages/local-server/_assets/api-server-logs.png
Normal file
BIN
docs/src/pages/local-server/_assets/api-server-logs.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 1.4 MiB |
BIN
docs/src/pages/local-server/_assets/ls.png
Normal file
BIN
docs/src/pages/local-server/_assets/ls.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 120 KiB |
@ -15,7 +15,10 @@
|
||||
"title": "TUTORIALS",
|
||||
"type": "separator"
|
||||
},
|
||||
"server-examples": "Integrations",
|
||||
"continue-dev": "Continue.dev",
|
||||
"llmcord": "LLMCord (Discord)",
|
||||
"n8n": "n8n",
|
||||
"tabby": "Tabby",
|
||||
|
||||
"explanation-separator": {
|
||||
"title": "EXPLANATION",
|
||||
@ -24,11 +27,6 @@
|
||||
"llama-cpp": "Local AI Engine",
|
||||
"data-folder": "Jan Data Folder",
|
||||
|
||||
"advanced-separator": {
|
||||
"title": "ADVANCED",
|
||||
"type": "separator"
|
||||
},
|
||||
"mcp": "Model Context Protocol",
|
||||
"reference-separator": {
|
||||
"title": "REFERENCE",
|
||||
"type": "separator"
|
||||
|
||||
@ -1,21 +1,17 @@
|
||||
---
|
||||
title: Jan Data Folder
|
||||
description: A guide to Jan's data structure.
|
||||
sidebar_position: 2
|
||||
description: Understand where Jan stores your data and how to monitor server logs.
|
||||
keywords:
|
||||
[
|
||||
Jan,
|
||||
Customizable Intelligence, LLM,
|
||||
local AI,
|
||||
privacy focus,
|
||||
free and open source,
|
||||
private and offline,
|
||||
conversational AI,
|
||||
no-subscription fee,
|
||||
large language models,
|
||||
quickstart,
|
||||
getting started,
|
||||
using AI model,
|
||||
data folder,
|
||||
logs,
|
||||
server logs,
|
||||
troubleshooting,
|
||||
privacy,
|
||||
local storage,
|
||||
file structure,
|
||||
]
|
||||
---
|
||||
|
||||
@ -23,198 +19,161 @@ import { Tabs } from 'nextra/components'
|
||||
import { Callout, Steps } from 'nextra/components'
|
||||
import { Settings, FolderOpen } from 'lucide-react'
|
||||
|
||||
|
||||
# Jan Data Folder
|
||||
Jan stores your data locally in JSON format. Your data is yours alone.
|
||||
|
||||
## Open Jan Data Folder
|
||||
Jan stores all your data locally on your computer. No cloud storage, no external servers - everything stays on your machine.
|
||||
|
||||
Via Jan:
|
||||
1. Settings (<Settings width={16} height={16} style={{display:"inline"}}/>) > Advanced Settings
|
||||
2. Click <FolderOpen width={16} height={16} style={{display:"inline"}}/>
|
||||
## Quick Access
|
||||
|
||||
**Via Jan Interface:**
|
||||
1. Go to Settings (<Settings width={16} height={16} style={{display:"inline"}}/>) > Advanced Settings
|
||||
2. Click the folder icon <FolderOpen width={16} height={16} style={{display:"inline"}}/>
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
Via Terminal:
|
||||
**Via File Explorer:**
|
||||
|
||||
<Tabs items={['Windows', 'macOS', 'Linux']}>
|
||||
<Tabs.Tab>
|
||||
```cmd
|
||||
%APPDATA%\Jan\data
|
||||
```
|
||||
</Tabs.Tab>
|
||||
<Tabs.Tab>
|
||||
```bash
|
||||
# Windows
|
||||
cd %APPDATA%/Jan/data
|
||||
~/Library/Application Support/Jan/data
|
||||
```
|
||||
</Tabs.Tab>
|
||||
<Tabs.Tab>
|
||||
```bash
|
||||
# Default installation
|
||||
~/.config/Jan/data
|
||||
|
||||
# Mac
|
||||
cd ~/Library/Application\ Support/Jan/data
|
||||
# Custom installation
|
||||
$XDG_CONFIG_HOME/Jan/data
|
||||
```
|
||||
</Tabs.Tab>
|
||||
</Tabs>
|
||||
|
||||
# Linux
|
||||
cd $XDG_CONFIG_HOME/Jan/data # Custom install
|
||||
cd ~/.config/Jan/data # Default install
|
||||
## Monitoring Server Logs
|
||||
|
||||
When Jan's local server is running, you can monitor real-time activity in the logs folder:
|
||||
|
||||
|
||||
|
||||
### Live Log Monitoring
|
||||
|
||||
**Real-time logs show:**
|
||||
- API requests and responses
|
||||
- Model loading and inference activity
|
||||
- Error messages and warnings
|
||||
- Performance metrics
|
||||
- Connection attempts from external applications
|
||||
|
||||
**Accessing logs:**
|
||||
- **In Jan**: System Monitor (footer) > App Log
|
||||
- **File location**: `/logs/app.log`
|
||||
|
||||
### Log Categories
|
||||
|
||||
| Log Type | What It Shows | When It's Useful |
|
||||
|----------|---------------|------------------|
|
||||
| **[APP]** | Core application events | Startup issues, crashes, general errors |
|
||||
| **[SERVER]** | API server activity | Connection problems, request failures |
|
||||
| **[SPECS]** | Hardware information | Performance issues, compatibility problems |
|
||||
| **[MODEL]** | Model operations | Loading failures, inference errors |
|
||||
|
||||
## Data Structure
|
||||
|
||||
```
|
||||
jan/
|
||||
├── assistants/ # AI personality settings
|
||||
│ └── jan/
|
||||
│ └── assistant.json
|
||||
├── engines/ # Engine configurations
|
||||
│ └── llama.cpp/
|
||||
├── extensions/ # Add-on modules
|
||||
│ └── extensions.json
|
||||
├── logs/ # Server and application logs
|
||||
│ └── app.log # Main log file
|
||||
├── models/ # Downloaded AI models
|
||||
│ └── huggingface.co/
|
||||
└── threads/ # Chat conversations
|
||||
└── thread_id/
|
||||
├── messages.jsonl
|
||||
└── thread.json
|
||||
```
|
||||
|
||||
## Directory Structure
|
||||
## Key Folders Explained
|
||||
|
||||
### `/logs/` - Server Activity Hub
|
||||
Contains all application and server logs. Essential for troubleshooting and monitoring API activity.
|
||||
|
||||
**What you'll find:**
|
||||
- Real-time server requests
|
||||
- Model loading status
|
||||
- Error diagnostics
|
||||
- Performance data
|
||||
|
||||
### `/models/` - AI Model Storage
|
||||
Where your downloaded models live. Each model includes:
|
||||
- `model.gguf` - The actual AI model file
|
||||
- `model.json` - Configuration and metadata
|
||||
|
||||
### `/threads/` - Chat History
|
||||
Every conversation gets its own folder with:
|
||||
- `messages.jsonl` - Complete chat history
|
||||
- `thread.json` - Thread metadata and settings
|
||||
|
||||
### `/assistants/` - AI Personalities
|
||||
Configuration files that define how your AI assistants behave, including their instructions and available tools.
|
||||
|
||||
## Privacy & Security
|
||||
|
||||
**Your data stays local:**
|
||||
- No cloud backups or syncing
|
||||
- Files stored in standard JSON/JSONL formats
|
||||
- Complete control over your data
|
||||
- Easy to backup or migrate
|
||||
|
||||
**File permissions:**
|
||||
- Only you and Jan can access these files
|
||||
- Standard user-level permissions
|
||||
- No elevated access required
|
||||
|
||||
<Callout type="info">
|
||||
Root directory: `~/jan`
|
||||
When using cloud AI services through Jan, those conversations follow the cloud provider's data policies. Local model conversations never leave your computer.
|
||||
</Callout>
|
||||
|
||||
```sh
|
||||
/assistants/
|
||||
/jan/
|
||||
assistant.json
|
||||
/engines/
|
||||
/llama.cpp/
|
||||
/extensions/
|
||||
extensions.json
|
||||
/@janhq/
|
||||
/assistant-extension/
|
||||
/conversational-extension/
|
||||
/download-extension/
|
||||
/engine-management-extension/
|
||||
/hardware-management-extension/
|
||||
/inference-cortex-extension/
|
||||
/model-extension/
|
||||
/files/
|
||||
/logs/
|
||||
app.log
|
||||
/models/
|
||||
/huggingface.co/
|
||||
/Model_Provider_A/
|
||||
/Model_A
|
||||
model_A.gguf
|
||||
model_A.yaml
|
||||
/threads/
|
||||
/thread_A/
|
||||
messages.jsonl
|
||||
thread.json
|
||||
## Common Tasks
|
||||
|
||||
```
|
||||
### Backup Your Data
|
||||
Copy the entire Jan data folder to backup:
|
||||
- All chat history
|
||||
- Model configurations
|
||||
- Assistant settings
|
||||
- Extension data
|
||||
|
||||
### `assistants/`
|
||||
Where AI personalities live. The default one (`/assistants/jan/`):
|
||||
### Clear Chat History
|
||||
Delete individual thread folders in `/threads/` or use Jan's interface to delete conversations.
|
||||
|
||||
```json
|
||||
{
|
||||
"avatar": "👋",
|
||||
"id": "jan",
|
||||
"object": "assistant",
|
||||
"created_at": 1750945742.536,
|
||||
"name": "Jan",
|
||||
"description": "Jan is a helpful AI assistant that can use tools and help complete tasks for its users.",
|
||||
"model": "*",
|
||||
"instructions": "You have access to a set of tools to help you answer the user’s question. You can use only one tool per message, and you’ll receive the result of that tool in the user’s next response. To complete a task, use tools step by step—each step should be guided by the outcome of the previous one.\nTool Usage Rules:\n1. Always provide the correct values as arguments when using tools. Do not pass variable names—use actual values instead.\n2. You may perform multiple tool steps to complete a task.\n3. Avoid repeating a tool call with exactly the same parameters to prevent infinite loops.",
|
||||
"tools": [
|
||||
{
|
||||
"type": "retrieval",
|
||||
"enabled": false,
|
||||
"useTimeWeightedRetriever": false,
|
||||
"settings": {
|
||||
"top_k": 2,
|
||||
"chunk_size": 1024,
|
||||
"chunk_overlap": 64,
|
||||
"retrieval_template": "Use the following pieces of context to answer the question at the end.\n----------------\nCONTEXT: {CONTEXT}\n----------------\nQUESTION: {QUESTION}\n----------------\nHelpful Answer:"
|
||||
}
|
||||
}
|
||||
],
|
||||
"file_ids": []
|
||||
}
|
||||
```
|
||||
### Export Conversations
|
||||
Thread files are in standard JSON format - readable by any text editor or compatible with other applications.
|
||||
|
||||
Parameters:
|
||||
### Troubleshooting Data Issues
|
||||
1. Check `/logs/app.log` for error messages
|
||||
2. Verify folder permissions
|
||||
3. Ensure sufficient disk space
|
||||
4. Restart Jan if files appear corrupted
|
||||
|
||||
| Parameter | Description | Type | Default |
|
||||
|------------------------|--------------------------------------------------------------|---------|---------|
|
||||
| id | Assistant identifier | string | jan |
|
||||
| avatar | Assistant image | string | None |
|
||||
| object | OpenAI API compatibility marker | string | None |
|
||||
| created_at | Creation timestamp | string | None |
|
||||
| name | Display name | string | Jan |
|
||||
| description | Role description | string | Default |
|
||||
| model | Allowed models (* = all) | string | * |
|
||||
| instructions | Default thread instructions | string | None |
|
||||
| file_ids | OpenAI compatibility field | string | None |
|
||||
| tools | Available tools (retrieval only currently) | array | retrieval|
|
||||
| type | Tool type | string | retrieval|
|
||||
| enabled | Tool status | boolean | true |
|
||||
| useTimeWeightedRetriever| Time-weighted retrieval toggle | boolean | false |
|
||||
| settings | Tool configuration | object | None |
|
||||
| top_k | Max retrieval results | number | 2 |
|
||||
| chunk_size | Text chunk size | number | 1024 |
|
||||
| chunk_overlap | Chunk overlap amount | number | 64 |
|
||||
| retrieval_template | Response format template | string | None |
|
||||
## Uninstalling Jan
|
||||
|
||||
### `extensions/`
|
||||
Add-on central. Organization extensions live in `@janhq/`, solo ones in root.
|
||||
If you need to completely remove Jan and all data:
|
||||
|
||||
### `logs/`
|
||||
Debugging headquarters (`/logs/app.txt`):
|
||||
- **[APP]**: Core logs
|
||||
- **[SERVER]**: API drama
|
||||
- **[SPECS]**: Hardware confessions
|
||||
**Keep data (reinstall later):** Just uninstall the application
|
||||
**Remove everything:** Delete the Jan data folder after uninstalling
|
||||
|
||||
### `models/`
|
||||
The silicon brain collection. Each model has its own `model.json`.
|
||||
|
||||
<Callout type="info">
|
||||
Full parameters: [here](/docs/model-parameters)
|
||||
</Callout>
|
||||
|
||||
### `threads/`
|
||||
Chat archive. Each thread (`/threads/jan_unixstamp/`) contains:
|
||||
|
||||
- `messages.jsonl`:
|
||||
```json
|
||||
{
|
||||
"completed_at": 0,
|
||||
"content": [
|
||||
{
|
||||
"text": {
|
||||
"annotations": [],
|
||||
"value": "Hello! I can help you with various tasks. I can search for information on the internet, including news, videos, images, shopping, and more. I can also scrape webpages to extract specific information. Let me know what you need!"
|
||||
},
|
||||
"type": "text"
|
||||
}
|
||||
],
|
||||
"created_at": 1751012639307,
|
||||
"id": "01JYR7S0JB5ZBGMJV52KWMW5VW",
|
||||
"metadata": {
|
||||
"assistant": {
|
||||
"avatar": "👋",
|
||||
"id": "jan",
|
||||
"instructions": "You have access to a set of tools to help you answer the user's question. You can use only one tool per message, and you'll receive the result of that tool in the user's next response. To complete a task, use tools step by step—each step should be guided by the outcome of the previous one.\nTool Usage Rules:\n1. Always provide the correct values as arguments when using tools. Do not pass variable names—use actual values instead.\n2. You may perform multiple tool steps to complete a task.\n3. Avoid repeating a tool call with exactly the same parameters to prevent infinite loops.",
|
||||
"name": "Jan",
|
||||
"parameters": ""
|
||||
},
|
||||
"tokenSpeed": {
|
||||
"lastTimestamp": 1751012637097,
|
||||
"message": "01JYR7S0GW5M9PSHMRE7T8VQJM",
|
||||
"tokenCount": 49,
|
||||
"tokenSpeed": 22.653721682847895
|
||||
}
|
||||
},
|
||||
"object": "thread.message",
|
||||
"role": "assistant",
|
||||
"status": "ready",
|
||||
"thread_id": "8f2c9922-db49-4d1e-8620-279c05baf2d0",
|
||||
"type": "text"
|
||||
}
|
||||
```
|
||||
|
||||
- `thread.json`:
|
||||
|
||||
| Parameter | Description |
|
||||
|------------|------------------------------------------------|
|
||||
| assistants | Assistant configuration clone |
|
||||
| created | Creation timestamp |
|
||||
| id | Thread identifier |
|
||||
| metadata | Additional thread data |
|
||||
| model | Active model settings |
|
||||
| object | OpenAI compatibility marker |
|
||||
| title | Thread name |
|
||||
| updated | Updated timestamp |
|
||||
|
||||
|
||||
|
||||
|
||||
## Delete Jan Data
|
||||
Uninstall guides: [Mac](/docs/desktop/mac#step-2-clean-up-data-optional),
|
||||
[Windows](/docs/desktop/windows#step-2-handle-jan-data), or [Linux](docs/desktop/linux#uninstall-jan).
|
||||
Detailed uninstall guides:
|
||||
- [macOS](/docs/desktop/mac#step-2-clean-up-data-optional)
|
||||
- [Windows](/docs/desktop/windows#step-2-handle-jan-data)
|
||||
- [Linux](/docs/desktop/linux#uninstall-jan)
|
||||
|
||||
@ -4,15 +4,15 @@ description: Run Jan as a local AI server with OpenAI-compatible API for buildin
|
||||
keywords:
|
||||
[
|
||||
Jan,
|
||||
Jan Local Server,
|
||||
local AI server,
|
||||
OpenAI API,
|
||||
local API,
|
||||
AI server,
|
||||
self-hosted AI,
|
||||
local LLM server,
|
||||
AI development,
|
||||
self-hosted AI,
|
||||
offline AI,
|
||||
privacy-focused AI,
|
||||
API integration,
|
||||
local LLM server,
|
||||
]
|
||||
---
|
||||
|
||||
@ -21,124 +21,181 @@ import FAQBox from '@/components/FaqBox'
|
||||
|
||||
# Jan Local Server
|
||||
|
||||
|
||||
|
||||
|
||||
Jan Local Server provides an OpenAI-compatible API that runs 100% offline on your machine. Build AI applications
|
||||
using familiar API patterns while maintaining full control over your data and models with complete
|
||||
[privacy](https://www.reuters.com/legal/legalindustry/privacy-paradox-with-ai-2023-10-31/).
|
||||
Jan Local Server provides an OpenAI-compatible API that runs entirely on your computer. Build AI applications using familiar API patterns while keeping complete control over your data and models.
|
||||
|
||||
Jan is powered by [Llama.cpp](https://github.com/ggerganov/llama.cpp), a local AI engine that provides an OpenAI-compatible
|
||||
API that can run in the background by default at `https://localhost:1337` (or your custom port). This enables you to power all sorts of
|
||||
applications with AI capabilities from your laptop/PC. For example, you can connect local tools like [Continue](https://jan.ai/docs/server-examples/continue-dev)
|
||||
and [Cline](https://cline.bot/) to Jan and power them using your favorite models.
|
||||
## How It Works
|
||||
|
||||
Jan doesn't limit you to locally hosted models, meaning, you can create an API key from your favorite model provider,
|
||||
add it to Jan via the configuration's page and start talking to your favorite models.
|
||||
Jan runs a local server powered by [llama.cpp](https://github.com/ggerganov/llama.cpp) that provides an OpenAI-compatible API. By default, it runs at `https://localhost:1337` and works completely offline.
|
||||
|
||||
### Features
|
||||
**What this enables:**
|
||||
- Connect development tools like [Continue](./continue-dev) and [Cline](https://cline.bot/) to Jan
|
||||
- Build AI applications without cloud dependencies
|
||||
- Use both local and cloud models through the same API
|
||||
- Maintain full privacy for local model interactions
|
||||
|
||||
- Download popular open-source LLMs (Llama3, Gemma3, Qwen3, and more) from the HuggingFace [Model Hub](./docs/manage-models.mdx)
|
||||
or import any GGUF files (the model format used by llama.cpp) available locally
|
||||
- Connect to [cloud services](/docs/remote-models/openai) (OpenAI, Anthropic, Mistral, Groq, etc.)
|
||||
- [Chat](./docs/threads.mdx) with AI models & [customize their parameters](/docs/model-parameters.mdx) via our
|
||||
intuitive interface
|
||||
- Use our [local API server](https://jan.ai/api-reference) with an OpenAI-equivalent API to power other apps.
|
||||
## Key Features
|
||||
|
||||
### Philosophy
|
||||
**Local AI Models**
|
||||
- Download popular open-source models (Llama, Gemma, Qwen) from Hugging Face
|
||||
- Import any GGUF files from your computer
|
||||
- Run models completely offline
|
||||
|
||||
Jan is built to be [user-owned](about#-user-owned), this means that Jan is:
|
||||
- Truly open source via the [Apache 2.0 license](https://github.com/menloresearch/jan/blob/dev/LICENSE)
|
||||
- [Data is stored locally, following one of the many local-first principles](https://www.inkandswitch.com/local-first)
|
||||
- Internet is optional, Jan can run 100% offline
|
||||
- Free choice of AI models, both local and cloud-based
|
||||
- We do not collect or sell user data. See our [Privacy Policy](./privacy).
|
||||
**Cloud Integration**
|
||||
- Connect to cloud services (OpenAI, Anthropic, Mistral, Groq)
|
||||
- Use your own API keys
|
||||
- Switch between local and cloud models seamlessly
|
||||
|
||||
<Callout>
|
||||
You can read more about our [philosophy](/about#philosophy) here.
|
||||
**Developer-Friendly**
|
||||
- OpenAI-compatible API for easy integration
|
||||
- Chat interface for testing and configuration
|
||||
- Model parameter customization
|
||||
|
||||
**Complete Privacy**
|
||||
- All data stored locally
|
||||
- No cloud dependencies for local models
|
||||
- You control what data leaves your machine
|
||||
|
||||
## Why Choose Jan?
|
||||
|
||||
**Truly Open Source**
|
||||
- Apache 2.0 license - no restrictions
|
||||
- Community-driven development
|
||||
- Full transparency
|
||||
|
||||
**Local-First Design**
|
||||
- Works 100% offline with local models
|
||||
- Data stays on your machine
|
||||
- No vendor lock-in
|
||||
|
||||
**Flexible Model Support**
|
||||
- Your choice of AI models
|
||||
- Both local and cloud options
|
||||
- Easy model switching
|
||||
|
||||
**No Data Collection**
|
||||
- We don't collect or sell user data
|
||||
- Local conversations stay local
|
||||
- [Read our Privacy Policy](./privacy)
|
||||
|
||||
<Callout type="info">
|
||||
Jan follows [local-first principles](https://www.inkandswitch.com/local-first) - your data belongs to you and stays on your device.
|
||||
</Callout>
|
||||
|
||||
### Inspirations
|
||||
## Philosophy
|
||||
|
||||
Jan is inspired by the concepts of [Calm Computing](https://en.wikipedia.org/wiki/Calm_technology), and the Disappearing Computer.
|
||||
Jan is built to be **user-owned**. This means:
|
||||
- **True open source** - Apache 2.0 license with no hidden restrictions
|
||||
- **Local data storage** - following [local-first principles](https://www.inkandswitch.com/local-first)
|
||||
- **Internet optional** - works completely offline
|
||||
- **Free choice** - use any AI models you want
|
||||
- **No surveillance** - we don't collect or sell your data
|
||||
|
||||
## Acknowledgements
|
||||
Read more about our [philosophy](/about#philosophy).
|
||||
|
||||
Jan is built on the shoulders of many open-source projects like:
|
||||
## Inspiration
|
||||
|
||||
- [Llama.cpp](https://github.com/ggerganov/llama.cpp/blob/master/LICENSE)
|
||||
- [Scalar](https://github.com/scalar/scalar)
|
||||
Jan draws inspiration from [Calm Computing](https://en.wikipedia.org/wiki/Calm_technology) and the Disappearing Computer - technology that works quietly in the background without demanding constant attention.
|
||||
|
||||
## FAQs
|
||||
## Built With
|
||||
|
||||
Jan stands on the shoulders of excellent open-source projects:
|
||||
- [llama.cpp](https://github.com/ggerganov/llama.cpp) - Local AI model inference
|
||||
- [Scalar](https://github.com/scalar/scalar) - API documentation
|
||||
|
||||
## Frequently Asked Questions
|
||||
|
||||
<FAQBox title="What is Jan?">
|
||||
Jan is a customizable AI assistant that can run offline on your computer - a privacy-focused alternative to tools like
|
||||
ChatGPT, Anthropic's Claude, and Google Gemini, with optional cloud AI support.
|
||||
Jan is a privacy-focused AI assistant that runs locally on your computer. It's an alternative to ChatGPT, Claude, and other cloud-based AI tools, with optional cloud AI support when you want it.
|
||||
</FAQBox>
|
||||
|
||||
<FAQBox title="How do I get started with Jan?">
|
||||
Download Jan on your computer, download a model or add API key for a cloud-based one, and start chatting. For
|
||||
detailed setup instructions, see our [Quick Start](/docs/quickstart) guide.
|
||||
<FAQBox title="How do I get started?">
|
||||
Download Jan, add a model (either download locally or add a cloud API key), and start chatting. Check our [Quick Start guide](/docs/quickstart) for detailed setup instructions.
|
||||
</FAQBox>
|
||||
|
||||
<FAQBox title="Is Jan compatible with my system?">
|
||||
Jan supports all major operating systems,
|
||||
- [Mac](/docs/desktop/mac#compatibility)
|
||||
- [Windows](/docs/desktop/windows#compatibility)
|
||||
- [Linux](/docs/desktop/linux)
|
||||
<FAQBox title="What systems does Jan support?">
|
||||
Jan works on all major operating systems:
|
||||
- [macOS](/docs/desktop/mac#compatibility) - Intel and Apple Silicon
|
||||
- [Windows](/docs/desktop/windows#compatibility) - x64 systems
|
||||
- [Linux](/docs/desktop/linux) - Most distributions
|
||||
|
||||
Hardware compatibility includes:
|
||||
- NVIDIA GPUs (CUDA)
|
||||
- AMD GPUs (Vulkan)
|
||||
- Intel Arc GPUs (Vulkan)
|
||||
Jan supports various hardware:
|
||||
- NVIDIA GPUs (CUDA acceleration)
|
||||
- AMD GPUs (Vulkan support)
|
||||
- Intel Arc GPUs (Vulkan support)
|
||||
- Any GPU with Vulkan support
|
||||
- CPU-only operation
|
||||
</FAQBox>
|
||||
|
||||
<FAQBox title="How does Jan protect my privacy?">
|
||||
Jan prioritizes privacy by:
|
||||
- Running 100% offline with locally-stored data
|
||||
- Using open-source models that keep your conversations private
|
||||
- Storing all files and chat history on your device in the [Jan Data Folder](/docs/data-folder)
|
||||
- Never collecting or selling your data
|
||||
Jan prioritizes privacy through:
|
||||
- **100% offline operation** with local models
|
||||
- **Local data storage** - everything stays on your device
|
||||
- **Open-source transparency** - you can verify what Jan does
|
||||
- **No data collection** - we never see your conversations
|
||||
|
||||
<Callout type="warning">
|
||||
When using third-party cloud AI services through Jan, their data policies apply. Check their privacy terms.
|
||||
When using cloud AI services, their privacy policies apply. Jan doesn't add any tracking.
|
||||
</Callout>
|
||||
|
||||
You can optionally share anonymous usage statistics to help improve Jan, but your conversations are never
|
||||
shared. See our complete [Privacy Policy](./docs/privacy).
|
||||
All your files and chat history are stored locally in the [Jan Data Folder](./data-folder). See our complete [Privacy Policy](./privacy).
|
||||
</FAQBox>
|
||||
|
||||
<FAQBox title="What models can I use with Jan?">
|
||||
<FAQBox title="What AI models can I use?">
|
||||
**Local models:**
|
||||
- Download optimized models from the [Jan Hub](/docs/manage-models)
|
||||
- Import GGUF models from Hugging Face or your local files
|
||||
- Connect to cloud providers like OpenAI, Anthropic, Mistral and Groq (requires your own API keys)
|
||||
- Import GGUF models from Hugging Face
|
||||
- Use any compatible local model files
|
||||
|
||||
**Cloud models:**
|
||||
- OpenAI (GPT-4, ChatGPT)
|
||||
- Anthropic (Claude)
|
||||
- Mistral, Groq, and others
|
||||
- Bring your own API keys
|
||||
</FAQBox>
|
||||
|
||||
<FAQBox title="Is Jan really free? What's the catch?">
|
||||
Jan is completely free and open-source with no subscription fees for local models and features. When using cloud-based
|
||||
models (like GPT-4o or Claude Sonnet 3.7), you'll only pay the standard rates to those providers—we add no markup.
|
||||
<FAQBox title="Is Jan really free?">
|
||||
Yes! Jan is completely free and open-source with no subscription fees.
|
||||
|
||||
**What's free:**
|
||||
- Jan application and all features
|
||||
- Local model usage (once downloaded)
|
||||
- Local server and API
|
||||
|
||||
**What costs money:**
|
||||
- Cloud model usage (you pay providers directly)
|
||||
- We add no markup to cloud service costs
|
||||
</FAQBox>
|
||||
|
||||
<FAQBox title="Can I use Jan offline?">
|
||||
Yes! Once you've downloaded a local model, Jan works completely offline with no internet connection needed.
|
||||
<FAQBox title="Can Jan work offline?">
|
||||
Absolutely! Once you download a local model, Jan works completely offline with no internet connection needed. This is one of Jan's core features.
|
||||
</FAQBox>
|
||||
|
||||
<FAQBox title="How can I contribute or get community help?">
|
||||
- Join our [Discord community](https://discord.gg/qSwXFx6Krr) to connect with other users
|
||||
- Contribute through [GitHub](https://github.com/menloresearch/jan) (no permission needed!)
|
||||
- Get troubleshooting help in our [Discord](https://discord.com/invite/FTk2MvZwJH) channel [#🆘|jan-help](https://discord.com/channels/1107178041848909847/1192090449725358130)
|
||||
- Check our [Troubleshooting](./docs/troubleshooting) guide for common issues
|
||||
<FAQBox title="How can I get help or contribute?">
|
||||
**Get help:**
|
||||
- Join our [Discord community](https://discord.gg/qSwXFx6Krr)
|
||||
- Check the [Troubleshooting guide](./troubleshooting)
|
||||
- Ask in [#🆘|jan-help](https://discord.com/channels/1107178041848909847/1192090449725358130)
|
||||
|
||||
**Contribute:**
|
||||
- Contribute on [GitHub](https://github.com/menloresearch/jan)
|
||||
- No permission needed to submit improvements
|
||||
- Help other users in Discord
|
||||
</FAQBox>
|
||||
|
||||
<FAQBox title="Can I self-host Jan?">
|
||||
Yes! We fully support the self-hosted movement. Either download Jan directly or fork it on
|
||||
[GitHub repository](https://github.com/menloresearch/jan) and build it from source.
|
||||
Yes! We fully support self-hosting. You can:
|
||||
- Download Jan directly for personal use
|
||||
- Fork the [GitHub repository](https://github.com/menloresearch/jan)
|
||||
- Build from source
|
||||
- Deploy on your own infrastructure
|
||||
</FAQBox>
|
||||
|
||||
<FAQBox title="What does Jan stand for?">
|
||||
Jan stands for "Just a Name". We are, admittedly, bad at marketing 😂.
|
||||
<FAQBox title="What does 'Jan' stand for?">
|
||||
"Just a Name" - we admit we're not great at marketing! 😄
|
||||
</FAQBox>
|
||||
|
||||
<FAQBox title="Are you hiring?">
|
||||
Yes! We love hiring from our community. Check out our open positions at [Careers](https://menlo.bamboohr.com/careers).
|
||||
Yes! We love hiring from our community. Check our open positions at [Careers](https://menlo.bamboohr.com/careers).
|
||||
</FAQBox>
|
||||
|
||||
@ -1,165 +1,171 @@
|
||||
---
|
||||
title: llama.cpp
|
||||
description: A step-by-step guide on how to customize the llama.cpp engine.
|
||||
title: llama.cpp Engine
|
||||
description: Configure Jan's local AI engine for optimal performance.
|
||||
keywords:
|
||||
[
|
||||
Jan,
|
||||
Customizable Intelligence, LLM,
|
||||
local AI,
|
||||
privacy focus,
|
||||
free and open source,
|
||||
private and offline,
|
||||
conversational AI,
|
||||
no-subscription fee,
|
||||
large language models,
|
||||
Llama CPP integration,
|
||||
llama.cpp Engine,
|
||||
Intel CPU,
|
||||
AMD CPU,
|
||||
NVIDIA GPU,
|
||||
AMD GPU Radeon,
|
||||
Apple Silicon,
|
||||
Intel Arc GPU,
|
||||
llama.cpp,
|
||||
AI engine,
|
||||
local models,
|
||||
performance,
|
||||
GPU acceleration,
|
||||
CPU processing,
|
||||
model optimization,
|
||||
]
|
||||
---
|
||||
|
||||
import { Tabs } from 'nextra/components'
|
||||
import { Callout, Steps } from 'nextra/components'
|
||||
import { Settings, EllipsisVertical, Plus, FolderOpen, Pencil } from 'lucide-react'
|
||||
import { Settings } from 'lucide-react'
|
||||
|
||||
# Local Model Management
|
||||
# Local AI Engine (llama.cpp)
|
||||
|
||||
## Overview
|
||||
Jan uses **llama.cpp** for running local AI models. You can find its settings in **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Local Engine** > **llama.cpp**:
|
||||
llama.cpp is the engine that runs AI models locally on your computer. It's what makes Jan work without needing internet or cloud services.
|
||||
|
||||
## Accessing Engine Settings
|
||||
|
||||
Find llama.cpp settings at **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Local Engine** > **llama.cpp**:
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
These settings are for advanced users, you would want to check these settings when:
|
||||
- Your AI models are running slowly or not working
|
||||
- You've installed new hardware (like a graphics card)
|
||||
- You want to tinker & test performance with different [backends](/docs/local-engines/llama-cpp#available-backends)
|
||||
|
||||
## Engine Version and Updates
|
||||
- **Engine Version**: View current version of llama.cpp engine
|
||||
- **Check Updates**: Verify if a newer version is available & install available updates when it's available
|
||||
|
||||
|
||||
## Available Backends
|
||||
|
||||
Jan offers different backend variants for **llama.cpp** based on your operating system, you can:
|
||||
- Download different backends as needed
|
||||
- Switch between backends for different hardware configurations
|
||||
- View currently installed backends in the list
|
||||
|
||||
<Callout type="warning">
|
||||
Choose the backend that matches your hardware. Using the wrong variant may cause performance issues or prevent models from loading.
|
||||
<Callout type="info">
|
||||
Most users don't need to change these settings. Jan picks good defaults for your hardware automatically.
|
||||
</Callout>
|
||||
|
||||
## When to Adjust Settings
|
||||
|
||||
You might need to modify these settings if:
|
||||
- Models load slowly or don't work
|
||||
- You've installed new hardware (like a graphics card)
|
||||
- You want to optimize performance for your specific setup
|
||||
|
||||
## Engine Management
|
||||
|
||||
| Feature | What It Does | When You Need It |
|
||||
|---------|-------------|------------------|
|
||||
| **Engine Version** | Shows current llama.cpp version | Check compatibility with newer models |
|
||||
| **Check Updates** | Downloads engine updates | When new models require updated engine |
|
||||
| **Backend Selection** | Choose hardware-optimized version | After hardware changes or performance issues |
|
||||
|
||||
## Hardware Backends
|
||||
|
||||
Different backends are optimized for different hardware. Pick the one that matches your computer:
|
||||
|
||||
<Tabs items={['Windows', 'Linux', 'macOS']}>
|
||||
|
||||
<Tabs.Tab>
|
||||
### CUDA Support (NVIDIA GPUs)
|
||||
- `llama.cpp-avx-cuda-11-7`
|
||||
- `llama.cpp-avx-cuda-12-0`
|
||||
- `llama.cpp-avx2-cuda-11-7`
|
||||
- `llama.cpp-avx2-cuda-12-0`
|
||||
- `llama.cpp-avx512-cuda-11-7`
|
||||
- `llama.cpp-avx512-cuda-12-0`
|
||||
- `llama.cpp-noavx-cuda-11-7`
|
||||
- `llama.cpp-noavx-cuda-12-0`
|
||||
|
||||
### NVIDIA Graphics Cards (Fastest)
|
||||
**For CUDA 12.0:**
|
||||
- `llama.cpp-avx2-cuda-12-0` (most common)
|
||||
- `llama.cpp-avx512-cuda-12-0` (newer Intel/AMD CPUs)
|
||||
|
||||
**For CUDA 11.7:**
|
||||
- `llama.cpp-avx2-cuda-11-7` (older drivers)
|
||||
|
||||
### CPU Only
|
||||
- `llama.cpp-avx`
|
||||
- `llama.cpp-avx2`
|
||||
- `llama.cpp-avx512`
|
||||
- `llama.cpp-noavx`
|
||||
- `llama.cpp-avx2` (modern CPUs)
|
||||
- `llama.cpp-avx` (older CPUs)
|
||||
- `llama.cpp-noavx` (very old CPUs)
|
||||
|
||||
### Other Accelerators
|
||||
- `llama.cpp-vulkan`
|
||||
|
||||
<Callout type="info">
|
||||
- For detailed hardware compatibility, please visit our guide for [Windows](/docs/desktop/windows#compatibility).
|
||||
- AVX, AVX2, and AVX-512 are CPU instruction sets. For best performance, use the most advanced instruction set your CPU supports.
|
||||
- CUDA versions should match your installed NVIDIA drivers.
|
||||
</Callout>
|
||||
### Other Graphics Cards
|
||||
- `llama.cpp-vulkan` (AMD, Intel Arc)
|
||||
|
||||
</Tabs.Tab>
|
||||
|
||||
<Tabs.Tab>
|
||||
### CUDA Support (NVIDIA GPUs)
|
||||
- `llama.cpp-avx-cuda-11-7`
|
||||
- `llama.cpp-avx-cuda-12-0`
|
||||
- `llama.cpp-avx2-cuda-11-7`
|
||||
- `llama.cpp-avx2-cuda-12-0`
|
||||
- `llama.cpp-avx512-cuda-11-7`
|
||||
- `llama.cpp-avx512-cuda-12-0`
|
||||
- `llama.cpp-noavx-cuda-11-7`
|
||||
- `llama.cpp-noavx-cuda-12-0`
|
||||
|
||||
### NVIDIA Graphics Cards
|
||||
- `llama.cpp-avx2-cuda-12-0` (recommended)
|
||||
- `llama.cpp-avx2-cuda-11-7` (older drivers)
|
||||
|
||||
### CPU Only
|
||||
- `llama.cpp-avx`
|
||||
- `llama.cpp-avx2`
|
||||
- `llama.cpp-avx512`
|
||||
- `llama.cpp-noavx`
|
||||
- `llama.cpp-avx2` (modern CPUs)
|
||||
- `llama.cpp-arm64` (ARM processors)
|
||||
|
||||
### Other Accelerators
|
||||
- `llama.cpp-vulkan`
|
||||
- `llama.cpp-arm64`
|
||||
|
||||
<Callout type="info">
|
||||
- For detailed hardware compatibility, please visit our guide for [Linux](docs/desktop/linux).
|
||||
- AVX, AVX2, and AVX-512 are CPU instruction sets. For best performance, use the most advanced instruction set your CPU supports.
|
||||
- CUDA versions should match your installed NVIDIA drivers.
|
||||
</Callout>
|
||||
### Other Graphics Cards
|
||||
- `llama.cpp-vulkan` (AMD, Intel graphics)
|
||||
|
||||
</Tabs.Tab>
|
||||
|
||||
<Tabs.Tab>
|
||||
### Apple Silicon
|
||||
- `llama.cpp-mac-arm64`: For M1/M2/M3 Macs
|
||||
|
||||
### Intel
|
||||
- `llama.cpp-mac-amd64`: For Intel-based Macs
|
||||
### Apple Silicon (M1/M2/M3/M4)
|
||||
- `llama.cpp-mac-arm64` (recommended)
|
||||
|
||||
### Intel Macs
|
||||
- `llama.cpp-mac-amd64`
|
||||
|
||||
<Callout type="info">
|
||||
For detailed hardware compatibility, please visit our guide for [Mac](/docs/desktop/mac#compatibility).
|
||||
Apple Silicon automatically uses GPU acceleration through Metal.
|
||||
</Callout>
|
||||
|
||||
|
||||
</Tabs.Tab>
|
||||
|
||||
</Tabs>
|
||||
|
||||
## Performance Settings
|
||||
|
||||
|
||||
| Setting | Description | Default |
|
||||
|---------|-------------|---------|
|
||||
| **Continuous Batching** | - Enables processing multiple requests simultaneously<br></br>- Improves throughput for concurrent operations | Enabled |
|
||||
| **Parallel Operations** | - Number of prompts to run in parallel<br></br>- Affects model inference speed | 4 |
|
||||
| **CPU Threads** | - Number of CPU cores to use when running without GPU<br></br>- Higher thread counts may improve performance but increase CPU usage | Auto-detected based on your system's capabilities |
|
||||
|
||||
|
||||
| Setting | What It Does | Recommended | Impact |
|
||||
|---------|-------------|-------------|---------|
|
||||
| **Continuous Batching** | Handle multiple requests simultaneously | Enabled | Faster when using tools or multiple chats |
|
||||
| **Parallel Operations** | Number of concurrent requests | 4 | Higher = more multitasking, uses more memory |
|
||||
| **CPU Threads** | Processor cores to use | Auto | More threads can speed up CPU processing |
|
||||
|
||||
## Memory Settings
|
||||
|
||||
| Setting | Description | Default |
|
||||
|---------|-------------|---------|
|
||||
| **Flash Attention** | - Optimizes attention computation<br></br>- Reduces memory usage<br></br>- Recommended for most cases | Enabled |
|
||||
| **Caching** | - Enable to store recent prompts and responses<br></br>- Improves response time for repeated prompts | Enabled |
|
||||
| **KV Cache Type** | - KV cache implementation type; controls memory usage and precision trade-off<br></br>- Options:<br></br>• f16 (most stable)<br></br>• q8_0 (balanced)<br></br>• q4_0 (lowest memory) | f16 |
|
||||
| **mmap** | - Enables memory-mapped model loading<br></br>- Reduces memory usage<br></br>- Recommended for large models | Enabled |
|
||||
| **Context Shift** | - Automatically shifts the context window when the model is unable to process the entire prompt<br/> - Ensures that the most relevant information is always included <br/> - Recommended for long conversations and multiple tool calls | Disabled |
|
||||
| Setting | What It Does | Recommended | When to Change |
|
||||
|---------|-------------|-------------|----------------|
|
||||
| **Flash Attention** | Efficient memory usage | Enabled | Leave enabled unless problems occur |
|
||||
| **Caching** | Remember recent conversations | Enabled | Speeds up follow-up questions |
|
||||
| **KV Cache Type** | Memory vs quality trade-off | f16 | Change to q8_0 if low on memory |
|
||||
| **mmap** | Efficient model loading | Enabled | Helps with large models |
|
||||
| **Context Shift** | Handle very long conversations | Disabled | Enable for very long chats |
|
||||
|
||||
### Memory Options Explained
|
||||
- **f16**: Best quality, uses more memory
|
||||
- **q8_0**: Balanced memory and quality
|
||||
- **q4_0**: Least memory, slight quality reduction
|
||||
|
||||
## Best Practices
|
||||
- Start with default settings
|
||||
- Adjust based on your hardware capabilities
|
||||
- Monitor system performance
|
||||
- Test changes with your specific use case
|
||||
## Quick Troubleshooting
|
||||
|
||||
**Models won't load:**
|
||||
- Try a different backend
|
||||
- Check available RAM/VRAM
|
||||
- Update engine version
|
||||
|
||||
**Slow performance:**
|
||||
- Verify GPU acceleration is active
|
||||
- Close memory-intensive applications
|
||||
- Increase GPU Layers in model settings
|
||||
|
||||
**Out of memory:**
|
||||
- Change KV Cache Type to q8_0
|
||||
- Reduce Context Size in model settings
|
||||
- Try a smaller model
|
||||
|
||||
**Crashes or errors:**
|
||||
- Switch to a more stable backend (avx instead of avx2)
|
||||
- Update graphics drivers
|
||||
- Check system temperature
|
||||
|
||||
## Quick Setup Guide
|
||||
|
||||
**Most users:**
|
||||
1. Use default settings
|
||||
2. Only change if problems occur
|
||||
|
||||
**NVIDIA GPU users:**
|
||||
1. Download CUDA backend
|
||||
2. Ensure GPU Layers is set high
|
||||
3. Enable Flash Attention
|
||||
|
||||
**Performance optimization:**
|
||||
1. Enable Continuous Batching
|
||||
2. Use appropriate backend for hardware
|
||||
3. Monitor memory usage
|
||||
|
||||
<Callout type="info">
|
||||
Performance impact varies by hardware, model size, and usage patterns.
|
||||
The default settings work well for most hardware. Only adjust these if you're experiencing specific issues or want to optimize for your particular setup.
|
||||
</Callout>
|
||||
|
||||
@ -1,201 +0,0 @@
|
||||
---
|
||||
title: Model Context Protocol
|
||||
description: Manage your interaction with AI locally.
|
||||
keywords:
|
||||
[
|
||||
Jan,
|
||||
Customizable Intelligence, LLM,
|
||||
local AI,
|
||||
privacy focus,
|
||||
free and open source,
|
||||
private and offline,
|
||||
conversational AI,
|
||||
no-subscription fee,
|
||||
large language models,
|
||||
threads,
|
||||
chat history,
|
||||
thread history,
|
||||
]
|
||||
---
|
||||
import { Callout, Steps } from 'nextra/components'
|
||||
|
||||
# Using the Model Context Protocol (MCP) in Jan
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph "What is MCP?"
|
||||
You[You using Jan Desktop]
|
||||
Claude[Jan AI Assistant]
|
||||
|
||||
subgraph "Your Connected Tools"
|
||||
Files[📁 Your Files<br/>Documents, folders,<br/>text files]
|
||||
Database[📊 Your Data<br/>Spreadsheets,<br/>databases]
|
||||
WebServices[🌐 Online Services<br/>GitHub, Slack,<br/>Google Drive]
|
||||
Custom[🔧 Custom Tools<br/>Special programs<br/>you've added]
|
||||
end
|
||||
|
||||
subgraph "What Jan Can Do"
|
||||
Read[Read & Understand<br/>- View your files<br/>- Check your data<br/>- See updates]
|
||||
Action[Take Actions<br/>- Search for info<br/>- Create content<br/>- Run commands]
|
||||
Templates[Use Templates<br/>- Common tasks<br/>- Saved prompts<br/>- Workflows]
|
||||
end
|
||||
end
|
||||
|
||||
You --> Claude
|
||||
Claude -->|"Can I see this file?"| Files
|
||||
Claude -->|"What's in my database?"| Database
|
||||
Claude -->|"Check my GitHub"| WebServices
|
||||
Claude -->|"Run this tool"| Custom
|
||||
|
||||
Files --> Read
|
||||
Database --> Read
|
||||
WebServices --> Action
|
||||
Custom --> Templates
|
||||
|
||||
style You fill:transparent
|
||||
style Claude fill:transparent
|
||||
style Files fill:transparent
|
||||
style Database fill:transparent
|
||||
style WebServices fill:transparent
|
||||
style Custom fill:transparent
|
||||
style Read fill:transparent
|
||||
style Action fill:transparent
|
||||
style Templates fill:transparent
|
||||
```
|
||||
|
||||
|
||||
Jan now supports the **Model Context Protocol (MCP)**, an open standard designed to allow language models to
|
||||
interact with external tools and data sources.
|
||||
|
||||
MCPs act as a common interface, standardizing the way an AI model can interact with external tools and data
|
||||
sources. This enables a model to connect to any MCP-compliant tool without requiring custom
|
||||
integration work. The way this works is via clients and servers. Clients are connected to an AI model and a host
|
||||
where a user will describe the task needed to be done. These applications hosting client will want to connect
|
||||
to different data sources to accomplish a task, for example, notion, google sheets, or even custom APIs. These
|
||||
applications will be connected to a server with prompts, tools, and data sources which will be used to complete
|
||||
the task.
|
||||
|
||||
Jan is an MCP host that allows you to download different clients and servers and use them to accomplish a task.
|
||||
|
||||
This document outlines the benefits, risks, and implementation of MCPs within Jan.
|
||||
|
||||
## Core Benefits of MCP
|
||||
|
||||
Integrating MCP provides a structured way to extend the capabilities of the models you use in Jan. Here are the three
|
||||
|
||||
* **Standardization:** MCP aims to solve the "M x N" integration problem, where every model (M) needs a
|
||||
unique connector for every tool (N). By adapting to a single standard, any compliant model can interface with any compliant tool.
|
||||
* **Extensibility:** This allows you to augment your models with new abilities. For instance, an AI can be granted
|
||||
access to search your local codebase, query a database, or interact with web APIs, all through the same protocol.
|
||||
* **Flexibility:** Because the interface is standardized, you can swap out models or tools with minimal friction,
|
||||
making your workflows more modular and adaptable over time.
|
||||
|
||||
<Callout type="warning">
|
||||
Please note that not all models that you can download and use, whether in Jan or other tools, may be good at
|
||||
tool calling or compatible with MCP. Make sure that the model you choose is MCP-compliant before integrating
|
||||
it into your workflows. This might be available in the model card or you may need to implement it yourself to
|
||||
test the capabilities of the model.
|
||||
</Callout>
|
||||
|
||||
<Callout type="info">
|
||||
To use MCP effectively, ensure your AI model supports tool calling capabilities:
|
||||
- For cloud models (like Claude or GPT-4): Verify tool calling is enabled in your API settings
|
||||
- For local models: Enable tool calling in the model parameters [click the edit button in Model Capabilities](/docs/model-parameters#model-capabilities-edit-button)
|
||||
- Check the model's documentation to confirm MCP compatibility
|
||||
</Callout>
|
||||
|
||||
## Considerations and Risks
|
||||
|
||||
While powerful, MCP is an evolving standard, and its use requires careful consideration of the following points:
|
||||
|
||||
* **Security:** Granting a model access to external tools is a significant security consideration. A compromised
|
||||
tool or a malicious prompt could potentially lead to unintended actions or data exposure. Jan's implementation
|
||||
focuses on user-managed permissions to mitigate this risk, meaning, you have to turn on the permission for each
|
||||
tool individually.
|
||||
* **Standard Maturity:** As a relatively new protocol, best practices or sensible defaults are still being
|
||||
established. Users should be aware of potential issues like prompt injection, where an input could be crafted to
|
||||
misuse a tool's capabilities.
|
||||
* **Resource Management:** Active MCP connections may consume a portion of a model's context window, which could
|
||||
affect performance (i.e., the more tools the model and the larger the context of the conversation has the longer
|
||||
you will need to wait for a response). Efficient management of tools and their outputs is important.
|
||||
|
||||
|
||||
## Configure and Use MCPs within Jan
|
||||
|
||||
To illustrate how MCPs can be used within Jan, we will walk through an example using the [Browser MCP](https://browsermcp.io/).
|
||||
|
||||
Before we begin, you will need to enable Jan's MCP functionality at `Settings` > `MCP Servers`, and toggle
|
||||
the `Allow All MCP Tool Permission` switch ON.
|
||||
|
||||
|
||||
|
||||
Please note that you will also need to have **NodeJS** and/or **Python** installed on your machine. In case you don't
|
||||
have either, you can download them from the official websites at the links below:
|
||||
- [Node.js](https://nodejs.org/)
|
||||
- [Python](https://www.python.org/)
|
||||
|
||||
|
||||
### Browser MCP
|
||||
|
||||
- Click on the `+` sign on the upper right-hand corner of the MCP box.
|
||||
|
||||
|
||||
|
||||
- Enter the following details to configure the BrowserMCP:
|
||||
- **Server Name**: `browsermcp`
|
||||
- **Command**: `npx`
|
||||
- **Arguments**: `@browsermcp/mcp`
|
||||
- **Environment Variables**: You can leave this field empty.
|
||||
|
||||
|
||||
|
||||
- Check that the server has been activated successfully.
|
||||
|
||||
|
||||
|
||||
- Open your favorite chrome-based browser (e.g., Google Chrome, Brave, Vivaldi, Microsoft Edge, etc...) and navigate to the
|
||||
[Browser MCP Extension Page](https://chromewebstore.google.com/detail/browser-mcp-automate-your/bjfgambnhccakkhmkepdoekmckoijdlc).
|
||||
|
||||
|
||||
|
||||
- Make sure to enable the extension to run on private windows. Since Browser Use will have access to all sites you've
|
||||
already logged into in your regular browser session, it is best to give it a clean slate to start from.
|
||||
|
||||
|
||||
|
||||
- Enable the extension to run on private windows by clicking on it and Connecting to the Browser MCP server.
|
||||
|
||||
|
||||
|
||||
- Go back to Jan and pick a model with good tool use capabilities, for example, Claude 3.7 and 4 Sonnet, or Claude 4 Opus,
|
||||
and make sure to enable tool calling via the UI by going to **Model Providers > Anthropic** and, after you have entered your
|
||||
API key, enable tool from the **+** button.
|
||||
|
||||
|
||||
|
||||
You can check and see if this was accurate below.
|
||||
|
||||
|
||||
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- The MCP server won't connect even though I've already added it to my list of MCP Servers?
|
||||
- Make sure you have NodeJS and Python installed
|
||||
- Make sure you typed the commands correctly in the MCP Server form
|
||||
- Make sure the model you are using has tools enabled
|
||||
- Restart Jan
|
||||
- The open source model I picked won't use the MCPs I enabled.
|
||||
- Make sure the model you are using has tools enabled
|
||||
- Lots of open source models are not designed to use tools or simply don't work well with them, so you may need to try a different model
|
||||
- The model you have selected might be good at tool calling but it is possible that it does not support images, effectively making it unsuitable for some tools that take screenshots of a website like the Browser MCP
|
||||
|
||||
## Future Potential
|
||||
|
||||
This integration is the foundation for creating more capable and context-aware AI assistants within Jan. The
|
||||
long-term goal is to enable more sophisticated workflows that make use of your local environment securely as
|
||||
well as your favorite tools.
|
||||
|
||||
For example, an AI could cross-reference information between a local document and a remote API, or use a
|
||||
local script toanalyze data and then summarize the findings, all orchestrated through Jan's interface. As
|
||||
the MCP ecosystem grows, so will the potential applications within Jan.
|
||||
Binary file not shown.
|
Before Width: | Height: | Size: 262 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 618 KiB |
@ -1,232 +1,246 @@
|
||||
---
|
||||
title: Settings
|
||||
description: Explore how to adjust Jan's settings to suit your specific requirements.
|
||||
description: Configure Jan to work best for your needs and hardware.
|
||||
keywords:
|
||||
[
|
||||
Jan,
|
||||
Customizable Intelligence, LLM,
|
||||
local AI,
|
||||
privacy focus,
|
||||
free and open source,
|
||||
private and offline,
|
||||
conversational AI,
|
||||
no-subscription fee,
|
||||
large language models,
|
||||
Advanced Settings,
|
||||
HTTPS Proxy,
|
||||
SSL,
|
||||
settings,
|
||||
Jan settings,
|
||||
configuration,
|
||||
model management,
|
||||
privacy,
|
||||
hardware settings,
|
||||
local AI,
|
||||
customization,
|
||||
]
|
||||
---
|
||||
|
||||
import { Tabs, Steps, Callout } from 'nextra/components'
|
||||
import { Settings, EllipsisVertical, Plus, FolderOpen, Pencil } from 'lucide-react'
|
||||
|
||||
|
||||
# Settings
|
||||
|
||||
To access the **Settings**, click <Settings width={16} height={16} style={{display:"inline"}}/> icon in the bottom left corner of Jan.
|
||||
Access Jan's settings by clicking the <Settings width={16} height={16} style={{display:"inline"}}/> icon in the bottom left corner.
|
||||
|
||||
## Model Management
|
||||
## Managing AI Models
|
||||
|
||||
Manage your installed AI models in **Settings** > **Model Providers**:
|
||||
Find all model options at **Settings** > **Model Providers**:
|
||||
|
||||
### Import Models
|
||||
- **From Hugging Face:**
|
||||
- Enter a model's Hugging Face ID (e.g., `org/model_name_or_id`) in the Hub's search bar.
|
||||
- **Note:** Some models require a Hugging Face Access Token. Enter your token in **Settings > Model Providers > Hugging Face Access Token**.
|
||||
- **From Local Files:**
|
||||
- Click **Import Model** and select your GGUF files.
|
||||
### Adding Models
|
||||
|
||||
### Remove Models
|
||||
**From Hugging Face:**
|
||||
- Enter a model's ID (like `microsoft/DialoGPT-medium`) in the search bar
|
||||
- **Need authentication?** Some models require a Hugging Face token - add yours at **Settings > Model Providers > Hugging Face Access Token**
|
||||
|
||||
- Click the trash icon next to the **Start** button and then click **Delete**.
|
||||
**From Your Computer:**
|
||||
- Click **Import Model** and select GGUF files from your computer
|
||||
- Works with any compatible model files you've downloaded
|
||||
|
||||
### Start Models
|
||||
### Managing Existing Models
|
||||
|
||||
1. Open a new chat and select the model you want to start.
|
||||
2. Click the **Start** button on the **Settings > Model Providers**
|
||||
**Start a model:**
|
||||
1. Open a new chat and select the model you want
|
||||
2. Or go to **Settings > Model Providers** and click the **Start** button
|
||||
|
||||
### Hugging Face Access Token
|
||||
To download models from Hugging Face that require authentication, for example, like the llama models from meta:
|
||||
**Remove a model:**
|
||||
- Click the trash icon next to the **Start** button
|
||||
- Confirm deletion when prompted
|
||||
|
||||
### Hugging Face Token Setup
|
||||
|
||||
For restricted models (like Meta's Llama models):
|
||||
1. Get your token from [Hugging Face Tokens](https://huggingface.co/docs/hub/en/security-tokens)
|
||||
2. Enter it in **Settings > Model Providers > Hugging Face**.
|
||||
2. Add it at **Settings > Model Providers > Hugging Face**
|
||||
|
||||
## Model Settings (Gear Icon)
|
||||
## Model Configuration (Gear Icon)
|
||||
|
||||
|
||||
|
||||
Click the gear icon next to a model to configure advanced settings:
|
||||
- **Context Size**: Maximum prompt context length
|
||||
- **GPU Layers**: Number of model layers to offload to GPU. If you have an NVIDIA GPU and notice that your model won't fully load in it, you can reduce this value to load smaller parts of the model and try again.
|
||||
- **Temperature**: Controls randomness (higher = more random)
|
||||
- **Top K**: Limits token selection to K most likely next tokens (smaller K = more focused responses)
|
||||
- **Top P**: Limits token selection to tokens comprising P probability mass (smaller P = more focused responses)
|
||||
- **Min P**: Sets a minimum threshold for words the model can select (higher values filter out less likely words)
|
||||
- **Repeat Last N**: Determines how many recent words the model checks to avoid repetition
|
||||
- **Repeat Penalty**: Controls how strongly the model avoids repeating phrases (higher values reduce repetition)
|
||||
- **Presence Penalty**: Discourages reusing words that already appeared in the text (helps with variety)
|
||||
Click the gear icon next to any model to adjust how it behaves:
|
||||
|
||||
_See [Model Parameters](/docs/model-parameters) for a more detailed explanation._
|
||||
**Basic Settings:**
|
||||
- **Context Size**: How much conversation history the model remembers
|
||||
- **GPU Layers**: How much of the model runs on your graphics card (higher = faster, but uses more GPU memory)
|
||||
- **Temperature**: Controls creativity (0.1 = focused, 1.0 = creative)
|
||||
|
||||
|
||||
## Hardware
|
||||
|
||||
Monitor and manage system resources at **Settings > Hardware**:
|
||||
- **CPU, RAM, GPU**: View usage and specs
|
||||
- **GPU Acceleration**: Enable/disable and configure GPU settings
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
## Preferences
|
||||
|
||||
### Appearance & Theme
|
||||
Control the visual theme of Jan's interface with any color combo you'd like. You can also control the color use in the code blocks.
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
### Spell Check
|
||||
Jan includes a built-in spell check feature to help catch typing errors in your messages.
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
|
||||
## Privacy
|
||||
At **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Privacy**, you can control analytics & logs in Jan:
|
||||
|
||||
### Analytics
|
||||
Jan is built with privacy at its core. By default, no data is collected. Everything stays local on your device.
|
||||
You can help improve Jan by sharing anonymous usage data:
|
||||
1. Toggle on **Analytics** to share anonymous data
|
||||
2. You can change this setting at any time
|
||||
**Advanced Controls:**
|
||||
- **Top K & Top P**: Fine-tune how the model picks words (lower = more focused)
|
||||
- **Min P**: Minimum probability threshold for word selection
|
||||
- **Repeat Penalty**: Prevents the model from repeating itself too much
|
||||
- **Presence Penalty**: Encourages the model to use varied vocabulary
|
||||
|
||||
<Callout type="info">
|
||||
Read more about that we collect with opt-in users at [Privacy](/docs/privacy).
|
||||
For detailed explanations of these parameters, see our [Model Parameters Guide](/docs/model-parameters).
|
||||
</Callout>
|
||||
|
||||
## Hardware Monitoring
|
||||
|
||||
Check your computer's performance at **Settings** > **Hardware**:
|
||||
|
||||
- **CPU, RAM, GPU**: Real-time usage and specifications
|
||||
- **GPU Acceleration**: Turn GPU acceleration on/off
|
||||
- **Temperature monitoring**: Keep an eye on system heat
|
||||
|
||||
|
||||
|
||||
<Callout type="warning">
|
||||
If your computer gets very hot, consider using smaller models or reducing GPU layers.
|
||||
</Callout>
|
||||
|
||||
## Personalization
|
||||
|
||||
### Visual Appearance
|
||||
|
||||
Customize Jan's look at **Settings** > **Appearance**:
|
||||
- **Theme**: Choose light or dark mode
|
||||
- **Colors**: Pick your preferred color scheme
|
||||
- **Code highlighting**: Adjust syntax colors for programming discussions
|
||||
|
||||
|
||||
|
||||
### Writing Assistance
|
||||
|
||||
**Spell Check:** Jan can help catch typing mistakes in your messages.
|
||||
|
||||
|
||||
|
||||
## Privacy & Data Control
|
||||
|
||||
Access privacy settings at **Settings** > **Privacy**:
|
||||
|
||||
### Usage Analytics
|
||||
|
||||
**Default: No data collection.** Everything stays on your computer.
|
||||
|
||||
**Optional: Help improve Jan**
|
||||
- Toggle **Analytics** to share anonymous usage patterns
|
||||
- No conversations or personal data ever shared
|
||||
- Change this setting anytime
|
||||
|
||||
<Callout type="info">
|
||||
See exactly what we collect (with your permission) in our [Privacy Policy](/docs/privacy).
|
||||
</Callout>
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
### Log Management
|
||||
|
||||
**1. View Logs**
|
||||
- Logs are stored at:
|
||||
- App log: `~/Library/Application\ Support/jan/data/logs/app.log`
|
||||
- Cortex log: `~/Library/Application\ Support/jan/data/logs/cortex.log`
|
||||
- To open logs from Jan's interface: at **Logs**, click <FolderOpen width={16} height={16} style={{display:"inline"}}/> icon to open App Logs & Cortex Logs:
|
||||
**Viewing System Logs:**
|
||||
- Logs help troubleshoot problems
|
||||
- Click the folder icon to open App Logs and System Logs
|
||||
- Logs are automatically deleted after 24 hours
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
**2. Clear Logs**
|
||||
|
||||
Jan retains your logs for only **24 hours**. To remove all logs from Jan, at **Clear Logs**, click the **Clear** button:
|
||||
**Clearing Logs:**
|
||||
- Click **Clear** to remove all log files immediately
|
||||
- Useful before sharing your computer or troubleshooting
|
||||
|
||||
<Callout type="warning">
|
||||
This action cannot be undone.
|
||||
Clearing logs cannot be undone.
|
||||
</Callout>
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
### Data Folder Management
|
||||
|
||||
### Jan Data Folder
|
||||
Jan stores your data locally in your own filesystem in a universal file format. See detailed [Jan Folder Structure](docs/data-folder#folder-structure).
|
||||
Jan stores everything locally on your computer in standard file formats.
|
||||
|
||||
**1. Open Jan Data Folder**
|
||||
**Access Your Data:**
|
||||
- Click the folder icon to open Jan's data directory
|
||||
- Find your chat history, models, and settings
|
||||
- All files are yours to backup, move, or examine
|
||||
|
||||
At **Jan Data Folder**, click <FolderOpen width={16} height={16} style={{display:"inline"}}/> icon to open Jan application's folder:
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
**2. Edit Jan Data Folder**
|
||||
**Change Storage Location:**
|
||||
1. Click the pencil icon to edit the data folder location
|
||||
2. Choose an empty directory
|
||||
3. Confirm the move (original folder stays intact)
|
||||
4. Restart Jan to complete the change
|
||||
|
||||
1. At **Jan Data Folder**, click <Pencil width={16} height={16} style={{display:"inline"}}/> icon to edit Jan application's folder
|
||||
2. Choose a new directory & click **Select**, make sure the new folder is empty
|
||||
3. Confirmation pop-up shows up:
|
||||
|
||||
> Are you sure you want to relocate Jan Data Folder to `new directory`?
|
||||
Jan Data Folder will be duplicated into the new location while the original folder remains intact.
|
||||
An app restart will be required afterward.
|
||||
|
||||
4. Click **Yes, Proceed**
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
### HTTPs Proxy
|
||||
HTTPS Proxy encrypts data between your browser and the internet, making it hard for outsiders to intercept or read. It also helps you maintain your privacy and security while bypassing regional restrictions on the internet.
|
||||
|
||||
<Callout type="info">
|
||||
- Model download speeds may be affected due to the encryption/decryption process and your cloud service provider's networking
|
||||
- HTTPS Proxy does not affect the remote model usage.
|
||||
This duplicates your data to the new location - your original files stay safe.
|
||||
</Callout>
|
||||
|
||||
## Network Settings
|
||||
|
||||
### HTTPS Proxy Setup
|
||||
|
||||
If you need to connect through a corporate network or want enhanced privacy:
|
||||
|
||||
1. **Enable** the proxy toggle
|
||||
2. Enter your proxy server details in the following format:
|
||||
2. Enter your proxy details:
|
||||
```
|
||||
http://<user>:<password>@<domain or IP>:<port>
|
||||
http://<username>:<password>@<server>:<port>
|
||||
```
|
||||
|
||||
**Example:**
|
||||
```
|
||||
http://user:pass@proxy.company.com:8080
|
||||
```
|
||||
Where:
|
||||
- `<user>`: Your proxy username (if authentication is required)
|
||||
- `<password>`: Your proxy password (if authentication is required)
|
||||
- `<domain or IP>`: Your proxy server's domain name or IP address
|
||||
- `<port>`: The port number for the proxy server
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
**Ignore SSL Certificates**
|
||||
|
||||
This setting allows Jan to accept self-signed or unverified SSL certificates. This may be necessary when:
|
||||
- Working with corporate proxies using internal certificates
|
||||
- Testing in development environments
|
||||
- Connecting through specialized network security setups
|
||||
|
||||
<Callout type="info">
|
||||
Only enable this option if you trust your network environment.
|
||||
Proxy connections may slow down model downloads but don't affect local model performance.
|
||||
</Callout>
|
||||
|
||||
<br/>
|
||||
### SSL Certificate Handling
|
||||
|
||||
**Ignore SSL Certificates:** Only enable this for:
|
||||
- Corporate networks with internal certificates
|
||||
- Development/testing environments
|
||||
- Trusted network setups
|
||||
|
||||
|
||||
<br/>
|
||||
|
||||
|
||||
### Factory Reset
|
||||
Reset to Factory Settings restores Jan to its initial state by erasing all user data, including downloaded models and chat history. This action is irreversible and should only be used as a last resort when experiencing serious application issues.
|
||||
|
||||
<Callout type="warning">
|
||||
This action cannot be undone. All data will be permanently deleted.
|
||||
Only enable if you trust your network environment completely.
|
||||
</Callout>
|
||||
|
||||
Only use factory reset if:
|
||||
- The application is corrupted
|
||||
- You're experiencing persistent technical issues that other solutions haven't fixed
|
||||
- You want to completely start fresh with a clean installation
|
||||
## Emergency Options
|
||||
|
||||
To begin the process:
|
||||
1. At **Reset to Factory Settings**, click **Reset** button
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
2. In the confirmation dialog:
|
||||
- Type the word **RESET** to confirm
|
||||
- Optionally check **Keep the current app data location** to maintain the same data folder
|
||||
- Click **Reset Now**
|
||||
3. App restart is required upon confirmation
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
### Factory Reset
|
||||
|
||||
**When to use:** Only as a last resort for serious problems that other solutions can't fix.
|
||||
|
||||
**What it does:** Returns Jan to its original state - deletes everything.
|
||||
|
||||
**Steps:**
|
||||
1. Click **Reset** under "Reset to Factory Settings"
|
||||
2. Type **RESET** to confirm you understand this deletes everything
|
||||
3. Optionally keep your current data folder location
|
||||
4. Click **Reset Now**
|
||||
5. Restart Jan
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
<Callout type="error">
|
||||
**This cannot be undone.** All chat history, downloaded models, and settings will be permanently deleted.
|
||||
</Callout>
|
||||
|
||||
**Try these first:**
|
||||
- Restart Jan
|
||||
- Check the [Troubleshooting Guide](./troubleshooting)
|
||||
- Ask for help on [Discord](https://discord.gg/qSwXFx6Krr)
|
||||
|
||||
## Quick Tips
|
||||
|
||||
**For new users:**
|
||||
- Start with default settings
|
||||
- Try a few different models to find what works best
|
||||
- Enable GPU acceleration if you have a graphics card
|
||||
|
||||
**For performance:**
|
||||
- Monitor hardware usage in real-time
|
||||
- Adjust GPU layers based on your graphics card memory
|
||||
- Use smaller models on older hardware
|
||||
|
||||
**For privacy:**
|
||||
- All data stays local by default
|
||||
- Check the data folder to see exactly what's stored
|
||||
- Analytics are opt-in only
|
||||
|
||||
@ -92,13 +92,9 @@ which can leverage the Jan LLM and related contexts like code, documentation, an
|
||||
|
||||
Simply open the Tabby homepage at [localhost:8080](http://localhost:8080) and ask your questions.
|
||||
|
||||
|
||||
|
||||
### IDE Chat Sidebar
|
||||
|
||||
After installing the Tabby extension on your preferred IDEs, you can engage in a conversation with Jan to:
|
||||
|
||||
1. Discuss your code, receive suggestions, and seek assistance.
|
||||
2. Request Jan to inline edit your code, and then review and accept the proposed changes.
|
||||
|
||||
|
||||
2. Request Jan to inline edit your code, and then review and accept the proposed changes.
|
||||
@ -1,401 +1,327 @@
|
||||
---
|
||||
title: Troubleshooting
|
||||
description: Explore solutions for common issues and optimize Jan's performance with this comprehensive troubleshooting guide.
|
||||
description: Fix common issues and optimize Jan's performance with this comprehensive guide.
|
||||
keywords:
|
||||
[
|
||||
Jan,
|
||||
Customizable Intelligence, LLM,
|
||||
local AI,
|
||||
privacy focus,
|
||||
free and open source,
|
||||
private and offline,
|
||||
conversational AI,
|
||||
no-subscription fee,
|
||||
large language models,
|
||||
troubleshooting,
|
||||
error codes,
|
||||
broken build,
|
||||
something amiss,
|
||||
unexpected token,
|
||||
undefined issue,
|
||||
permission denied,
|
||||
error fixes,
|
||||
performance issues,
|
||||
GPU problems,
|
||||
installation issues,
|
||||
common errors,
|
||||
local AI,
|
||||
technical support,
|
||||
]
|
||||
---
|
||||
|
||||
import { Tabs } from 'nextra/components'
|
||||
import { Callout, Steps } from 'nextra/components'
|
||||
|
||||
|
||||
# Troubleshooting
|
||||
|
||||
## How to Get Error Logs
|
||||
## Getting Help: Error Logs
|
||||
|
||||
Error logs are essential for troubleshooting issues and getting help from Jan team. To get error logs from Jan, follow the steps below:
|
||||
When Jan isn't working properly, error logs help identify the problem. Here's how to get them:
|
||||
|
||||
#### Through Jan Interface
|
||||
### Quick Access to Logs
|
||||
|
||||
1. Open **System Monitor** in the footer
|
||||
2. Choose **App Log**
|
||||
**In Jan Interface:**
|
||||
1. Look for **System Monitor** in the footer
|
||||
2. Click **App Log**
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
#### Through Terminal
|
||||
**Application Logs**
|
||||
**Via Terminal:**
|
||||
```bash
|
||||
# macOS/Linux
|
||||
tail -n 50 ~/Library/Application\ Support/Jan/data/logs/app.log
|
||||
```
|
||||
**Server Logs**
|
||||
```bash
|
||||
tail -n 50 ~/Library/Application\ Support/Jan/data/logs/cortex.log
|
||||
|
||||
# Windows
|
||||
type %APPDATA%\Jan\data\logs\app.log
|
||||
```
|
||||
|
||||
<Callout type="warning">
|
||||
Ensure to redact any private or sensitive information when sharing logs or error details. We retain your logs for only 24 hours.
|
||||
Remove any personal information before sharing logs. We only keep logs for 24 hours.
|
||||
</Callout>
|
||||
|
||||
## Common Issues & Solutions
|
||||
|
||||
## Broken Build
|
||||
### Jan Won't Start (Broken Installation)
|
||||
|
||||
To resolve the issue where Jan is stuck in a broken build after installation:
|
||||
If Jan gets stuck after installation or won't start properly:
|
||||
|
||||
<Tabs items={['Mac', 'Windows', 'Linux']}>
|
||||
<Tabs.Tab>
|
||||
1. **Uninstall** Jan
|
||||
<Tabs items={['macOS', 'Windows', 'Linux']}>
|
||||
<Tabs.Tab>
|
||||
|
||||
2. **Delete** Application Data, Cache, and User Data:
|
||||
**Clean Reinstall Steps:**
|
||||
|
||||
```zsh
|
||||
rm -rf ~/Library/Application\ Support/Jan
|
||||
```
|
||||
1. **Uninstall Jan** from Applications folder
|
||||
|
||||
3. If you are using a version before `0.4.2`, you need to run the following commands:
|
||||
2. **Delete all Jan data:**
|
||||
```bash
|
||||
rm -rf ~/Library/Application\ Support/Jan
|
||||
```
|
||||
|
||||
```zsh
|
||||
ps aux | grep nitro
|
||||
# Looks for processes like `nitro` and `nitro_arm_64`, and kill them one by one by process ID
|
||||
kill -9 <PID>
|
||||
```
|
||||
3. **Kill any background processes** (for versions before 0.4.2):
|
||||
```bash
|
||||
ps aux | grep nitro
|
||||
# Find process IDs and kill them:
|
||||
kill -9 <PID>
|
||||
```
|
||||
|
||||
4. **Download** the [latest version of Jan](/download)
|
||||
</Tabs.Tab>
|
||||
4. **Download fresh copy** from [jan.ai](/download)
|
||||
|
||||
<Tabs.Tab>
|
||||
1. **Uninstall** Jan, using the [Windows Control Panel](https://support.microsoft.com/en-us/windows/uninstall-or-remove-apps-and-programs-in-windows-4b55f974-2cc6-2d2b-d092-5905080eaf98)
|
||||
</Tabs.Tab>
|
||||
<Tabs.Tab>
|
||||
|
||||
2. **Delete** Application Data, Cache, and User Data:
|
||||
**Clean Reinstall Steps:**
|
||||
|
||||
```cmd
|
||||
cd C:\Users\%USERNAME%\AppData\Roaming
|
||||
rmdir /S Jan
|
||||
```
|
||||
1. **Uninstall Jan** via Control Panel
|
||||
|
||||
3. If you are using a version before `0.4.2`, you need to run the following commands:
|
||||
2. **Delete application data:**
|
||||
```cmd
|
||||
cd C:\Users\%USERNAME%\AppData\Roaming
|
||||
rmdir /S Jan
|
||||
```
|
||||
|
||||
```bash
|
||||
# Find the process ID (PID) of the nitro process by filtering the list by process name
|
||||
tasklist | findstr "nitro"
|
||||
# Once you have the PID of the process you want to terminate, run the `taskkill`
|
||||
taskkill /F /PID <PID>
|
||||
```
|
||||
3. **Kill background processes** (for versions before 0.4.2):
|
||||
```cmd
|
||||
# Find nitro processes
|
||||
tasklist | findstr "nitro"
|
||||
# Kill them by PID
|
||||
taskkill /F /PID <PID>
|
||||
```
|
||||
|
||||
4. **Download** the [latest version of Jan](/download)
|
||||
</Tabs.Tab>
|
||||
4. **Download fresh copy** from [jan.ai](/download)
|
||||
|
||||
<Tabs.Tab>
|
||||
1. **Uninstall** Jan
|
||||
</Tabs.Tab>
|
||||
<Tabs.Tab>
|
||||
|
||||
Choose the appropriate method based on how you installed Jan:
|
||||
**Clean Reinstall Steps:**
|
||||
|
||||
**For Debian/Ubuntu:**
|
||||
```
|
||||
sudo apt-get remove Jan
|
||||
```
|
||||
**For Others:** Delete the Jan `.AppImage` file from your system
|
||||
1. **Uninstall Jan:**
|
||||
```bash
|
||||
# For Debian/Ubuntu
|
||||
sudo apt-get remove jan
|
||||
|
||||
2. Delete Application Data, Cache, and User Data:
|
||||
# For AppImage - just delete the file
|
||||
```
|
||||
|
||||
```bash
|
||||
# Default dir
|
||||
~/.config/Jan
|
||||
# Custom installation directory
|
||||
$XDG_CONFIG_HOME = /home/username/custom_config/Jan
|
||||
```
|
||||
2. **Delete application data:**
|
||||
```bash
|
||||
# Default location
|
||||
rm -rf ~/.config/Jan
|
||||
|
||||
3. If you are using a version before `0.4.2`, you need to run the following commands:
|
||||
# Or custom location
|
||||
rm -rf $XDG_CONFIG_HOME/Jan
|
||||
```
|
||||
|
||||
```zsh
|
||||
ps aux | grep nitro
|
||||
# Looks for processes like `nitro` and `nitro_arm_64`, and kill them one by one by process ID
|
||||
kill -9 <PID>
|
||||
```
|
||||
3. **Kill background processes** (for versions before 0.4.2):
|
||||
```bash
|
||||
ps aux | grep nitro
|
||||
kill -9 <PID>
|
||||
```
|
||||
|
||||
4. **Download** the [latest version of Jan](/download)
|
||||
</Tabs.Tab>
|
||||
4. **Download fresh copy** from [jan.ai](/download)
|
||||
|
||||
</Tabs.Tab>
|
||||
</Tabs>
|
||||
|
||||
Following these steps, you can cleanly uninstall and reinstall Jan, ensuring a smooth and error-free experience with the latest version.
|
||||
|
||||
<Callout type="info">
|
||||
Before reinstalling Jan, ensure it's completely removed from all shared spaces if installed on multiple user accounts on your device.
|
||||
Make sure Jan is completely removed from all user accounts before reinstalling.
|
||||
</Callout>
|
||||
|
||||
## Troubleshooting NVIDIA GPU
|
||||
To resolve issues when Jan does not utilize the NVIDIA GPU on Windows and Linux systems.
|
||||
### NVIDIA GPU Not Working
|
||||
|
||||
If Jan isn't using your NVIDIA graphics card for acceleration:
|
||||
|
||||
<Steps>
|
||||
|
||||
### Step 1: Verify Hardware and System Requirements
|
||||
### Step 1: Check Your Hardware Setup
|
||||
|
||||
#### 1.1. Check GPU Detection
|
||||
First, verify that your system recognizes the NVIDIA GPU:
|
||||
**Windows:**
|
||||
- Right-click desktop → NVIDIA Control Panel
|
||||
- Or check Device Manager → Display Adapters
|
||||
**Linux:**
|
||||
```
|
||||
lspci | grep -i nvidia
|
||||
```
|
||||
#### 1.2. Install Required components
|
||||
**NVIDIA Driver:**
|
||||
1. Install [NVIDIA Driver](https://www.nvidia.com/en-us/drivers/) for your GPU (NVIDIA driver **470.63.01 or higher**).
|
||||
2. Verify installation:
|
||||
**Verify GPU Detection:**
|
||||
|
||||
```
|
||||
nvidia-smi
|
||||
```
|
||||
Expected output should show your GPU model and driver version.
|
||||
*Windows:* Right-click desktop → NVIDIA Control Panel, or check Device Manager → Display Adapters
|
||||
|
||||
**CUDA Toolkit:**
|
||||
1. Download and install [CUDA toolkit](https://developer.nvidia.com/cuda-downloads) (**CUDA 11.7 or higher**)
|
||||
2. Verify installation:
|
||||
*Linux:* Run `lspci | grep -i nvidia`
|
||||
|
||||
**Install Required Software:**
|
||||
|
||||
**NVIDIA Driver (470.63.01 or newer):**
|
||||
1. Download from [nvidia.com/drivers](https://www.nvidia.com/drivers/)
|
||||
2. Test: Run `nvidia-smi` in terminal
|
||||
|
||||
**CUDA Toolkit (11.7 or newer):**
|
||||
1. Download from [CUDA Downloads](https://developer.nvidia.com/cuda-downloads)
|
||||
2. Test: Run `nvcc --version`
|
||||
|
||||
```
|
||||
nvcc --version
|
||||
```
|
||||
**Linux Additional Requirements:**
|
||||
1. Required packages are installed:
|
||||
```
|
||||
sudo apt update
|
||||
sudo apt install gcc-11 g++-11 cpp-11
|
||||
```
|
||||
See [detailed instructions](https://gcc.gnu.org/projects/cxx-status.html#cxx17).
|
||||
```bash
|
||||
# Install required packages
|
||||
sudo apt update && sudo apt install gcc-11 g++-11 cpp-11
|
||||
|
||||
2. Set up CUDA environment:
|
||||
```
|
||||
# Set CUDA environment
|
||||
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda/lib64
|
||||
```
|
||||
See [detailed instructions](https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#post-installation-actions).
|
||||
|
||||
<Callout type="info">
|
||||
Ensure your (V)RAM is accessible; some users with virtual RAM may require additional configuration.
|
||||
</Callout>
|
||||
### Step 2: Enable GPU Acceleration in Jan
|
||||
|
||||
### Step 2: Turn on GPU acceleration
|
||||
|
||||
Jan manages GPU usage automatically:
|
||||
- Switches to GPU mode when supported
|
||||
- Automatically selects GPU with highest VRAM
|
||||
|
||||
To verify GPU acceleration is turned on:
|
||||
1. Open **Settings** > **Hardware**
|
||||
2. Verify that **GPU Acceleration** is turned on
|
||||
3. Verify your selected GPU(s) are visible in **System Monitor** from Jan's footer
|
||||
2. Turn on **GPU Acceleration**
|
||||
3. Check **System Monitor** (footer) to verify GPU is detected
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
### Step 3: Verify Configuration
|
||||
|
||||
### Step 3: GPU Settings Check
|
||||
1. Go to **Settings** > **Advanced Settings** > **Data Folder**
|
||||
2. Open `settings.json` file
|
||||
3. Check these settings:
|
||||
|
||||
1. Go to **Settings** > **General** > **Data Folder**
|
||||
2. Click on **Open Containing Folder**
|
||||
3. Open `settings.json` file
|
||||
|
||||
Example `settings.json`:
|
||||
|
||||
```
|
||||
```json
|
||||
{
|
||||
"notify": true,
|
||||
"run_mode": "gpu",
|
||||
"run_mode": "gpu", // Should be "gpu"
|
||||
"nvidia_driver": {
|
||||
"exist": true,
|
||||
"exist": true, // Should be true
|
||||
"version": "531.18"
|
||||
},
|
||||
"cuda": {
|
||||
"exist": true,
|
||||
"exist": true, // Should be true
|
||||
"version": "12"
|
||||
},
|
||||
"gpus": [
|
||||
{
|
||||
"id": "0",
|
||||
"vram": "12282"
|
||||
},
|
||||
{
|
||||
"id": "1",
|
||||
"vram": "6144"
|
||||
},
|
||||
{
|
||||
"id": "2",
|
||||
"vram": "6144"
|
||||
"vram": "12282" // Your GPU memory in MB
|
||||
}
|
||||
],
|
||||
"gpu_highest_vram": "0"
|
||||
]
|
||||
}
|
||||
```
|
||||
**Key Configuration Values:**
|
||||
- `run_mode`: Should be "gpu" for GPU acceleration
|
||||
- `nvidia_driver`: Shows driver status and version
|
||||
- `cuda`: Shows CUDA toolkit status and version
|
||||
- `gpus`: Lists available GPUs and their VRAM (in MB)
|
||||
- `gpu_highest_vram`: ID of GPU with most VRAM
|
||||
|
||||
|
||||
|
||||
### Step 4: Restart Jan
|
||||
|
||||
Restart Jan to make sure it works.
|
||||
Close and restart Jan to apply changes.
|
||||
|
||||
</Steps>
|
||||
|
||||
### Tested Configurations
|
||||
These configurations have been verified to work with Jan's GPU acceleration. You can use them as reference points for your setup.
|
||||
#### Tested Working Configurations
|
||||
|
||||
**Bare Metal Installations**
|
||||
**Desktop Systems:**
|
||||
- Windows 11 + RTX 4070Ti + CUDA 12.2 + Driver 531.18
|
||||
- Ubuntu 22.04 + RTX 4070Ti + CUDA 12.2 + Driver 545
|
||||
|
||||
Windows 11 Pro (64-bit)
|
||||
| Component | Version/Model |
|
||||
|-----------|--------------|
|
||||
| GPU | NVIDIA GeForce RTX 4070Ti |
|
||||
| CUDA | 12.2 |
|
||||
| NVIDIA Driver | 531.18 |
|
||||
| OS | Windows 11 Pro 64-bit |
|
||||
| RAM | 32GB |
|
||||
**Virtual Machines:**
|
||||
- Ubuntu on Proxmox + GTX 1660Ti + CUDA 12.1 + Driver 535
|
||||
|
||||
Ubuntu 22.04 LTS
|
||||
| Component | Version/Model |
|
||||
|-----------|--------------|
|
||||
| GPU | NVIDIA GeForce RTX 4070Ti |
|
||||
| CUDA | 12.2 |
|
||||
| NVIDIA Driver | 545 |
|
||||
| OS | Ubuntu 22.04 LTS |
|
||||
<Callout type="info">
|
||||
Desktop installations perform better than virtual machines. VMs need proper GPU passthrough setup.
|
||||
</Callout>
|
||||
|
||||
**Virtual Machine Setups**
|
||||
### "Failed to Fetch" or "Something's Amiss" Errors
|
||||
|
||||
Ubuntu on Proxmox VM
|
||||
| Component | Version/Model |
|
||||
|-----------|--------------|
|
||||
| GPU | NVIDIA GeForce GTX 1660Ti |
|
||||
| CUDA | 12.1 |
|
||||
| NVIDIA Driver | 535 |
|
||||
| OS | Ubuntu 20.04/18.04 LTS |
|
||||
| VM Type | Proxmox |
|
||||
When models won't respond or show these errors:
|
||||
|
||||
**Performance Notes**
|
||||
- Bare metal installations provide better performance
|
||||
- VM setups require proper GPU passthrough configuration
|
||||
- Some laptop GPUs may have reduced performance
|
||||
- Hybrid graphics (Optimus) may need additional configuration
|
||||
**1. Check System Requirements**
|
||||
- **RAM:** Use models under 80% of available memory
|
||||
- 8GB system: Use models under 6GB
|
||||
- 16GB system: Use models under 13GB
|
||||
- **Hardware:** Verify your system meets [minimum requirements](/docs/troubleshooting#step-1-verify-hardware-and-system-requirements)
|
||||
|
||||
## Permission Denied
|
||||
**2. Adjust Model Settings**
|
||||
- Open model settings in the chat sidebar
|
||||
- Lower the **GPU Layers (ngl)** setting
|
||||
- Start low and increase gradually
|
||||
|
||||
When running Jan, you might encounter the following error message:
|
||||
|
||||
```
|
||||
Uncaught (in promise) Error: Error invoking layout-480796bff433a3a3.js:538 remote method 'installExtension':
|
||||
Error Package /Applications/Jan.app/Contents/Resources/app.asar.unpacked/pre-install/janhq-assistant-extension-1.0.0.tgz does not contain a valid manifest:
|
||||
Error EACCES: permission denied, mkdtemp '/Users/username/.npm/_cacache/tmp/ueCMn4'
|
||||
```
|
||||
|
||||
Permission problems mainly cause this error during installation. To resolve this issue, follow these steps:
|
||||
|
||||
1. Open your **Terminal**
|
||||
|
||||
2. Execute the following command to change ownership of the `~/.npm` directory to the current user:
|
||||
**3. Check Port Conflicts**
|
||||
If logs show "Bind address failed":
|
||||
|
||||
```bash
|
||||
sudo chown -R $(whoami) ~/.npm
|
||||
```
|
||||
|
||||
This command ensures that the necessary permissions are granted for Jan's installation.
|
||||
|
||||
|
||||
## "Failed to fetch" or "Something's Amiss" errors
|
||||
|
||||
When you start a chat with a model and encounter a **Failed to Fetch** or **Something's Amiss** error, here are some possible solutions to resolve it:
|
||||
|
||||
**1. Check System & Hardware Requirements**
|
||||
- Hardware dependencies: Ensure your device meets all [hardware requirements](docs/troubleshooting#step-1-verify-hardware-and-system-requirements)
|
||||
- OS: Ensure your operating system meets the minimum requirements ([Mac](/docs/desktop/mac#minimum-requirements), [Windows](/docs/desktop/windows#compatibility), [Linux](docs/desktop/linux#compatibility))
|
||||
- RAM: Choose models that use less than 80% of your available RAM
|
||||
- For 8GB systems: Use models under 6GB
|
||||
- For 16GB systems: Use models under 13GB
|
||||
|
||||
**2. Check Model Parameters**
|
||||
- In **Engine Settings** in right sidebar, check your `ngl` ([number of GPU layers](/docs/models/model-parameters#engine-parameters)) setting to see if it's too high
|
||||
- Start with a lower NGL value and increase gradually based on your GPU memory
|
||||
|
||||
**3. Port Conflicts**
|
||||
|
||||
If you check your [app logs](/docs/troubleshooting#how-to-get-error-logs) & see "Bind address failed at 127.0.0.1:39291", check port availability:
|
||||
```
|
||||
# Mac
|
||||
netstat -an | grep 39291
|
||||
# Check if ports are in use
|
||||
# macOS/Linux
|
||||
netstat -an | grep 1337
|
||||
|
||||
# Windows
|
||||
netstat -ano | find "39291"
|
||||
tasklist /fi "PID eq 39291"
|
||||
|
||||
# Linux
|
||||
netstat -anpe | grep "39291"
|
||||
netstat -ano | find "1337"
|
||||
```
|
||||
<Callout type="info">
|
||||
`Netstat` displays the contents of various network-related data structures for active connections.
|
||||
</Callout>
|
||||
|
||||
Default Jan ports:
|
||||
- Jan and Cortex API Server: `1337`
|
||||
- Jan Documentation: `3001`
|
||||
**Default Jan ports:**
|
||||
- API Server: `1337`
|
||||
- Documentation: `3001`
|
||||
|
||||
**4. Factory Reset**
|
||||
|
||||
A factory reset can resolve persistent issues by returning Jan to its original state. This will remove all custom settings, downloaded models, and chat history.
|
||||
1. Go to **Settings** > **Advanced Settings**
|
||||
2. At **Reset To Factory Settings**, click **Reset**
|
||||
**4. Try Factory Reset**
|
||||
1. **Settings** > **Advanced Settings**
|
||||
2. Click **Reset** under "Reset To Factory Settings"
|
||||
|
||||
<Callout type="warning">
|
||||
This will delete all chat history, models, and settings.
|
||||
This deletes all chat history, models, and settings.
|
||||
</Callout>
|
||||
|
||||
**5. Try a clean installation**
|
||||
- Uninstall Jan & clean Jan data folders ([Mac](/docs/desktop/mac#uninstall-jan), [Windows](/docs/desktop/windows#uninstall-jan), [Linux](docs/desktop/linux#uninstall-jan))
|
||||
- Install the latest [stable release](/download)
|
||||
**5. Clean Reinstall**
|
||||
If problems persist, do a complete clean installation (see "Jan Won't Start" section above).
|
||||
|
||||
<Callout type="warning">
|
||||
This will delete all your Jan data.
|
||||
</Callout>
|
||||
### Permission Denied Errors
|
||||
|
||||
## OpenAI Unexpected Token Issue
|
||||
The "Unexpected token" error usually relates to OpenAI API authentication or regional restrictions.
|
||||
If you see permission errors during installation:
|
||||
|
||||
**Step 1: API Key Sepup**
|
||||
1. Get a valid API key from [OpenAI's developer platform](https://platform.openai.com/)
|
||||
2. Ensure the key has sufficient credits & appropriate permissions
|
||||
```bash
|
||||
# Fix npm permissions (macOS/Linux)
|
||||
sudo chown -R $(whoami) ~/.npm
|
||||
|
||||
**Step 2: Regional Access**
|
||||
1. If you're in a region with restricted access, use a VPN service from a supported region
|
||||
2. Verify your network can reach OpenAI's API endpoints
|
||||
# Windows - run as administrator
|
||||
```
|
||||
|
||||
### OpenAI API Issues ("Unexpected Token")
|
||||
|
||||
## Need Further Support?
|
||||
If you can't find what you need in our troubleshooting guide, feel free reach out to us for extra help:
|
||||
- **Copy** your [app logs](/docs/troubleshooting#how-to-get-error-logs)
|
||||
- Go to our [Discord](https://discord.com/invite/FTk2MvZwJH) & send it to **#🆘|jan-help** channel for further support.
|
||||
For OpenAI connection problems:
|
||||
|
||||
**1. Verify API Key**
|
||||
- Get valid key from [OpenAI Platform](https://platform.openai.com/)
|
||||
- Ensure sufficient credits and permissions
|
||||
|
||||
**2. Check Regional Access**
|
||||
- Some regions have API restrictions
|
||||
- Try using a VPN from a supported region
|
||||
- Test network connectivity to OpenAI endpoints
|
||||
|
||||
### Performance Issues
|
||||
|
||||
**Models Running Slowly:**
|
||||
- Enable GPU acceleration (see NVIDIA section)
|
||||
- Use appropriate model size for your hardware
|
||||
- Close other memory-intensive applications
|
||||
- Check Task Manager/Activity Monitor for resource usage
|
||||
|
||||
**High Memory Usage:**
|
||||
- Switch to smaller model variants
|
||||
- Reduce context length in model settings
|
||||
- Enable model offloading in engine settings
|
||||
|
||||
**Frequent Crashes:**
|
||||
- Update graphics drivers
|
||||
- Check system temperature
|
||||
- Reduce GPU layers if using GPU acceleration
|
||||
- Verify adequate power supply (desktop systems)
|
||||
|
||||
## Need More Help?
|
||||
|
||||
If these solutions don't work:
|
||||
|
||||
**1. Gather Information:**
|
||||
- Copy your error logs (see top of this page)
|
||||
- Note your system specifications
|
||||
- Describe what you were trying to do when the problem occurred
|
||||
|
||||
**2. Get Community Support:**
|
||||
- Join our [Discord](https://discord.com/invite/FTk2MvZwJH)
|
||||
- Post in the **#🆘|jan-help** channel
|
||||
- Include your logs and system info
|
||||
|
||||
**3. Check Resources:**
|
||||
- [System requirements](/docs/troubleshooting#step-1-verify-hardware-and-system-requirements)
|
||||
- [Model compatibility guides](/docs/manage-models)
|
||||
- [Hardware setup guides](/docs/desktop/)
|
||||
|
||||
<Callout type="info">
|
||||
Check the logs to ensure the information is what you intend to send. We retain your logs for only **24 hours**, so report any issues promptly.
|
||||
When sharing logs, remove personal information first. We only keep logs for 24 hours, so report issues promptly.
|
||||
</Callout>
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user