README, troubleshooting, and settings updated
This commit is contained in:
Ramon Perez
parent
af428a58b9
commit
f74024db66
320
README.md
320
README.md
@ -14,246 +14,174 @@
|
||||
<p align="center">
|
||||
<a href="https://jan.ai/docs/quickstart">Getting Started</a>
|
||||
- <a href="https://jan.ai/docs">Docs</a>
|
||||
- <a href="https://github.com/menloresearch/jan/releases">Changelog</a>
|
||||
- <a href="https://jan.ai/changelog">Changelog</a>
|
||||
- <a href="https://github.com/menloresearch/jan/issues">Bug reports</a>
|
||||
- <a href="https://discord.gg/AsJ8krTT3N">Discord</a>
|
||||
</p>
|
||||
|
||||
<p align="center">
|
||||
⚠️ <b> Jan is currently in Development</b>: Expect breaking changes and bugs!
|
||||
</p>
|
||||
Jan is your AI Assistant that runs entirely offline on your desktop. Because in a world where everything is shared, sometimes you just want to keep your conversations between you and your computer—call us old-fashioned.
|
||||
|
||||
Jan is a ChatGPT-alternative that runs 100% offline on your device. Our goal is to make it easy for a layperson to download and run LLMs and use AI with **full control** and **privacy**.
|
||||
**⚠️ Jan is in active development.**
|
||||
|
||||
Jan is powered by [Cortex](https://github.com/menloresearch/cortex.cpp), our embeddable local AI engine that runs on any hardware.
|
||||
From PCs to multi-GPU clusters, Jan & Cortex supports universal architectures:
|
||||
## Installation
|
||||
|
||||
- [x] NVIDIA GPUs (fast)
|
||||
- [x] Apple M-series (fast)
|
||||
- [x] Apple Intel
|
||||
- [x] Linux Debian
|
||||
- [x] Windows x64
|
||||
|
||||
#### Features:
|
||||
|
||||
- [Model Library](https://jan.ai/docs/models/manage-models#add-models) with popular LLMs like Llama, Gemma, Mistral, or Qwen
|
||||
- Connect to [Remote AI APIs](https://jan.ai/docs/remote-models/openai) like Groq and OpenRouter
|
||||
- Local API Server with OpenAI-equivalent API
|
||||
- [Extensions](https://jan.ai/docs/extensions) for customizing Jan
|
||||
|
||||
## Download
|
||||
Because clicking a button is still the easiest way to get started:
|
||||
|
||||
<table>
|
||||
<tr style="text-align:center">
|
||||
<td style="text-align:center"><b>Version Type</b></td>
|
||||
<td style="text-align:center"><b>Windows</b></td>
|
||||
<td style="text-align:center"><b>MacOS Universal</b></td>
|
||||
<td colspan="2" style="text-align:center"><b>Linux</b></td>
|
||||
<tr>
|
||||
<td><b>Platform</b></td>
|
||||
<td><b>Stable</b></td>
|
||||
<td><b>Beta</b></td>
|
||||
<td><b>Nightly</b></td>
|
||||
</tr>
|
||||
<tr style="text-align:center">
|
||||
<td style="text-align:center"><b>Stable (Recommended)</b></td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/latest/win-x64'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/windows.png' style="height:14px; width: 14px" />
|
||||
<b>jan.exe</b>
|
||||
</a>
|
||||
</td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/latest/mac-universal'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/mac.png' style="height:15px; width: 15px" />
|
||||
<b>jan.dmg</b>
|
||||
</a>
|
||||
</td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/latest/linux-amd64-deb'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/linux.png' style="height:14px; width: 14px" />
|
||||
<b>jan.deb</b>
|
||||
</a>
|
||||
</td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/latest/linux-amd64-appimage'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/linux.png' style="height:14px; width: 14px" />
|
||||
<b>jan.AppImage</b>
|
||||
</a>
|
||||
</td>
|
||||
<tr>
|
||||
<td><b>Windows</b></td>
|
||||
<td><a href='https://app.jan.ai/download/latest/win-x64'>jan.exe</a></td>
|
||||
<td><a href='https://app.jan.ai/download/beta/win-x64'>jan.exe</a></td>
|
||||
<td><a href='https://app.jan.ai/download/nightly/win-x64'>jan.exe</a></td>
|
||||
</tr>
|
||||
<tr style="text-align:center">
|
||||
<td style="text-align:center"><b>Beta (Preview)</b></td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/beta/win-x64'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/windows.png' style="height:14px; width: 14px" />
|
||||
<b>jan.exe</b>
|
||||
</a>
|
||||
</td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/beta/mac-universal'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/mac.png' style="height:15px; width: 15px" />
|
||||
<b>jan.dmg</b>
|
||||
</a>
|
||||
</td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/beta/linux-amd64-deb'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/linux.png' style="height:14px; width: 14px" />
|
||||
<b>jan.deb</b>
|
||||
</a>
|
||||
</td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/beta/linux-amd64-appimage'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/linux.png' style="height:14px; width: 14px" />
|
||||
<b>jan.AppImage</b>
|
||||
</a>
|
||||
</td>
|
||||
<tr>
|
||||
<td><b>macOS</b></td>
|
||||
<td><a href='https://app.jan.ai/download/latest/mac-universal'>jan.dmg</a></td>
|
||||
<td><a href='https://app.jan.ai/download/beta/mac-universal'>jan.dmg</a></td>
|
||||
<td><a href='https://app.jan.ai/download/nightly/mac-universal'>jan.dmg</a></td>
|
||||
</tr>
|
||||
<tr style="text-align:center">
|
||||
<td style="text-align:center"><b>Nightly Build (Experimental)</b></td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/nightly/win-x64'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/windows.png' style="height:14px; width: 14px" />
|
||||
<b>jan.exe</b>
|
||||
</a>
|
||||
</td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/nightly/mac-universal'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/mac.png' style="height:15px; width: 15px" />
|
||||
<b>jan.dmg</b>
|
||||
</a>
|
||||
</td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/nightly/linux-amd64-deb'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/linux.png' style="height:14px; width: 14px" />
|
||||
<b>jan.deb</b>
|
||||
</a>
|
||||
</td>
|
||||
<td style="text-align:center">
|
||||
<a href='https://app.jan.ai/download/nightly/linux-amd64-appimage'>
|
||||
<img src='https://github.com/menloresearch/jan/blob/dev/docs/static/img/linux.png' style="height:14px; width: 14px" />
|
||||
<b>jan.AppImage</b>
|
||||
</a>
|
||||
</td>
|
||||
<tr>
|
||||
<td><b>Linux (deb)</b></td>
|
||||
<td><a href='https://app.jan.ai/download/latest/linux-amd64-deb'>jan.deb</a></td>
|
||||
<td><a href='https://app.jan.ai/download/beta/linux-amd64-deb'>jan.deb</a></td>
|
||||
<td><a href='https://app.jan.ai/download/nightly/linux-amd64-deb'>jan.deb</a></td>
|
||||
</tr>
|
||||
<tr>
|
||||
<td><b>Linux (AppImage)</b></td>
|
||||
<td><a href='https://app.jan.ai/download/latest/linux-amd64-appimage'>jan.AppImage</a></td>
|
||||
<td><a href='https://app.jan.ai/download/beta/linux-amd64-appimage'>jan.AppImage</a></td>
|
||||
<td><a href='https://app.jan.ai/download/nightly/linux-amd64-appimage'>jan.AppImage</a></td>
|
||||
</tr>
|
||||
</table>
|
||||
|
||||
Download the latest version of Jan at https://jan.ai/ or visit the [GitHub Releases](https://github.com/menloresearch/jan/releases) to download any previous release.
|
||||
Download from [jan.ai](https://jan.ai/) or [GitHub Releases](https://github.com/menloresearch/jan/releases).
|
||||
|
||||
## Demo
|
||||
|
||||
https://github.com/user-attachments/assets/c3592fa2-c504-4d9d-a885-7e00122a50f3
|
||||
<video width="100%" controls>
|
||||
<source src="./docs/public/assets/videos/enable-tool-call-for-models.mp4" type="video/mp4">
|
||||
Your browser does not support the video tag.
|
||||
</video>
|
||||
|
||||
_Real-time Video: Jan v0.5.7 on a Mac M2, 16GB Sonoma 14.2_
|
||||
## Features
|
||||
|
||||
## Quicklinks
|
||||
- **Local AI Models**: Download and run LLMs (Llama, Gemma, Qwen, etc.) from HuggingFace
|
||||
- **Cloud Integration**: Connect to OpenAI, Anthropic, Mistral, Groq, and others
|
||||
- **Custom Assistants**: Create specialized AI assistants for your tasks
|
||||
- **OpenAI-Compatible API**: Local server at `localhost:1337` for other applications
|
||||
- **Model Context Protocol**: MCP integration for enhanced capabilities
|
||||
- **Privacy First**: Everything runs locally when you want it to
|
||||
|
||||
### Jan
|
||||
## Build from Source
|
||||
|
||||
- [Jan Website](https://jan.ai/)
|
||||
- [Jan GitHub](https://github.com/menloresearch/jan)
|
||||
- [Documentation](https://jan.ai/docs)
|
||||
- [Jan Changelog](https://jan.ai/changelog)
|
||||
- [Jan Blog](https://jan.ai/blog)
|
||||
For those who enjoy the scenic route:
|
||||
|
||||
### Cortex.cpp
|
||||
### Prerequisites
|
||||
|
||||
Jan is powered by **Cortex.cpp**. It is a C++ command-line interface (CLI) designed as an alternative to [Ollama](https://ollama.com/). By default, it runs on the llama.cpp engine but also supports other engines, including ONNX and TensorRT-LLM, making it a multi-engine platform.
|
||||
- Node.js ≥ 20.0.0
|
||||
- Yarn ≥ 1.22.0
|
||||
- Make ≥ 3.81
|
||||
- Rust (for Tauri)
|
||||
|
||||
- [Cortex Website](https://cortex.so/)
|
||||
- [Cortex GitHub](https://github.com/menloresearch/cortex.cpp)
|
||||
- [Documentation](https://cortex.so/docs/)
|
||||
- [Models Library](https://cortex.so/models)
|
||||
- API Reference: _Under development_
|
||||
### Quick Start
|
||||
|
||||
## Requirements for running Jan
|
||||
```bash
|
||||
git clone https://github.com/menloresearch/jan
|
||||
cd jan
|
||||
make dev
|
||||
```
|
||||
|
||||
- **MacOS**: 13 or higher
|
||||
- **Windows**:
|
||||
- Windows 10 or higher
|
||||
- To enable GPU support:
|
||||
- Nvidia GPU with CUDA Toolkit 11.7 or higher
|
||||
- Nvidia driver 470.63.01 or higher
|
||||
- **Linux**:
|
||||
- glibc 2.27 or higher (check with `ldd --version`)
|
||||
- gcc 11, g++ 11, cpp 11 or higher, refer to this [link](https://jan.ai/guides/troubleshooting/gpu-not-used/#specific-requirements-for-linux) for more information
|
||||
- To enable GPU support:
|
||||
- Nvidia GPU with CUDA Toolkit 11.7 or higher
|
||||
- Nvidia driver 470.63.01 or higher
|
||||
This handles everything: installs dependencies, builds core components, and launches the app.
|
||||
|
||||
### Alternative Commands
|
||||
|
||||
If you prefer the verbose approach:
|
||||
|
||||
```bash
|
||||
# Setup and development
|
||||
yarn install
|
||||
yarn build:core
|
||||
yarn build:extensions
|
||||
yarn dev
|
||||
|
||||
# Production build
|
||||
yarn build
|
||||
|
||||
# Clean slate (when things inevitably break)
|
||||
make clean
|
||||
```
|
||||
|
||||
### Available Make Targets
|
||||
|
||||
- `make dev` - Full development setup and launch (recommended)
|
||||
- `make dev-tauri` - Tauri development (deprecated, use `make dev`)
|
||||
- `make build` - Production build
|
||||
- `make install-and-build` - Install dependencies and build core/extensions
|
||||
- `make test` - Run tests and linting
|
||||
- `make lint` - Check your code doesn't offend the linters
|
||||
- `make clean` - Nuclear option: delete everything and start fresh
|
||||
|
||||
## System Requirements
|
||||
|
||||
**Minimum specs for a decent experience:**
|
||||
|
||||
- **macOS**: 13.6+ (8GB RAM for 3B models, 16GB for 7B, 32GB for 13B)
|
||||
- **Windows**: 10+ with GPU support for NVIDIA/AMD/Intel Arc
|
||||
- **Linux**: Most distributions work, GPU acceleration available
|
||||
|
||||
For detailed compatibility, check our [installation guides](https://jan.ai/docs/installation).
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
As Jan is in development mode, you might get stuck on a some common issues:
|
||||
When things go sideways (they will):
|
||||
|
||||
- [Troubleshooting a broken build](https://jan.ai/docs/troubleshooting#broken-build)
|
||||
- [Troubleshooting NVIDIA GPU](https://jan.ai/docs/troubleshooting#troubleshooting-nvidia-gpu)
|
||||
- [Troubleshooting Something's Amiss](https://jan.ai/docs/troubleshooting#somethings-amiss)
|
||||
1. Check our [troubleshooting docs](https://jan.ai/docs/troubleshooting)
|
||||
2. Copy your error logs and system specs
|
||||
3. Ask for help in our [Discord](https://discord.gg/FTk2MvZwJH) `#🆘|jan-help` channel
|
||||
|
||||
If you can't find what you need in our troubleshooting guide, feel free reach out to us for extra help:
|
||||
|
||||
1. Copy your [error logs & device specifications](https://jan.ai/docs/troubleshooting#how-to-get-error-logs).
|
||||
2. Go to our [Discord](https://discord.com/invite/FTk2MvZwJH) & send it to **#🆘|get-help** channel for further support.
|
||||
|
||||
_Check the logs to ensure the information is what you intend to send. Note that we retain your logs for only 24 hours, so report any issues promptly._
|
||||
We keep logs for 24 hours, so don't procrastinate on reporting issues.
|
||||
|
||||
## Contributing
|
||||
|
||||
Contributions are welcome! Please read the [CONTRIBUTING.md](CONTRIBUTING.md) file
|
||||
Contributions welcome. See [CONTRIBUTING.md](CONTRIBUTING.md) for the full spiel.
|
||||
|
||||
### Pre-requisites
|
||||
## Links
|
||||
|
||||
- node >= 20.0.0
|
||||
- yarn >= 1.22.0
|
||||
- make >= 3.81
|
||||
|
||||
### Instructions
|
||||
|
||||
1. **Clone the repository and prepare:**
|
||||
|
||||
```bash
|
||||
git clone https://github.com/menloresearch/jan
|
||||
cd jan
|
||||
git checkout -b DESIRED_BRANCH
|
||||
```
|
||||
|
||||
2. **Run development and use Jan Desktop**
|
||||
|
||||
```bash
|
||||
make dev
|
||||
```
|
||||
|
||||
This will start the development server and open the desktop app.
|
||||
|
||||
### For production build
|
||||
|
||||
```bash
|
||||
# Do steps 1 and 2 in the previous section
|
||||
# Build the app
|
||||
make build
|
||||
```
|
||||
|
||||
This will build the app MacOS m1/m2 for production (with code signing already done) and put the result in `dist` folder.
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
Jan builds on top of other open-source projects:
|
||||
|
||||
- [llama.cpp](https://github.com/ggml-org/llama.cpp)
|
||||
- [LangChain](https://github.com/langchain-ai)
|
||||
- [TensorRT](https://github.com/NVIDIA/TensorRT)
|
||||
- [TensorRT-LLM](https://github.com/NVIDIA/TensorRT-LLM)
|
||||
- [Documentation](https://jan.ai/docs) - The manual you should read
|
||||
- [API Reference](https://jan.ai/api-reference) - For the technically inclined
|
||||
- [Changelog](https://jan.ai/changelog) - What we broke and fixed
|
||||
- [Discord](https://discord.gg/FTk2MvZwJH) - Where the community lives
|
||||
|
||||
## Contact
|
||||
|
||||
- Bugs & requests: file a GitHub ticket
|
||||
- For discussion: join our Discord [here](https://discord.gg/FTk2MvZwJH)
|
||||
- For business inquiries: email hello@jan.ai
|
||||
- For jobs: please email hr@jan.ai
|
||||
- **Bugs**: [GitHub Issues](https://github.com/menloresearch/jan/issues)
|
||||
- **Business**: hello@jan.ai
|
||||
- **Jobs**: hr@jan.ai
|
||||
- **General Discussion**: [Discord](https://discord.gg/FTk2MvZwJH)
|
||||
|
||||
## Trust & Safety
|
||||
|
||||
Beware of scams!
|
||||
**Friendly reminder**: We're not trying to scam you.
|
||||
|
||||
- We will never request your personal information.
|
||||
- Our product is completely free; no paid version exists.
|
||||
- We do not have a token or ICO.
|
||||
- We are a [bootstrapped company](https://en.wikipedia.org/wiki/Bootstrapping), and don't have any external investors (_yet_). We're open to exploring opportunities with strategic partners want to tackle [our mission](https://jan.ai/about#mission) together.
|
||||
- We won't ask for personal information
|
||||
- Jan is completely free (no premium version exists)
|
||||
- We don't have a cryptocurrency or ICO
|
||||
- We're bootstrapped and not seeking your investment (yet)
|
||||
|
||||
## License
|
||||
|
||||
Jan is free and open source, under the **Apache 2.0** license.
|
||||
Apache 2.0 - Because sharing is caring.
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
Built on the shoulders of giants:
|
||||
|
||||
- [Llama.cpp](https://github.com/ggerganov/llama.cpp)
|
||||
- [Tauri](https://tauri.app/)
|
||||
- [Scalar](https://github.com/scalar/scalar)
|
||||
|
||||
BIN
docs/public/assets/videos/enable-tool-call-for-models.mp4
Normal file
BIN
docs/public/assets/videos/enable-tool-call-for-models.mp4
Normal file
Binary file not shown.
BIN
docs/src/pages/docs/_assets/hardware.png
Normal file
BIN
docs/src/pages/docs/_assets/hardware.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 576 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 150 KiB After Width: | Height: | Size: 562 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 168 KiB After Width: | Height: | Size: 678 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 173 KiB After Width: | Height: | Size: 677 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 173 KiB After Width: | Height: | Size: 454 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 174 KiB After Width: | Height: | Size: 221 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 174 KiB After Width: | Height: | Size: 670 KiB |
Binary file not shown.
|
Before Width: | Height: | Size: 178 KiB After Width: | Height: | Size: 313 KiB |
BIN
docs/src/pages/docs/_assets/trouble-shooting-04.png
Normal file
BIN
docs/src/pages/docs/_assets/trouble-shooting-04.png
Normal file
Binary file not shown.
|
After Width: | Height: | Size: 888 KiB |
@ -15,7 +15,6 @@
|
||||
},
|
||||
"quickstart": "Quickstart",
|
||||
"remote-models": "Connect to Remote Models",
|
||||
"custom-providers": "Add Custom Providers",
|
||||
"server-examples": "Provide AI to Tools",
|
||||
"mcp": "Model Context Protocol",
|
||||
|
||||
|
||||
@ -1,263 +0,0 @@
|
||||
---
|
||||
title: Adding New Providers
|
||||
description: Learn about how to add new providers to Jan.
|
||||
[
|
||||
Jan,
|
||||
Customizable Intelligence, LLM,
|
||||
local AI,
|
||||
privacy focus,
|
||||
free and open source,
|
||||
private and offline,
|
||||
conversational AI,
|
||||
no-subscription fee,
|
||||
large language models,
|
||||
Jan Extensions,
|
||||
Extensions,
|
||||
]
|
||||
---
|
||||
|
||||
import { Callout } from 'nextra/components'
|
||||
import { Settings, EllipsisVertical } from 'lucide-react'
|
||||
|
||||
# Adding New Providers
|
||||
|
||||
Any local and remote model provider that has a compatible OpenAI API can be used in combination with Jan and both
|
||||
can be added in the same way.
|
||||
|
||||
## Local
|
||||
|
||||
Jan only allows you to run GGUF files but you can add other files via an OpenAI API compatible server like [vllm](https://vllm.ai/)
|
||||
or [SGLang]().
|
||||
|
||||
### vLLM
|
||||
|
||||
First, make sure you have [uv](https://docs.astral.sh/uv/) installed.
|
||||
|
||||
Next, create a virtual environment to install vLLM in it.
|
||||
|
||||
```sh
|
||||
uv venv .venv --python 3.12
|
||||
```
|
||||
|
||||
Activate your environment.
|
||||
```sh
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
Install `vllm` in it.
|
||||
```sh
|
||||
uv pip install vllm
|
||||
```
|
||||
|
||||
Next, start a server and download a model at the same time.
|
||||
|
||||
|
||||
|
||||
|
||||
### SGLang
|
||||
|
||||
Create a virtual environment to install vLLM in it.
|
||||
|
||||
```sh
|
||||
uv venv .venv --python 3.12
|
||||
```
|
||||
|
||||
Activate your environment.
|
||||
```sh
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
Install `vllm` in it.
|
||||
```sh
|
||||
uv pip install vllm
|
||||
```
|
||||
|
||||
Next, start a server and download a model at the same time.
|
||||
|
||||
|
||||
## Install Remote Engines
|
||||
|
||||
You can add any OpenAI API-compatible providers like Together AI, Fireworks AI, and others. Let's walk through
|
||||
some examples.
|
||||
|
||||
### Together AI
|
||||
|
||||
Create an account or login to existing [Together AI](https://together.ai)'s Dashboard.
|
||||
|
||||
|
||||
|
||||
Click on a View all models.
|
||||
|
||||
|
||||
Pick a free model like `deepseek-ai/DeepSeek-R1-Distill-Llama-70B-free`.
|
||||
|
||||
|
||||
Where it says **Run Inference**, click on `curl` and grab:
|
||||
|
||||
- `url`: `https://api.together.xyz/v1/chat/completions`
|
||||
- `model`: `deepseek-ai/DeepSeek-R1-Distill-Llama-70B-free`
|
||||
|
||||
Get your API key.
|
||||
|
||||
|
||||
Go back to Jan and open **Settings > Model Providers** and click on **Add Provider**.
|
||||
|
||||
|
||||
|
||||
Name it Together and proceed.
|
||||
|
||||
|
||||
Add your API key and change the URL.
|
||||
|
||||
|
||||
Enter the `model id` by clicking on the `+` sign where it says Models.
|
||||
|
||||
|
||||
|
||||
|
||||
1. Navigate to **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Engines**
|
||||
1. At **Remote Engine** category, click **+ Install Engine**
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
2. Fill in the following required information:
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
| Field | Description | Required |
|
||||
|-------|-------------|----------|
|
||||
| Engine Name | Name for your engine (e.g., "OpenAI", "Claude") | ✓ |
|
||||
| API URL | The base URL of the provider's API | ✓ |
|
||||
| API Key | Your authentication key to activate this engine | ✓ |
|
||||
| Model List URL | The endpoint URL to fetch available models |
|
||||
| API Key Template | Custom authorization header format | |
|
||||
| Request Format Conversion | Function to convert Jan's request format to provider's format | |
|
||||
| Response Format Conversion | Function to convert provider's response format to Jan's format | |
|
||||
|
||||
|
||||
> - The conversion functions are only needed for providers that don't follow the OpenAI API format. For OpenAI-compatible APIs, you can leave these empty.
|
||||
> - For OpenAI-compatible APIs like OpenAI, Anthropic, or Groq, you only need to fill in the required fields. Leave optional fields empty.
|
||||
|
||||
4. Click **Install**
|
||||
5. Once completed, you should see your engine in **Engines** page:
|
||||
- You can rename or uninstall your engine
|
||||
- You can navigate to its own settings page
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
### Examples
|
||||
#### OpenAI-Compatible Setup
|
||||
Here's how to set up OpenAI as a remote engine:
|
||||
|
||||
1. Engine Name: `OpenAI`
|
||||
2. API URL: `https://api.openai.com`
|
||||
3. Model List URL: `https://api.openai.com/v1/models`
|
||||
4. API Key: Your OpenAI API key
|
||||
5. Leave other fields as default
|
||||
|
||||
|
||||
#### Custom APIs Setup
|
||||
If you're integrating an API that doesn't follow OpenAI's format, you'll need to use the conversion functions.
|
||||
Let's say you have a custom API with this format:
|
||||
|
||||
```javascript
|
||||
// Custom API Request Format
|
||||
{
|
||||
"prompt": "What is AI?",
|
||||
"max_length": 100,
|
||||
"temperature": 0.7
|
||||
}
|
||||
|
||||
// Custom API Response Format
|
||||
{
|
||||
"generated_text": "AI is...",
|
||||
"tokens_used": 50,
|
||||
"status": "success"
|
||||
}
|
||||
```
|
||||
|
||||
**Here's how to set it up in Jan:**
|
||||
```
|
||||
Engine Name: Custom LLM
|
||||
API Key: your_api_key_here
|
||||
Header template: your header template here
|
||||
Transform request template: your transform request template here
|
||||
Transform response template: your transform response template here
|
||||
```
|
||||
|
||||
1. Header template
|
||||
```json
|
||||
"Authorization: Bearer {{api_key}}"
|
||||
```
|
||||
2. Transform request template:
|
||||
Convert from Jan's OpenAI-style format to your API's format
|
||||
|
||||
```json
|
||||
"chat_completions": {
|
||||
"url": "https://api.custom_endpoint.com/v1/messages",
|
||||
"template": "{
|
||||
{% for key, value in input_request %}
|
||||
{% if key == "messages" %}
|
||||
"prompt": "{{ last(input_request.messages).content }}"
|
||||
{% else if key == "max_tokens" or key == "temperature" %}
|
||||
"{{ key }}": {{ tojson(value) }}
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
}"
|
||||
}
|
||||
```
|
||||
|
||||
3. Transform response template
|
||||
Convert from your API's format back to OpenAI-style format
|
||||
|
||||
```json
|
||||
"chat_completions": {
|
||||
"template": {
|
||||
"choices": [{
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "{{ input_request.generated_text }}"
|
||||
}
|
||||
}],
|
||||
"usage": {
|
||||
"total_tokens": "{{ input_request.tokens_used }}"
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Expected Formats:**
|
||||
|
||||
1. Jan's Request Format
|
||||
|
||||
```json
|
||||
{
|
||||
"messages": [
|
||||
{"role": "user", "content": "What is AI?"}
|
||||
],
|
||||
"max_tokens": 100,
|
||||
"temperature": 0.7
|
||||
}
|
||||
```
|
||||
|
||||
2. Jan's Expected Response Format
|
||||
|
||||
```json
|
||||
{
|
||||
"choices": [{
|
||||
"message": {
|
||||
"role": "assistant",
|
||||
"content": "AI is..."
|
||||
}
|
||||
}],
|
||||
"usage": {
|
||||
"total_tokens": 50
|
||||
}
|
||||
}
|
||||
```
|
||||
@ -30,91 +30,61 @@ To access the **Settings**, click <Settings width={16} height={16} style={{displ
|
||||
|
||||
## Model Management
|
||||
|
||||
Manage your installed AI models in **Settings** > **Model Providers** or **My Models**:
|
||||
Manage your installed AI models in **Settings** > **Model Providers**:
|
||||
|
||||
### Import Models
|
||||
- **From Hugging Face:**
|
||||
- Enter a model's Hugging Face URL or ID in the search bar.
|
||||
- **Note:** Some models require a Hugging Face Access Token. Enter your token in **Settings > Model Providers > Hugging Face**.
|
||||
- Enter a model's Hugging Face ID (e.g., `org/model_name_or_id`) in the Hub's search bar.
|
||||
- **Note:** Some models require a Hugging Face Access Token. Enter your token in **Settings > Model Providers > Hugging Face Access Token**.
|
||||
- **From Local Files:**
|
||||
- Click **Import Model** and select your GGUF files.
|
||||
|
||||
### Remove Models
|
||||
- Click the three dots <EllipsisVertical width={16} height={16} style={{display:"inline"}}/> next to a model and select **Delete**.
|
||||
|
||||
- Click the trash icon next to the **Start** button and then click **Delete**.
|
||||
|
||||
### Start Models
|
||||
- Click the three dots <EllipsisVertical width={16} height={16} style={{display:"inline"}}/> next to a model and select **Start**.
|
||||
|
||||
1. Open a new chat and select the model you want to start.
|
||||
2. Click the **Start** button on the **Settings > Model Providers**
|
||||
|
||||
### Hugging Face Access Token
|
||||
To download models from Hugging Face that require authentication:
|
||||
To download models from Hugging Face that require authentication, for example, like the llama models from meta:
|
||||
1. Get your token from [Hugging Face Tokens](https://huggingface.co/docs/hub/en/security-tokens)
|
||||
2. Enter it in **Settings > Model Providers > Hugging Face**
|
||||
2. Enter it in **Settings > Model Providers > Hugging Face**.
|
||||
|
||||
## Model Settings (Gear Icon)
|
||||
|
||||
|
||||
|
||||
Click the gear icon next to a model to configure advanced settings:
|
||||
- **Context Size**: Maximum prompt context length
|
||||
- **GPU Layers**: Number of model layers to offload to GPU
|
||||
- **GPU Layers**: Number of model layers to offload to GPU. If you have an NVIDIA GPU and notice that your model won't fully load in it, you can reduce this value to load smaller parts of the model and try again.
|
||||
- **Temperature**: Controls randomness (higher = more random)
|
||||
- **Top K**: Top-K sampling
|
||||
- **Top P**: Top-P sampling
|
||||
- **Min P**: Minimum probability for token selection
|
||||
- **Repeat Last N**: Number of tokens to consider for repeat penalty
|
||||
- **Repeat Penalty**: Penalize repeating token sequences
|
||||
- **Presence Penalty**: Penalize alpha presence
|
||||
- **Top K**: Limits token selection to K most likely next tokens (smaller K = more focused responses)
|
||||
- **Top P**: Limits token selection to tokens comprising P probability mass (smaller P = more focused responses)
|
||||
- **Min P**: Sets a minimum threshold for words the model can select (higher values filter out less likely words)
|
||||
- **Repeat Last N**: Determines how many recent words the model checks to avoid repetition
|
||||
- **Repeat Penalty**: Controls how strongly the model avoids repeating phrases (higher values reduce repetition)
|
||||
- **Presence Penalty**: Discourages reusing words that already appeared in the text (helps with variety)
|
||||
|
||||
_See [Model Parameters](/docs/models/model-parameters) for detailed explanations._
|
||||
_See [Model Parameters](/docs/model-parameters) for a more detailed explanation._
|
||||
|
||||
## Model Capabilities (Edit Button)
|
||||
|
||||
Click the edit button next to a model to toggle capabilities:
|
||||
- **Vision**: Enable image input/output
|
||||
- **Tools**: Enable advanced tools (web search, file ops, code)
|
||||
- **Embeddings**: Enable embedding generation
|
||||
- **Web Search**: Allow model to search the web
|
||||
- **Reasoning**: Enable advanced reasoning features
|
||||
|
||||
|
||||
**2. Remove Models**: Use the same instructions in [Delete Local Models](/docs/models/manage-models#delete-models)
|
||||
|
||||
## Shortcuts
|
||||
|
||||
See and customize keyboard shortcuts for navigation, chat, and thread management.
|
||||
|
||||
## Hardware
|
||||
|
||||
Monitor and manage system resources:
|
||||
Monitor and manage system resource at **Settings > Hardware**s:
|
||||
- **CPU, RAM, GPU**: View usage and specs
|
||||
- **GPU Acceleration**: Enable/disable and configure GPU settings
|
||||
|
||||
## MCP Servers, Local API Server, HTTPS Proxy, Extensions
|
||||
|
||||
- **MCP Servers**: Add/edit servers for advanced integrations
|
||||
- **Local API Server**: Configure OpenAI-compatible local HTTP server
|
||||
- **HTTPS Proxy**: Set up proxy for secure connections
|
||||
- **Extensions**: Manage and configure Jan extensions
|
||||
|
||||
## Factory Reset
|
||||
|
||||
Restore Jan to its initial state (erases all data and models). Use only if necessary.
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
## Preferences
|
||||
|
||||
At **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Preferences**, you can customize how Jan looks.
|
||||
|
||||
### Appearance & Theme
|
||||
Control the visual theme of Jan's interface.
|
||||
- **Joi Light:** Clean, bright theme for daytime use
|
||||
- **Joi Dark:** Dark theme with high contrast
|
||||
- **Dark Dimmed:** Softer dark theme to reduce eye strain
|
||||
- **Night Blue:** Dark theme with blue accents
|
||||
|
||||
To change:
|
||||
1. Choose your preferred **Appearance** from the dropdown
|
||||
2. With **Joi Dark** & **Joi Light**, you can choose additional options:
|
||||
- **Solid:** Traditional opaque background
|
||||
- **Translucent:** Semi-transparent interface
|
||||
3. Changes apply immediately
|
||||
Control the visual theme of Jan's interface with any color combo you'd like. You can also control the color use in the code blocks.
|
||||
|
||||
<br/>
|
||||
|
||||
@ -122,78 +92,11 @@ To change:
|
||||
|
||||
### Spell Check
|
||||
Jan includes a built-in spell check feature to help catch typing errors in your messages.
|
||||
1. Switch the toggle on to enable spell checking, or off to disable it
|
||||
2. Changes apply immediately for all new messages you type
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
### Scrolling Bar
|
||||
Turn on to make scrolling bar visible across windows.
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
## Keyboard Shortcuts
|
||||
At **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Keyboard shortcuts**, you can see Jan's shortcuts list:
|
||||
|
||||
**1. Thread Management**
|
||||
- `⌘ N` - Create a new thread
|
||||
- `⌘ Shift Backspace` - Delete current active thread
|
||||
- `⌘ Shift C` - Clean current active thread
|
||||
|
||||
**2. Navigation**
|
||||
- `⌘ B` - Toggle left panel
|
||||
- `⌘ Shift B` - Toggle right panel
|
||||
- `⌘ ,` - Navigate to settings
|
||||
|
||||
**3. Message Input**
|
||||
- `Enter` - Send a message (in input field)
|
||||
- `Shift Enter` - Insert a new line (in input field)
|
||||
|
||||
> Note: On **Windows** and **Linux**, use `Ctrl` (Control) instead of `⌘` (Command)
|
||||
|
||||
## Hardware
|
||||
At **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Hardware**, you can monitor and manage your system resources when running local models.
|
||||
|
||||
### CPU & RAM
|
||||
|
||||
<Callout type="info">
|
||||
See detailed minimum system requirements to run local models on [Mac](/docs/desktop/mac#compatibility), [Windows](/docs/desktop/windows#compatibility) & [Linux](/docs/desktop/linux#compatibility).
|
||||
</Callout>
|
||||
|
||||
- **CPU:** important for model processing speed. With CPU only, 20-90% usage during model running is normal.
|
||||
|
||||
- **RAM:**
|
||||
- Different models require different amounts of RAM.
|
||||
- When running local models, please keep at least **4GB free**.
|
||||
- If usage is near max, try closing other applications or use smaller models.
|
||||
|
||||
|
||||
### GPU Acceleration
|
||||
|
||||
<Callout type="info">
|
||||
Ensure you have installed all required dependencies and drivers before enabling GPU acceleration. See **GPU Setup Guide** on [Windows](/docs/desktop/windows#gpu-acceleration) & [Linux](/docs/desktop/linux#gpu-acceleration) for detailed instructions.
|
||||
</Callout>
|
||||
|
||||
Turn on GPU acceleration to improve performance:
|
||||
1. Navigate to **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Local Engine** > **Llama.cpp**
|
||||
2. At **llama-cpp Backend**, select backend. For example `windows-amd64-vulkan` if you have and AMD gaphic card. For more info, see [our guide](/docs/local-engines/llama-cpp).
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
**GPU Performance Optimization**
|
||||
- Monitor VRAM usage - should not exceed 90%. Higher **VRAM** capacity typically enables better performance for larger language models.
|
||||
- Adjust `ngl` ([number of GPU Layers](/docs/models/model-parameters#engine-parameters)) in **Model** settings:
|
||||
- Higher NGL = more VRAM used with faster performance
|
||||
- Lower NGL = Less VRAM used with slower performance
|
||||
- Start with 35 layers for 8GB VRAM. Increase if you have more VRAM available & decrease if you see **out-of-memory** errors.
|
||||
|
||||
|
||||
|
||||
## Privacy
|
||||
At **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Privacy**, you can control analytics & logs in Jan:
|
||||
@ -237,27 +140,6 @@ This action cannot be undone.
|
||||
<br/>
|
||||
|
||||
|
||||
## Advanced Settings
|
||||
At **Settings** (<Settings width={16} height={16} style={{display:"inline"}}/>) > **Advanced Settings**, Jan stores settings for advanced use cases.
|
||||
### Experimental Mode
|
||||
<Callout type="warning">
|
||||
Experimental features are unstable and are recommended only for testing purposes. Please turn on with caution!
|
||||
</Callout>
|
||||
|
||||
Current experimental features:
|
||||
|
||||
| Feature | Description |
|
||||
|---------|-------------|
|
||||
| [Tools](/docs/tools/retrieval) | Advanced tooling capabilities including web search, file operations, and code interpretation |
|
||||
| Vulkan Settings | GPU acceleration using Vulkan API for improved model performance |
|
||||
| [Jan Quick Ask](/docs/settings#jan-quick-ask) | Streamlined interface for rapid AI queries and responses |
|
||||
|
||||
|
||||
To try out these beta features, turn on **Experimental Mode** setting:
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
### Jan Data Folder
|
||||
Jan stores your data locally in your own filesystem in a universal file format. See detailed [Jan Folder Structure](docs/data-folder#folder-structure).
|
||||
|
||||
@ -322,24 +204,6 @@ Only enable this option if you trust your network environment.
|
||||
|
||||
<br/>
|
||||
|
||||
### Jan Quick Ask
|
||||
Jan Quick Ask provides a faster way to interact with Jan without opening the full application window. It's designed for quick queries and responses when you need immediate assistance.
|
||||
<Callout type="warning">
|
||||
This feature is currently experimental and must be enabled through [Experimental Mode](/docs/settings#experimental-mode) in **Advanced Settings**.
|
||||
</Callout>
|
||||
|
||||
1. Once you've turned on [Experimental Mode](/docs/settings#experimental-mode), toggle on **Jan Quick Ask**
|
||||
2. App restart is required upon selection
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
3. Once enabled, you can open **Quick Ask** overlay with default shortcut: `CMD/Ctr` + `J`
|
||||
|
||||
<br/>
|
||||
|
||||
<br/>
|
||||
|
||||
### Factory Reset
|
||||
Reset to Factory Settings restores Jan to its initial state by erasing all user data, including downloaded models and chat history. This action is irreversible and should only be used as a last resort when experiencing serious application issues.
|
||||
|
||||
@ -211,8 +211,8 @@ To verify GPU acceleration is turned on:
|
||||
|
||||
### Step 3: GPU Settings Check
|
||||
|
||||
1. Go to **Settings** > **Advanced Settings** > Open **Jan Data Folder**
|
||||
2. Open **Settings** folder
|
||||
1. Go to **Settings** > **General** > **Data Folder**
|
||||
2. Click on **Open Containing Folder**
|
||||
3. Open `settings.json` file
|
||||
|
||||
Example `settings.json`:
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user