docs: add LM Studio and Ollama integrations guide & import model using absolute path (#2076)

docs: add LM Studio and Ollama integrations guide  & import model using absolute path (#2076)
This commit is contained in:
Henry 2024-02-22 09:36:47 +07:00 committed by GitHub
commit 322c834055
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
19 changed files with 366 additions and 5 deletions

View File

@ -29,6 +29,10 @@ In this section, we will show you how to import a GGUF model from [HuggingFace](
> We are fast shipping a UI to make this easier, but it's a bit manual for now. Apologies.
## Import Models Using Absolute Filepath (version 0.4.7)
Starting from version 0.4.7, Jan has introduced the capability to import models using an absolute file path. It allows you to import models from any directory on your computer. Please check the [import models using absolute filepath](../import-models-using-absolute-filepath) guide for more information.
## Manually Importing a Downloaded Model (nightly versions and v0.4.4+)
### 1. Create a Model Folder
@ -186,7 +190,6 @@ This means that you can easily reconfigure your models, export them, and share y
Edit `model.json` and include the following configurations:
- Ensure the filename must be `model.json`.
- Ensure the `id` property matches the folder name you created.
- Ensure the GGUF filename should match the `id` property exactly.
- Ensure the `source.url` property is the direct binary download link ending in `.gguf`. In HuggingFace, you can find the direct links in the `Files and versions` tab.

View File

@ -0,0 +1,84 @@
---
title: Import Models Using Absolute Filepath
slug: /guides/using-models/import-models-using-absolute-filepath
description: Guide to import model using absolute filepath in Jan.
keywords:
[
Jan AI,
Jan,
ChatGPT alternative,
local AI,
private AI,
conversational AI,
no-subscription fee,
large language model,
import-models-manually,
absolute-filepath,
]
---
In this guide, we will walk you through the process of importing a model using an absolute filepath in Jan, using our latest model, [Trinity](https://huggingface.co/janhq/trinity-v1-GGUF), as an example.
### 1. Get the Absolute Filepath of the Model
After downloading .gguf model, you can get the absolute filepath of the model file.
### 2. Configure the Model JSON
1. Navigate to the `~/jan/models` folder.
2. Create a folder named `<modelname>`, for example, `tinyllama`.
3. Create a `model.json` file inside the folder, including the following configurations:
- Ensure the `id` property matches the folder name you created.
- Ensure the `url` property is the direct binary download link ending in `.gguf`. Now, you can use the absolute filepath of the model file.
- Ensure the `engine` property is set to `nitro`.
```json
{
"sources": [
{
"filename": "tinyllama.gguf",
// highlight-next-line
"url": "<absolute-filepath-of-the-model-file>"
}
],
"id": "tinyllama-1.1b",
"object": "model",
"name": "(Absolute Path) TinyLlama Chat 1.1B Q4",
"version": "1.0",
"description": "TinyLlama is a tiny model with only 1.1B. It's a good model for less powerful computers.",
"format": "gguf",
"settings": {
"ctx_len": 4096,
"prompt_template": "<|system|>\n{system_message}<|user|>\n{prompt}<|assistant|>",
"llama_model_path": "tinyllama-1.1b-chat-v1.0.Q4_K_M.gguf"
},
"parameters": {
"temperature": 0.7,
"top_p": 0.95,
"stream": true,
"max_tokens": 2048,
"stop": [],
"frequency_penalty": 0,
"presence_penalty": 0
},
"metadata": {
"author": "TinyLlama",
"tags": ["Tiny", "Foundation Model"],
"size": 669000000
},
"engine": "nitro"
}
```
:::warning
- If you are using Windows, you need to use double backslashes in the url property, for example: `C:\\Users\\username\\filename.gguf`.
:::
### 3. Start the Model
Restart Jan and navigate to the Hub. Locate your model and click the Use button.
![Demo](assets/03-demo-absolute-filepath.gif)

View File

@ -88,7 +88,7 @@ You can find your API keys in the [OpenAI Platform](https://platform.openai.com/
Restart Jan and navigate to the Hub. Then, select your configured model and start the model.
![image-01](assets/03-openai-platform-configuration.png)
![image-01](assets/04-openai-platform-configuration.png)
## Engines with OAI Compatible Configuration
@ -159,7 +159,7 @@ Navigate to the `~/jan/models` folder. Create a folder named `mistral-ins-7b-q4`
Restart Jan and navigate to the Hub. Locate your model and click the Use button.
![image-02](assets/03-oai-compatible-configuration.png)
![image-02](assets/04-oai-compatible-configuration.png)
## Assistance and Support

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.8 MiB

View File

@ -1,6 +1,6 @@
---
title: Start Local Server
slug: /guides/using-server/server
slug: /guides/using-server/start-server
description: How to run Jan's built-in API server.
keywords:
[

View File

@ -35,7 +35,7 @@ To get started with Continue in VS Code, please follow this [guide to install Co
### 2. Enable Jan API Server
To configure the Continue to use Jan's Local Server, you need to enable Jan API Server with your preferred model, please follow this [guide to enable Jan API Server](../05-using-server/01-server.md)
To configure the Continue to use Jan's Local Server, you need to enable Jan API Server with your preferred model, please follow this [guide to enable Jan API Server](/guides/using-server/start-server).
### 3. Configure Continue to Use Jan's Local Server

View File

@ -0,0 +1,184 @@
---
title: Integrate LM Studio with Jan
slug: /guides/integrations/lmstudio
description: Guide to integrate LM Studio with Jan
keywords:
[
Jan AI,
Jan,
ChatGPT alternative,
local AI,
private AI,
conversational AI,
no-subscription fee,
large language model,
LM Studio integration,
]
---
## Quick Introduction
With [LM Studio](https://lmstudio.ai/), you can discover, download, and run local Large Language Models (LLMs). In this guide, we will show you how to integrate and use your current models on LM Studio with Jan using 2 methods. The first method is integrating LM Studio server with Jan UI. The second method is migrating your downloaded model from LM Studio to Jan. We will use the [Phi 2 - GGUF](https://huggingface.co/TheBloke/phi-2-GGUF) model on Hugging Face as an example.
## Steps to Integrate LM Studio Server with Jan UI
### 1. Start the LM Studio Server
1. Navigate to the `Local Inference Server` on the LM Studio application.
2. Select the model you want to use.
3. Start the server after configuring the server port and options.
![LM Studio Server](assets/05-setting-lmstudio-server.gif)
<br></br>
Modify the `openai.json` file in the `~/jan/engines` folder to include the full URL of the LM Studio server.
```json title="~/jan/engines/openai.json"
{
"full_url": "http://localhost:<port>/v1/chat/completions"
}
```
:::tip
- Replace `<port>` with the port number you set in the LM Studio server. The default port is `1234`.
:::
### 2. Modify a Model JSON
Navigate to the `~/jan/models` folder. Create a folder named `<lmstudio-modelname>`, for example, `lmstudio-phi-2` and create a `model.json` file inside the folder including the following configurations:
- Set the `format` property to `api`.
- Set the `engine` property to `openai`.
- Set the `state` property to `ready`.
```json title="~/jan/models/lmstudio-phi-2/model.json"
{
"sources": [
{
"filename": "phi-2-GGUF",
"url": "https://huggingface.co/TheBloke/phi-2-GGUF"
}
],
"id": "lmstudio-phi-2",
"object": "model",
"name": "LM Studio - Phi 2 - GGUF",
"version": "1.0",
"description": "TheBloke/phi-2-GGUF",
// highlight-next-line
"format": "api",
"settings": {},
"parameters": {},
"metadata": {
"author": "Microsoft",
"tags": ["General", "Big Context Length"]
},
// highlight-start
"engine": "openai"
// highlight-end
}
```
### 3. Start the Model
1. Restart Jan and navigate to the **Hub**.
2. Locate your model and click the **Use** button.
![LM Studio Model](assets/05-lmstudio-run.png)
### 4. Try Out the Integration of Jan and LM Studio
![LM Studio Integration Demo](assets/05-lmstudio-integration-demo.gif)
## Steps to Migrate Your Downloaded Model from LM Studio to Jan (version 0.4.6 and older)
### 1. Migrate Your Downloaded Model
1. Navigate to `My Models` in the LM Studio application and reveal the model folder.
![Reveal-model-folder-lmstudio](assets/05-reveal-model-folder-lmstudio.gif)
2. Copy the model folder that you want to migrate to `~/jan/models` folder.
3. Ensure the folder name property is the same as the model name of `.gguf` filename by changing the folder name if necessary. For example, in this case, we changed foldername from `TheBloke` to `phi-2.Q4_K_S`.
### 2. Start the Model
1. Restart Jan and navigate to the **Hub**. Jan will automatically detect the model and display it in the **Hub**.
2. Locate your model and click the **Use** button to try the migrating model.
![Demo](assets/05-demo-migrating-model.gif)
## Steps to Pointing to the Downloaded Model of LM Studio from Jan (version 0.4.7+)
Starting from version 0.4.7, Jan supports importing models using an absolute filepath, so you can directly use the model from the LM Studio folder.
### 1. Reveal the Model Absolute Path
Navigate to `My Models` in the LM Studio application and reveal the model folder. Then, you can get the absolute path of your model.
![Reveal-model-folder-lmstudio](assets/05-reveal-model-folder-lmstudio.gif)
### 2. Modify a Model JSON
Navigate to the `~/jan/models` folder. Create a folder named `<modelname>`, for example, `phi-2.Q4_K_S` and create a `model.json` file inside the folder including the following configurations:
- Ensure the `id` property matches the folder name you created.
- Ensure the `url` property is the direct binary download link ending in `.gguf`. Now, you can use the absolute filepath of the model file. In this example, the absolute filepath is `/Users/<username>/.cache/lm-studio/models/TheBloke/phi-2-GGUF/phi-2.Q4_K_S.gguf`.
- Ensure the `engine` property is set to `nitro`.
```json
{
"object": "model",
"version": 1,
"format": "gguf",
"sources": [
{
"filename": "phi-2.Q4_K_S.gguf",
"url": "<absolute-path-of-model-file>"
}
],
"id": "phi-2.Q4_K_S",
"name": "phi-2.Q4_K_S",
"created": 1708308111506,
"description": "phi-2.Q4_K_S - user self import model",
"settings": {
"ctx_len": 4096,
"embedding": false,
"prompt_template": "{system_message}\n### Instruction: {prompt}\n### Response:",
"llama_model_path": "phi-2.Q4_K_S.gguf"
},
"parameters": {
"temperature": 0.7,
"top_p": 0.95,
"stream": true,
"max_tokens": 2048,
"stop": ["<endofstring>"],
"frequency_penalty": 0,
"presence_penalty": 0
},
"metadata": {
"size": 1615568736,
"author": "User",
"tags": []
},
"engine": "nitro"
}
```
:::warning
- If you are using Windows, you need to use double backslashes in the url property, for example: `C:\\Users\\username\\filename.gguf`.
:::
### 3. Start the Model
1. Restart Jan and navigate to the **Hub**.
2. Jan will automatically detect the model and display it in the **Hub**.
3. Locate your model and click the **Use** button to try the migrating model.
![Demo](assets/05-demo-pointing-model.gif)

View File

@ -0,0 +1,90 @@
---
title: Integrate Ollama with Jan
slug: /guides/integrations/ollama
description: Guide to integrate Ollama with Jan
keywords:
[
Jan AI,
Jan,
ChatGPT alternative,
local AI,
private AI,
conversational AI,
no-subscription fee,
large language model,
Ollama integration,
]
---
## Quick Introduction
With [Ollama](https://ollama.com/), you can run large language models locally. In this guide, we will show you how to integrate and use your current models on Ollama with Jan using 2 methods. The first method is integrating Ollama server with Jan UI. The second method is migrating your downloaded model from Ollama to Jan. We will use the [llama2](https://ollama.com/library/llama2) model as an example.
## Steps to Integrate Ollama Server with Jan UI
### 1. Start the Ollama Server
1. Select the model you want to use from the [Ollama library](https://ollama.com/library).
2. Run your model by using the following command:
```bash
ollama run <model-name>
```
3. According to the [Ollama documentation on OpenAI compatibility](https://github.com/ollama/ollama/blob/main/docs/openai.md), you can use the `http://localhost:11434/v1/chat/completions` endpoint to interact with the Ollama server. Thus, modify the `openai.json` file in the `~/jan/engines` folder to include the full URL of the Ollama server.
```json title="~/jan/engines/openai.json"
{
"full_url": "http://localhost:11434/v1/chat/completions"
}
```
### 2. Modify a Model JSON
1. Navigate to the `~/jan/models` folder.
2. Create a folder named `<ollam-modelname>`, for example, `lmstudio-phi-2`.
3. Create a `model.json` file inside the folder including the following configurations:
- Set the `id` property to the model name as Ollama model name.
- Set the `format` property to `api`.
- Set the `engine` property to `openai`.
- Set the `state` property to `ready`.
```json title="~/jan/models/llama2/model.json"
{
"sources": [
{
"filename": "llama2",
"url": "https://ollama.com/library/llama2"
}
],
// highlight-next-line
"id": "llama2",
"object": "model",
"name": "Ollama - Llama2",
"version": "1.0",
"description": "Llama 2 is a collection of foundation language models ranging from 7B to 70B parameters.",
// highlight-next-line
"format": "api",
"settings": {},
"parameters": {},
"metadata": {
"author": "Meta",
"tags": ["General", "Big Context Length"]
},
// highlight-next-line
"engine": "openai"
}
```
### 3. Start the Model
1. Restart Jan and navigate to the **Hub**.
2. Locate your model and click the **Use** button.
![Ollama Model](assets/06-ollama-run.png)
### 4. Try Out the Integration of Jan and Ollama
![Ollama Integration Demo](assets/06-ollama-integration-demo.gif)

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.7 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 6.6 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 3.3 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 11 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.0 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB