docs: add ollama integration

This commit is contained in:
hieu-jan 2024-02-19 17:31:46 +07:00
parent b9f5080f19
commit ac62f90653
4 changed files with 90 additions and 3 deletions

View File

@ -18,7 +18,7 @@ keywords:
## Quick Introduction
With [LM Studio](https://lmstudio.ai/), you can discover, download, and run local Large Language Models (LLMs). In this guide, we will show you how to integrate and use your current models on LM Studio with Jan using 2 methods. The first method is integrating LM Studio server with Jan UI. The second method is migrating your downloaded model from LM Studio to Jan. We will use the model [Phi 2 - GGUF](https://huggingface.co/TheBloke/phi-2-GGUF) on Hugging Face as an example.
With [LM Studio](https://lmstudio.ai/), you can discover, download, and run local Large Language Models (LLMs). In this guide, we will show you how to integrate and use your current models on LM Studio with Jan using 2 methods. The first method is integrating LM Studio server with Jan UI. The second method is migrating your downloaded model from LM Studio to Jan. We will use the [Phi 2 - GGUF](https://huggingface.co/TheBloke/phi-2-GGUF) model on Hugging Face as an example.
## Steps to Integrate LM Studio server with Jan UI
@ -90,7 +90,7 @@ Restart Jan and navigate to the Hub. Locate your model and click the Use button.
![LM Studio Integration Demo](assets/05-lmstudio-integration-demo.gif)
## Steps to Migrate Your Downloaded Model from LM Studio to Jan
## Steps to Migrate Your Downloaded Model from LM Studio to Jan (Version 0.4.6 and older)
### 1. Migrate Your Downloaded Model

View File

@ -0,0 +1,87 @@
---
title: Integrate Ollama with Jan
slug: /guides/integrations/ollama
description: Guide to integrate Ollama with Jan
keywords:
[
Jan AI,
Jan,
ChatGPT alternative,
local AI,
private AI,
conversational AI,
no-subscription fee,
large language model,
Ollama integration,
]
---
## Quick Introduction
With [Ollama](https://ollama.com/), you can run large language models locally. In this guide, we will show you how to integrate and use your current models on Ollama with Jan using 2 methods. The first method is integrating Ollama server with Jan UI. The second method is migrating your downloaded model from Ollama to Jan. We will use the [llama2](https://ollama.com/library/llama2) model as an example.
## Steps to Integrate Ollama server with Jan UI
### 1. Start the Ollama server
Firstly, you can select the model you want to use from the [Ollama library](https://ollama.com/library). Then, run your model by using the following command:
```bash
ollama run <model-name>
```
According to the [Ollama documentation on OpenAI compatibility](https://github.com/ollama/ollama/blob/main/docs/openai.md), you can use the `http://localhost:11434/v1/chat/completions` endpoint to interact with the Ollama server. Thus, modify the `openai.json` file in the `~/jan/engines` folder to include the full URL of the Ollama server.
```json title="~/jan/engines/openai.json"
{
"full_url": "http://localhost:11434/v1/chat/completions"
}
```
### 2. Modify a Model JSON
Navigate to the `~/jan/models` folder. Create a folder named `<ollam-modelname>`, for example, `lmstudio-phi-2` and create a `model.json` file inside the folder including the following configurations:
- Ensure the filename must be `model.json`.
- Ensure the `id` property is set to the model name as Ollama model name.
- Ensure the `format` property is set to `api`.
- Ensure the `engine` property is set to `openai`.
- Ensure the `state` property is set to `ready`.
```json title="~/jan/models/llama2/model.json"
{
"sources": [
{
"filename": "llama2",
"url": "https://ollama.com/library/llama2"
}
],
// highlight-next-line
"id": "llama2",
"object": "model",
"name": "Ollama - Llama2",
"version": "1.0",
"description": "Llama 2 is a collection of foundation language models ranging from 7B to 70B parameters.",
// highlight-next-line
"format": "api",
"settings": {},
"parameters": {},
"metadata": {
"author": "Meta",
"tags": ["General", "Big Context Length"]
},
// highlight-next-line
"engine": "openai"
}
```
### 3. Start the Model
Restart Jan and navigate to the Hub. Locate your model and click the Use button.
![Ollama Model](assets/06-ollama-run.png)
### 4. Try Out the Integration of Jan and Ollama
![Ollama Integration Demo](assets/06-ollama-integration-demo.gif)

Binary file not shown.

After

Width:  |  Height:  |  Size: 5.0 MiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 1.2 MiB