diff --git a/docs/docs/quickstart/integration/assets/lmstudio.png b/docs/docs/quickstart/integration/assets/lmstudio.png new file mode 100644 index 000000000..bffd0a00d Binary files /dev/null and b/docs/docs/quickstart/integration/assets/lmstudio.png differ diff --git a/docs/docs/quickstart/integration/assets/mistral.png b/docs/docs/quickstart/integration/assets/mistral.png new file mode 100644 index 000000000..0efeaef83 Binary files /dev/null and b/docs/docs/quickstart/integration/assets/mistral.png differ diff --git a/docs/docs/quickstart/integration/assets/ollama.png b/docs/docs/quickstart/integration/assets/ollama.png new file mode 100644 index 000000000..02a3278bf Binary files /dev/null and b/docs/docs/quickstart/integration/assets/ollama.png differ diff --git a/docs/docs/quickstart/integration/lmstudio.mdx b/docs/docs/quickstart/integration/lmstudio.mdx new file mode 100644 index 000000000..d391c1ade --- /dev/null +++ b/docs/docs/quickstart/integration/lmstudio.mdx @@ -0,0 +1,196 @@ +--- +title: LM Studio +sidebar_position: 8 +description: A step-by-step guide on how to integrate Jan with LM Studio. +keywords: + [ + Jan AI, + Jan, + ChatGPT alternative, + local AI, + private AI, + conversational AI, + no-subscription fee, + large language model, + LM Studio integration, + ] +--- +import flow from './assets/lmstudio.png'; + +## How to Integrate LM Studio with Jan + +[LM Studio](https://lmstudio.ai/) enables you to explore, download, and run local Large Language Models (LLMs). You can integrate Jan with LM Studio using two methods: +1. Integrate the LM Studio server with Jan UI +2. Migrate your downloaded model from LM Studio to Jan. + +
+ LM Studio +
+ +To integrate LM Studio with Jan follow the steps below: + +:::note + +In this guide, we're going to show you how to connect Jan to [LM Studio](https://lmstudio.ai/) using the second method. We'll use the [Phi 2 - GGUF](https://huggingface.co/TheBloke/phi-2-GGUF) model from Hugging Face as our example. +::: +### Step 1: Server Setup + +1. Access the `Local Inference Server` within LM Studio. +2. Select your desired model. +3. Start the server after configuring the port and options. + +4. Update the `openai.json` file in `~/jan/engines` to include the LM Studio server's full URL. + +```json title="~/jan/engines/openai.json" +{ + "full_url": "http://localhost:/v1/chat/completions" +} +``` + +:::tip + +Replace `(port)` with your chosen port number. The default is 1234. + +::: + +### Step 2: Model Configuration + +1. Navigate to `~/jan/models`. +2. Create a folder named `lmstudio-(modelname)`, like `lmstudio-phi-2`. +3. Inside, create a `model.json` file with these options: + - Set `format` to `api`. + - Specify `engine` as `openai`. + - Set `state` to `ready`. + +```json title="~/jan/models/lmstudio-phi-2/model.json" +{ + "sources": [ + { + "filename": "phi-2-GGUF", + "url": "https://huggingface.co/TheBloke/phi-2-GGUF" + } + ], + "id": "lmstudio-phi-2", + "object": "model", + "name": "LM Studio - Phi 2 - GGUF", + "version": "1.0", + "description": "TheBloke/phi-2-GGUF", + "format": "api", + "settings": {}, + "parameters": {}, + "metadata": { + "author": "Microsoft", + "tags": ["General", "Big Context Length"] + }, + "engine": "openai" +} +``` +### Regarding `model.json` + +- In `settings`, two crucial values are: + - `ctx_len`: Defined based on the model's context size. + - `prompt_template`: Defined based on the model's trained template (e.g., ChatML, Alpaca). + - To set up the `prompt_template`: + 1. Visit [Hugging Face](https://huggingface.co/), an open-source machine learning platform. + 2. Find the current model that you're using (e.g., [Gemma 7b it](https://huggingface.co/google/gemma-7b-it)). + 3. Review the text and identify the template. +- In `parameters`, consider the following options. The fields in `parameters` are typically general and can be the same across models. An example is provided below: + +```json +"parameters":{ + "temperature": 0.7, + "top_p": 0.95, + "stream": true, + "max_tokens": 4096, + "frequency_penalty": 0, + "presence_penalty": 0 +} +``` + + +### Step 3: Starting the Model + +1. Restart Jan and proceed to the **Hub**. +2. Locate your model and click **Use** to activate it. + +## Migrating Models from LM Studio to Jan (version 0.4.6 and older) + +### Step 1: Model Migration + +1. In LM Studio, navigate to `My Models` and access your model folder. +2. Copy the model folder to `~/jan/models`. +3. Ensure the folder name matches the model name in the `.gguf` filename. Rename as necessary. + +### Step 2: Activating the Model + +1. Restart Jan and navigate to the **Hub**, where the model will be automatically detected. +2. Click **Use** to use the model. + +## Direct Access to LM Studio Models from Jan (version 0.4.7+) + +Starting from version 0.4.7, Jan enables direct import of LM Studio models using absolute file paths. + + +### Step 1: Locating the Model Path + +1. Access `My Models` in LM Studio and locate your model folder. +2. Obtain the absolute path of your model. + +### Step 2: Model Configuration + +1. Go to `~/jan/models`. +2. Create a folder named `(modelname)` (e.g., `phi-2.Q4_K_S`). +3. Inside, craft a `model.json` file: + - Set `id` to match the folder name. + - Use the direct binary download link ending in `.gguf` as the `url`. You can now use the absolute filepath of the model file. + - Set `engine` as `nitro`. + +```json +{ + "object": "model", + "version": 1, + "format": "gguf", + "sources": [ + { + "filename": "phi-2.Q4_K_S.gguf", + "url": "" + } + ], + "id": "phi-2.Q4_K_S", + "name": "phi-2.Q4_K_S", + "created": 1708308111506, + "description": "phi-2.Q4_K_S - user self import model", + "settings": { + "ctx_len": 4096, + "embedding": false, + "prompt_template": "{system_message}\n### Instruction: {prompt}\n### Response:", + "llama_model_path": "phi-2.Q4_K_S.gguf" + }, + "parameters": { + "temperature": 0.7, + "top_p": 0.95, + "stream": true, + "max_tokens": 2048, + "stop": [""], + "frequency_penalty": 0, + "presence_penalty": 0 + }, + "metadata": { + "size": 1615568736, + "author": "User", + "tags": [] + }, + "engine": "nitro" +} +``` + +:::warning + +For Windows users, ensure to include double backslashes in the URL property, such as `C:\\Users\\username\\filename.gguf`. + +::: + +### Step 3: Starting the Model + +1. Restart Jan and proceed to the **Hub**. +2. Locate your model and click **Use** to activate it. \ No newline at end of file diff --git a/docs/docs/quickstart/integration/mistral.mdx b/docs/docs/quickstart/integration/mistral.mdx new file mode 100644 index 000000000..ba9a38994 --- /dev/null +++ b/docs/docs/quickstart/integration/mistral.mdx @@ -0,0 +1,114 @@ +--- +title: Mistral AI +sidebar_position: 7 +description: A step-by-step guide on how to integrate Jan with Mistral AI. +keywords: + [ + Jan AI, + Jan, + ChatGPT alternative, + local AI, + private AI, + conversational AI, + no-subscription fee, + large language model, + Mistral integration, + ] +--- + +import flow from './assets/mistral.png'; + +## How to Integrate Mistral AI with Jan + +[Mistral AI](https://docs.mistral.ai/) provides two ways to use their Large Language Models (LLM): +1. API +2. Open-source models on Hugging Face. + +
+ Mistral +
+ +To integrate Jan with Mistral AI, follow the steps below: + +:::note +This tutorial demonstrates integrating Mistral AI with Jan using the API. +::: + +### Step 1: Configure Mistral API Key + +1. Obtain Mistral API keys from your [Mistral](https://console.mistral.ai/user/api-keys/) dashboard. +2. Insert the Mistral AI API key into `~/jan/engines/openai.json`. + +```json title="~/jan/engines/openai.json" +{ + "full_url": "https://api.mistral.ai/v1/chat/completions", + "api_key": "" +} +``` + +### Step 2: Model Configuration + +1. Navigate to `~/jan/models`. +2. Create a folder named `mistral-(modelname)` (e.g., `mistral-tiny`). +3. Inside, create a `model.json` file with these settings: + - Set `id` to the Mistral AI model ID. + - Set `format` to `api`. + - Set `engine` to `openai`. + - Set `state` to `ready`. + +```json title="~/jan/models/mistral-tiny/model.json" +{ + "sources": [ + { + "filename": "mistral-tiny", + "url": "https://mistral.ai/" + } + ], + "id": "mistral-tiny", + "object": "model", + "name": "Mistral-7B-v0.2 (Tiny Endpoint)", + "version": "1.0", + "description": "Currently powered by Mistral-7B-v0.2, a better fine-tuning of the initial Mistral-7B released, inspired by the fantastic work of the community.", + "format": "api", + "settings": {}, + "parameters": {}, + "metadata": { + "author": "Mistral AI", + "tags": ["General", "Big Context Length"] + }, + "engine": "openai" +} + +``` +### Regarding `model.json` + +- In `settings`, two crucial values are: + - `ctx_len`: Defined based on the model's context size. + - `prompt_template`: Defined based on the model's trained template (e.g., ChatML, Alpaca). + - To set up the `prompt_template`: + 1. Visit [Hugging Face](https://huggingface.co/), an open-source machine learning platform. + 2. Find the current model that you're using (e.g., [Gemma 7b it](https://huggingface.co/google/gemma-7b-it)). + 3. Review the text and identify the template. +- In `parameters`, consider the following options. The fields in `parameters` are typically general and can be the same across models. An example is provided below: + +```json +"parameters":{ + "temperature": 0.7, + "top_p": 0.95, + "stream": true, + "max_tokens": 4096, + "frequency_penalty": 0, + "presence_penalty": 0 +} +``` + +:::note + +Mistral AI offers various endpoints. Refer to their [endpoint documentation](https://docs.mistral.ai/platform/endpoints/) to select the one that fits your requirements. Here, we use the `mistral-tiny` model as an example. + +::: + +### Step 3: Start the Model + +1. Restart Jan and navigate to the **Hub**. +2. Locate your model and click the **Use** button. \ No newline at end of file diff --git a/docs/docs/quickstart/integration/ollama.mdx b/docs/docs/quickstart/integration/ollama.mdx new file mode 100644 index 000000000..91d1db5ea --- /dev/null +++ b/docs/docs/quickstart/integration/ollama.mdx @@ -0,0 +1,112 @@ +--- +title: Ollama +sidebar_position: 9 +description: A step-by-step guide on how to integrate Jan with Ollama. +keywords: + [ + Jan AI, + Jan, + ChatGPT alternative, + local AI, + private AI, + conversational AI, + no-subscription fee, + large language model, + Ollama integration, + ] +--- + +import flow from './assets/ollama.png'; + +## How to Integrate Ollama with Jan + +Ollama provides you with largen language that you can run locally. There are two methods to integrate Ollama with Jan: +1. Integrate Ollama server with Jan. +2. Migrate the downloaded model from Ollama to Jan. + +
+ Ollama +
+ +To integrate Ollama with Jan, follow the steps below: + +:::note +In this tutorial, we'll show how to integrate Ollama with Jan using the first method. We will use the [llama2](https://ollama.com/library/llama2) model as an example. +::: + +### Step 1: Start the Ollama Server + +1. Choose your model from the [Ollama library](https://ollama.com/library). +2. Run your model with this command: + +```sh +ollama run +``` + +3. According to the [Ollama documentation on OpenAI compatibility](https://github.com/ollama/ollama/blob/main/docs/openai.md), you can connect to the Ollama server using the web address `http://localhost:11434/v1/chat/completions`. To do this, change the `openai.json` file in the `~/jan/engines` folder to add the Ollama server's full web address: + + +```json title="~/jan/engines/openai.json" +{ + "full_url": "http://localhost:11434/v1/chat/completions" +} +``` + +### Step 2: Model Configuration + +1. Navigate to the `~/jan/models` folder. +2. Create a folder named `(ollam-modelname)`, for example, `lmstudio-phi-2`. +3. Create a `model.json` file inside the folder including the following configurations: + - Set the `id` property to the model name as Ollama model name. + - Set the `format` property to `api`. + - Set the `engine` property to `openai`. + - Set the `state` property to `ready`. + +```json title="~/jan/models/llama2/model.json" +{ + "sources": [ + { + "filename": "llama2", + "url": "https://ollama.com/library/llama2" + } + ], + "id": "llama2", + "object": "model", + "name": "Ollama - Llama2", + "version": "1.0", + "description": "Llama 2 is a collection of foundation language models ranging from 7B to 70B parameters.", + "format": "api", + "settings": {}, + "parameters": {}, + "metadata": { + "author": "Meta", + "tags": ["General", "Big Context Length"] + }, + "engine": "openai" +} +``` +### Regarding `model.json` + +- In `settings`, two crucial values are: + - `ctx_len`: Defined based on the model's context size. + - `prompt_template`: Defined based on the model's trained template (e.g., ChatML, Alpaca). + - To set up the `prompt_template`: + 1. Visit [Hugging Face](https://huggingface.co/), an open-source machine learning platform. + 2. Find the current model that you're using (e.g., [Gemma 7b it](https://huggingface.co/google/gemma-7b-it)). + 3. Review the text and identify the template. +- In `parameters`, consider the following options. The fields in `parameters` are typically general and can be the same across models. An example is provided below: + +```json +"parameters":{ + "temperature": 0.7, + "top_p": 0.95, + "stream": true, + "max_tokens": 4096, + "frequency_penalty": 0, + "presence_penalty": 0 +} +``` + +### Step 3: Start the Model +1. Restart Jan and navigate to the **Hub**. +2. Locate your model and click the **Use** button. \ No newline at end of file diff --git a/docs/src/theme/DocCard/assets/lm.png b/docs/src/theme/DocCard/assets/lm.png new file mode 100644 index 000000000..ce703ece3 Binary files /dev/null and b/docs/src/theme/DocCard/assets/lm.png differ diff --git a/docs/src/theme/DocCard/assets/mistral.png b/docs/src/theme/DocCard/assets/mistral.png new file mode 100644 index 000000000..1a64414d7 Binary files /dev/null and b/docs/src/theme/DocCard/assets/mistral.png differ diff --git a/docs/src/theme/DocCard/assets/ollama.png b/docs/src/theme/DocCard/assets/ollama.png new file mode 100644 index 000000000..5f5fd4c8f Binary files /dev/null and b/docs/src/theme/DocCard/assets/ollama.png differ