169 lines
4.8 KiB
Plaintext
169 lines
4.8 KiB
Plaintext
---
|
|
sidebar_position: 2
|
|
---
|
|
|
|
# Remote Server Integration
|
|
|
|
A step-by-step guide on how to set up Jan to connect with any remote or local API server.
|
|
|
|
---
|
|
|
|
:::warning
|
|
|
|
This is currently under development.
|
|
|
|
:::
|
|
|
|
This guide will show you how to configure Jan as a client and point it to any remote & local (self-hosted) API server.
|
|
|
|
## OpenAI Platform Configuration
|
|
|
|
### 1. Create a Model JSON
|
|
|
|
1. In `~/jan/models`, create a folder named `gpt-3.5-turbo-16k`.
|
|
|
|
2. In this folder, add a `model.json` file with Filename as `model.json`, `id` matching folder name, `Format` as `api`, `Engine` as `openai`, and `State` as `ready`.
|
|
|
|
|
|
```json title="~/jan/models/gpt-3.5-turbo-16k/model.json"
|
|
{
|
|
"sources": [
|
|
{
|
|
"filename": "openai",
|
|
"url": "https://openai.com"
|
|
}
|
|
],
|
|
"id": "gpt-3.5-turbo-16k",
|
|
"object": "model",
|
|
"name": "OpenAI GPT 3.5 Turbo 16k",
|
|
"version": "1.0",
|
|
"description": "OpenAI GPT 3.5 Turbo 16k model is extremely good",
|
|
"format": "api",
|
|
"settings": {},
|
|
"parameters": {},
|
|
"metadata": {
|
|
"author": "OpenAI",
|
|
"tags": ["General", "Big Context Length"]
|
|
},
|
|
"engine": "openai"
|
|
}
|
|
```
|
|
|
|
#### Regarding `model.json`
|
|
|
|
- In `settings`, two crucial values are:
|
|
- `ctx_len`: Defined based on the model's context size.
|
|
- `prompt_template`: Defined based on the model's trained template (e.g., ChatML, Alpaca).
|
|
- To set up the `prompt_template`:
|
|
1. Visit Hugging Face.
|
|
2. Locate the model (e.g., [Gemma 7b it](https://huggingface.co/google/gemma-7b-it)).
|
|
3. Review the text and identify the template.
|
|
- In `parameters`, consider the following options. The fields in `parameters` are typically general and can be the same across models. An example is provided below:
|
|
|
|
```json
|
|
"parameters":{
|
|
"temperature": 0.7,
|
|
"top_p": 0.95,
|
|
"stream": true,
|
|
"max_tokens": 4096,
|
|
"frequency_penalty": 0,
|
|
"presence_penalty": 0
|
|
}
|
|
```
|
|
|
|
:::tip
|
|
|
|
- You can find the list of available models in the [OpenAI Platform](https://platform.openai.com/docs/models/overview).
|
|
- The `id` property needs to match the model name in the list.
|
|
- For example, if you want to use the [GPT-4 Turbo](https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo), you must set the `id` property to `gpt-4-1106-preview`.
|
|
|
|
:::
|
|
|
|
### 2. Configure OpenAI API Keys
|
|
|
|
1. Find your API keys in the [OpenAI Platform](https://platform.openai.com/api-keys).
|
|
2. Set the OpenAI API keys in `~/jan/engines/openai.json` file.
|
|
|
|
```json title="~/jan/engines/openai.json"
|
|
{
|
|
"full_url": "https://api.openai.com/v1/chat/completions",
|
|
"api_key": "sk-<your key here>"
|
|
}
|
|
```
|
|
|
|
### 3. Start the Model
|
|
|
|
Restart Jan and navigate to the Hub. Then, select your configured model and start the model.
|
|
|
|
## Engines with OAI Compatible Configuration
|
|
|
|
This section will show you how to configure a client connection to a remote/local server using Jan's API server running model `mistral-ins-7b-q4` as an example.
|
|
|
|
:::note
|
|
|
|
Currently, you can only connect to one OpenAI-compatible endpoint at a time.
|
|
|
|
:::
|
|
|
|
### 1. Configure a Client Connection
|
|
|
|
1. Navigate to the `~/jan/engines` folder.
|
|
2. Modify the `openai.json file`.
|
|
|
|
:::note
|
|
|
|
Please note that currently, the code that supports any OpenAI-compatible endpoint only reads `engine/openai.json` file. Thus, it will not search any other files in this directory.
|
|
|
|
:::
|
|
|
|
3. Configure `full_url` properties with the endpoint server that you want to connect. For example, if you're going to communicate to Jan's API server, you can configure it as follows:
|
|
|
|
```json title="~/jan/engines/openai.json"
|
|
{
|
|
// "full_url": "https://<server-ip-address>:<port>/v1/chat/completions"
|
|
"full_url": "https://<server-ip-address>:1337/v1/chat/completions"
|
|
// Skip api_key if your local server does not require authentication
|
|
// "api_key": "sk-<your key here>"
|
|
}
|
|
```
|
|
|
|
### 2. Create a Model JSON
|
|
|
|
1. In `~/jan/models`, create a folder named `mistral-ins-7b-q4`.
|
|
|
|
2. In this folder, add a `model.json` file with Filename as `model.json`, `id` matching folder name, `Format` as `api`, `Engine` as `openai`, and `State` as `ready`.
|
|
|
|
|
|
```json title="~/jan/models/mistral-ins-7b-q4/model.json"
|
|
{
|
|
"sources": [
|
|
{
|
|
"filename": "janai",
|
|
"url": "https://jan.ai"
|
|
}
|
|
],
|
|
"id": "mistral-ins-7b-q4",
|
|
"object": "model",
|
|
"name": "Mistral Instruct 7B Q4 on Jan API Server",
|
|
"version": "1.0",
|
|
"description": "Jan integration with remote Jan API server",
|
|
"format": "api",
|
|
"settings": {},
|
|
"parameters": {},
|
|
"metadata": {
|
|
"author": "MistralAI, The Bloke",
|
|
"tags": ["remote", "awesome"]
|
|
},
|
|
"engine": "openai"
|
|
}
|
|
```
|
|
|
|
### 3. Start the Model
|
|
|
|
Restart Jan and navigate to the **Hub**. Locate your model and click the **Use** button.
|
|
|
|
:::info[Assistance and Support]
|
|
|
|
If you have questions or want more preconfigured GGUF models, please join our [Discord community](https://discord.gg/Dt7MxDyNNZ) for support, updates, and discussions.
|
|
|
|
::: |