167 lines
5.2 KiB
Plaintext
167 lines
5.2 KiB
Plaintext
---
|
|
title: Integrate With a Remote Server
|
|
slug: /guides/using-models/integrate-with-remote-server
|
|
description: Guide to integrate with a remote server.
|
|
keywords:
|
|
[
|
|
Jan AI,
|
|
Jan,
|
|
ChatGPT alternative,
|
|
local AI,
|
|
private AI,
|
|
conversational AI,
|
|
no-subscription fee,
|
|
large language model,
|
|
import-models-manually,
|
|
remote server,
|
|
OAI compatible,
|
|
]
|
|
---
|
|
|
|
:::caution
|
|
This is currently under development.
|
|
:::
|
|
|
|
In this guide, we will show you how to configure Jan as a client and point it to any remote & local (self-hosted) API server.
|
|
|
|
## OpenAI Platform Configuration
|
|
|
|
In this section, we will show you how to configure with OpenAI Platform, using the OpenAI GPT 3.5 Turbo 16k model as an example.
|
|
|
|
### 1. Create a Model JSON
|
|
|
|
Navigate to the `~/jan/models` folder. Create a folder named `gpt-3.5-turbo-16k` and create a `model.json` file inside the folder including the following configurations:
|
|
|
|
- Ensure the filename must be `model.json`.
|
|
- Ensure the `id` property matches the folder name you created.
|
|
- Ensure the `format` property is set to `api`.
|
|
- Ensure the `engine` property is set to `openai`.
|
|
- Ensure the `state` property is set to `ready`.
|
|
|
|
```json title="~/jan/models/gpt-3.5-turbo-16k/model.json"
|
|
{
|
|
"sources": [
|
|
{
|
|
"filename": "openai",
|
|
"url": "https://openai.com"
|
|
}
|
|
],
|
|
// highlight-next-line
|
|
"id": "gpt-3.5-turbo-16k",
|
|
"object": "model",
|
|
"name": "OpenAI GPT 3.5 Turbo 16k",
|
|
"version": "1.0",
|
|
"description": "OpenAI GPT 3.5 Turbo 16k model is extremely good",
|
|
// highlight-start
|
|
"format": "api",
|
|
"settings": {},
|
|
"parameters": {},
|
|
"metadata": {
|
|
"author": "OpenAI",
|
|
"tags": ["General", "Big Context Length"]
|
|
},
|
|
"engine": "openai"
|
|
// highlight-end
|
|
}
|
|
```
|
|
|
|
:::tip
|
|
|
|
- You can find the list of available models in the [OpenAI Platform](https://platform.openai.com/docs/models/overview).
|
|
- Please note that the `id` property need to match the model name in the list. For example, if you want to use the [GPT-4 Turbo](https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo), you need to set the `id` property as `gpt-4-1106-preview`.
|
|
|
|
:::
|
|
|
|
### 2. Configure OpenAI API Keys
|
|
|
|
You can find your API keys in the [OpenAI Platform](https://platform.openai.com/api-keys) and set the OpenAI API keys in `~/jan/engines/openai.json` file.
|
|
|
|
```json title="~/jan/engines/openai.json"
|
|
{
|
|
"full_url": "https://api.openai.com/v1/chat/completions",
|
|
// highlight-next-line
|
|
"api_key": "sk-<your key here>"
|
|
}
|
|
```
|
|
|
|
### 3. Start the Model
|
|
|
|
Restart Jan and navigate to the Hub. Then, select your configured model and start the model.
|
|
|
|
|
|
|
|
## Engines with OAI Compatible Configuration
|
|
|
|
In this section, we will show you how to configure a client connection to a remote/local server, using Jan's API server that is running model `mistral-ins-7b-q4` as an example.
|
|
|
|
:::note
|
|
|
|
- Please note that at the moment, you can only connect to one OpenAI compatible endpoint at a time.
|
|
|
|
:::
|
|
|
|
### 1. Configure a Client Connection
|
|
|
|
Navigate to the `~/jan/engines` folder and modify the `openai.json` file. Please note that at the moment the code that supports any openai compatible endpoint only reads `engine/openai.json` file, thus, it will not search any other files in this directory.
|
|
|
|
Configure `full_url` properties with the endpoint server that you want to connect. For example, if you want to connect to Jan's API server, you can configure it as follows:
|
|
|
|
```json title="~/jan/engines/openai.json"
|
|
{
|
|
// highlight-start
|
|
// "full_url": "https://<server-ip-address>:<port>/v1/chat/completions"
|
|
"full_url": "https://<server-ip-address>:1337/v1/chat/completions"
|
|
// highlight-end
|
|
// Skip api_key if your local server does not require authentication
|
|
// "api_key": "sk-<your key here>"
|
|
}
|
|
```
|
|
|
|
### 2. Create a Model JSON
|
|
|
|
Navigate to the `~/jan/models` folder. Create a folder named `mistral-ins-7b-q4` and create a `model.json` file inside the folder including the following configurations:
|
|
|
|
- Ensure the filename must be `model.json`.
|
|
- Ensure the `id` property matches the folder name you created.
|
|
- Ensure the `format` property is set to `api`.
|
|
- Ensure the `engine` property is set to `openai`.
|
|
- Ensure the `state` property is set to `ready`.
|
|
|
|
```json title="~/jan/models/mistral-ins-7b-q4/model.json"
|
|
{
|
|
"sources": [
|
|
{
|
|
"filename": "janai",
|
|
"url": "https://jan.ai"
|
|
}
|
|
],
|
|
// highlight-next-line
|
|
"id": "mistral-ins-7b-q4",
|
|
"object": "model",
|
|
"name": "Mistral Instruct 7B Q4 on Jan API Server",
|
|
"version": "1.0",
|
|
"description": "Jan integration with remote Jan API server",
|
|
// highlight-next-line
|
|
"format": "api",
|
|
"settings": {},
|
|
"parameters": {},
|
|
"metadata": {
|
|
"author": "MistralAI, The Bloke",
|
|
"tags": ["remote", "awesome"]
|
|
},
|
|
// highlight-start
|
|
"engine": "openai"
|
|
// highlight-end
|
|
}
|
|
```
|
|
|
|
### 3. Start the Model
|
|
|
|
Restart Jan and navigate to the Hub. Locate your model and click the Use button.
|
|
|
|
|
|
|
|
## Assistance and Support
|
|
|
|
If you have questions or are looking for more preconfigured GGUF models, please feel free to join our [Discord community](https://discord.gg/Dt7MxDyNNZ) for support, updates, and discussions.
|