208 lines
7.3 KiB
Plaintext
208 lines
7.3 KiB
Plaintext
---
|
|
title: Remote Server Integration
|
|
sidebar_position: 1
|
|
slug: /guides/engines/remote-server
|
|
description: A step-by-step guide on how to set up Jan to connect with any remote or local API server.
|
|
keywords:
|
|
[
|
|
Jan AI,
|
|
Jan,
|
|
ChatGPT alternative,
|
|
local AI,
|
|
private AI,
|
|
conversational AI,
|
|
no-subscription fee,
|
|
large language model,
|
|
import-models-manually,
|
|
remote server,
|
|
OAI compatible,
|
|
]
|
|
---
|
|
|
|
<head>
|
|
<title>Remote Server Integration</title>
|
|
<meta name="description" content="A step-by-step guide on how to set up Jan to connect with any remote or local API server. Learn how to configure Jan as a client to connect with OpenAI Platform or any OpenAI-compatible endpoint, and how to start models using Jan's Hub."/>
|
|
<meta name="keywords" content="Jan AI, Jan, ChatGPT alternative, local AI, private AI, conversational AI, no-subscription fee, large language model, import-models-manually, remote server, OAI compatible"/>
|
|
<meta property="og:title" content="Remote Server Integration"/>
|
|
<meta property="og:description" content="A step-by-step guide on how to set up Jan to connect with any remote or local API server. Learn how to configure Jan as a client to connect with OpenAI Platform or any OpenAI-compatible endpoint, and how to start models using Jan's Hub."/>
|
|
<meta property="og:url" content="https://jan.ai/guides/remote-server-integration"/>
|
|
<meta name="twitter:card" content="summary"/>
|
|
<meta name="twitter:title" content="Remote Server Integration"/>
|
|
<meta name="twitter:description" content="A step-by-step guide on how to set up Jan to connect with any remote or local API server. Learn how to configure Jan as a client to connect with OpenAI Platform or any OpenAI-compatible endpoint, and how to start models using Jan's Hub."/>
|
|
</head>
|
|
|
|
This guide will show you how to configure Jan as a client and point it to any remote & local (self-hosted) API server.
|
|
|
|
## OpenAI Platform Configuration
|
|
|
|
### 1. Create a Model JSON
|
|
|
|
1. In `~/jan/models`, create a folder named `gpt-3.5-turbo-16k`.
|
|
|
|
2. In this folder, add a `model.json` file with Filename as `model.json`, `id` matching folder name, `Format` as `api`, `Engine` as `openai`, and `State` as `ready`.
|
|
|
|
|
|
```json title="~/jan/models/gpt-3.5-turbo-16k/model.json"
|
|
{
|
|
"sources": [
|
|
{
|
|
"filename": "openai",
|
|
"url": "https://openai.com"
|
|
}
|
|
],
|
|
"id": "gpt-3.5-turbo-16k",
|
|
"object": "model",
|
|
"name": "OpenAI GPT 3.5 Turbo 16k",
|
|
"version": "1.0",
|
|
"description": "OpenAI GPT 3.5 Turbo 16k model is extremely good",
|
|
"format": "api",
|
|
"settings": {},
|
|
"parameters": {},
|
|
"metadata": {
|
|
"author": "OpenAI",
|
|
"tags": ["General", "Big Context Length"]
|
|
},
|
|
"engine": "openai"
|
|
}
|
|
```
|
|
|
|
### `model.json`
|
|
|
|
The `model.json` file is used to set up your local models.
|
|
:::note
|
|
- If you've set up your model's configuration in `nitro.json`, please note that `model.json` can overwrite the settings.
|
|
- When using OpenAI models like GPT-3.5 and GPT-4, you can use the default settings in `model.json` file.
|
|
:::
|
|
|
|
There are two important fields in model.json that you need to setup:
|
|
|
|
#### Settings
|
|
This is the field where to set your engine configurations, there are two imporant field that you need to define for your local models:
|
|
|
|
| Term | Description |
|
|
|-------------------|---------------------------------------------------------|
|
|
| `ctx_len` | Defined based on the model's context size. |
|
|
| `prompt_template` | Defined based on the model's trained template (e.g., ChatML, Alpaca). |
|
|
|
|
To set up the `prompt_template` based on your model, follow the steps below:
|
|
1. Visit [Hugging Face](https://huggingface.co/), an open-source machine learning platform.
|
|
2. Find the current model that you're using (e.g., [Gemma 7b it](https://huggingface.co/google/gemma-7b-it)).
|
|
3. Review the text and identify the template.
|
|
|
|
#### Parameters
|
|
`parameters` is the adjustable settings that affect how your model operates or processes the data.
|
|
The fields in `parameters` are typically general and can be the same across models. An example is provided below:
|
|
|
|
```json
|
|
"parameters":{
|
|
"temperature": 0.7,
|
|
"top_p": 0.95,
|
|
"stream": true,
|
|
"max_tokens": 4096,
|
|
"frequency_penalty": 0,
|
|
"presence_penalty": 0
|
|
}
|
|
```
|
|
|
|
|
|
:::tip
|
|
|
|
- You can find the list of available models in the [OpenAI Platform](https://platform.openai.com/docs/models/overview).
|
|
- The `id` property needs to match the model name in the list.
|
|
- For example, if you want to use the [GPT-4 Turbo](https://platform.openai.com/docs/models/gpt-4-and-gpt-4-turbo), you must set the `id` property to `gpt-4-1106-preview`.
|
|
|
|
:::
|
|
|
|
### 2. Configure OpenAI API Keys
|
|
|
|
1. Find your API keys in the [OpenAI Platform](https://platform.openai.com/api-keys).
|
|
2. Set the OpenAI API keys in `~/jan/engines/openai.json` file.
|
|
|
|
```json title="~/jan/engines/openai.json"
|
|
{
|
|
"full_url": "https://api.openai.com/v1/chat/completions",
|
|
"api_key": "sk-<your key here>"
|
|
}
|
|
```
|
|
|
|
### 3. Start the Model
|
|
|
|
Restart Jan and navigate to the Hub. Then, select your configured model and start the model.
|
|
|
|
## Engines with OAI Compatible Configuration
|
|
|
|
This section will show you how to configure a client connection to a remote/local server using Jan's API server running model `mistral-ins-7b-q4` as an example.
|
|
|
|
:::note
|
|
|
|
Currently, you can only connect to one OpenAI-compatible endpoint at a time.
|
|
|
|
:::
|
|
|
|
### 1. Configure a Client Connection
|
|
|
|
1. Navigate to the `~/jan/engines` folder.
|
|
2. Modify the `openai.json file`.
|
|
|
|
:::note
|
|
|
|
Please note that currently, the code that supports any OpenAI-compatible endpoint only reads `engine/openai.json` file. Thus, it will not search any other files in this directory.
|
|
|
|
:::
|
|
|
|
3. Configure `full_url` properties with the endpoint server that you want to connect. For example, if you're going to communicate to Jan's API server, you can configure it as follows:
|
|
|
|
```json title="~/jan/engines/openai.json"
|
|
{
|
|
// "full_url": "https://<server-ip-address>:<port>/v1/chat/completions"
|
|
"full_url": "https://<server-ip-address>:1337/v1/chat/completions"
|
|
// Skip api_key if your local server does not require authentication
|
|
// "api_key": "sk-<your key here>"
|
|
}
|
|
```
|
|
|
|
### 2. Create a Model JSON
|
|
|
|
1. In `~/jan/models`, create a folder named `mistral-ins-7b-q4`.
|
|
|
|
2. In this folder, add a `model.json` file with Filename as `model.json`, ensure the following configurations:
|
|
- `id` matching folder name.
|
|
- `Format` set to `api`.
|
|
- `Engine` set to `openai`
|
|
- `State` set to `ready`.
|
|
|
|
|
|
```json title="~/jan/models/mistral-ins-7b-q4/model.json"
|
|
{
|
|
"sources": [
|
|
{
|
|
"filename": "janai",
|
|
"url": "https://jan.ai"
|
|
}
|
|
],
|
|
"id": "mistral-ins-7b-q4",
|
|
"object": "model",
|
|
"name": "Mistral Instruct 7B Q4 on Jan API Server",
|
|
"version": "1.0",
|
|
"description": "Jan integration with remote Jan API server",
|
|
"format": "api",
|
|
"settings": {},
|
|
"parameters": {},
|
|
"metadata": {
|
|
"author": "MistralAI, The Bloke",
|
|
"tags": ["remote", "awesome"]
|
|
},
|
|
"engine": "openai"
|
|
}
|
|
|
|
```
|
|
### 3. Start the Model
|
|
|
|
1. Restart Jan and navigate to the **Hub**.
|
|
2. Locate your model and click the **Use** button.
|
|
|
|
:::info[Assistance and Support]
|
|
|
|
If you have questions or want more preconfigured GGUF models, please join our [Discord community](https://discord.gg/Dt7MxDyNNZ) for support, updates, and discussions.
|
|
|
|
::: |