134 lines
6.9 KiB
Plaintext
134 lines
6.9 KiB
Plaintext
---
|
|
title: Install with Docker
|
|
sidebar_position: 4
|
|
slug: /guides/install/server
|
|
hide_table_of_contents: true
|
|
description: A step-by-step guide to install Jan using Docker.
|
|
keywords:
|
|
[
|
|
Jan AI,
|
|
Jan,
|
|
ChatGPT alternative,
|
|
local AI,
|
|
private AI,
|
|
conversational AI,
|
|
no-subscription fee,
|
|
large language model,
|
|
Install on Docker,
|
|
Docker,
|
|
Helm,
|
|
]
|
|
---
|
|
import Tabs from '@theme/Tabs';
|
|
import TabItem from '@theme/TabItem';
|
|
|
|
### Pre-requisites
|
|
Ensure that your system meets the following requirements:
|
|
- Linux or WSL2 Docker
|
|
- Latest Docker Engine and Docker Compose
|
|
|
|
To enable GPU support, you will need:
|
|
- `nvidia-driver`
|
|
- `nvidia-docker2`
|
|
|
|
:::note
|
|
- If you have not installed Docker, follow the instructions [here](https://docs.docker.com/engine/install/ubuntu/).
|
|
- If you have not installed the required file for GPU support, follow the instructions [here](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html).
|
|
:::
|
|
|
|
### Run Jan in Docker
|
|
You can run Jan in Docker with two methods:
|
|
1. Run Jan in CPU mode
|
|
2. Run Jan in GPU mode
|
|
<Tabs groupId = "ldocker_type">
|
|
<TabItem value="docker_cpu" label = "CPU">
|
|
|
|
To run Jan in Docker CPU mode, by using the following code:
|
|
|
|
```bash
|
|
# cpu mode with default file system
|
|
docker compose --profile cpu-fs up -d
|
|
|
|
# cpu mode with S3 file system
|
|
docker compose --profile cpu-s3fs up -d
|
|
```
|
|
|
|
</TabItem>
|
|
<TabItem value="docker_gpu" label = "GPU">
|
|
|
|
To run Jan in Docker CPU mode, follow the steps below:
|
|
1. Check CUDA compatibility with your NVIDIA driver by running nvidia-smi and check the CUDA version in the output as shown below:
|
|
```sh
|
|
nvidia-smi
|
|
|
|
# Output
|
|
+---------------------------------------------------------------------------------------+
|
|
| NVIDIA-SMI 531.18 Driver Version: 531.18 CUDA Version: 12.1 |
|
|
|-----------------------------------------+----------------------+----------------------+
|
|
| GPU Name TCC/WDDM | Bus-Id Disp.A | Volatile Uncorr. ECC |
|
|
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|
|
| | | MIG M. |
|
|
|=========================================+======================+======================|
|
|
| 0 NVIDIA GeForce RTX 4070 Ti WDDM | 00000000:01:00.0 On | N/A |
|
|
| 0% 44C P8 16W / 285W| 1481MiB / 12282MiB | 2% Default |
|
|
| | | N/A |
|
|
+-----------------------------------------+----------------------+----------------------+
|
|
| 1 NVIDIA GeForce GTX 1660 Ti WDDM | 00000000:02:00.0 Off | N/A |
|
|
| 0% 49C P8 14W / 120W| 0MiB / 6144MiB | 0% Default |
|
|
| | | N/A |
|
|
+-----------------------------------------+----------------------+----------------------+
|
|
| 2 NVIDIA GeForce GTX 1660 Ti WDDM | 00000000:05:00.0 Off | N/A |
|
|
| 29% 38C P8 11W / 120W| 0MiB / 6144MiB | 0% Default |
|
|
| | | N/A |
|
|
+-----------------------------------------+----------------------+----------------------+
|
|
|
|
+---------------------------------------------------------------------------------------+
|
|
| Processes: |
|
|
| GPU GI CI PID Type Process name GPU Memory |
|
|
| ID ID Usage |
|
|
|=======================================================================================|
|
|
```
|
|
2. Visit [NVIDIA NGC Catalog](https://catalog.ngc.nvidia.com/orgs/nvidia/containers/cuda/tags) and find the smallest minor version of image tag that matches your CUDA version (e.g., 12.1 -> 12.1.0)
|
|
3. Update the `Dockerfile.gpu` line number 5 with the latest minor version of the image tag from step 2 (e.g. change `FROM nvidia/cuda:12.2.0-runtime-ubuntu22.04 AS base` to `FROM nvidia/cuda:12.1.0-runtime-ubuntu22.04 AS base`)
|
|
4. Run Jan in GPU mode by using the following command:
|
|
|
|
```bash
|
|
# GPU mode with default file system
|
|
docker compose --profile gpu-fs up -d
|
|
|
|
# GPU mode with S3 file system
|
|
docker compose --profile gpu-s3fs up -d
|
|
```
|
|
|
|
</TabItem>
|
|
</Tabs>
|
|
### Docker Compose Profile and Environment
|
|
The available Docker Compose profile and the environment variables listed below:
|
|
|
|
#### Docker Compose Profile
|
|
|
|
| Profile | Description |
|
|
|-----------|-------------------------------------------|
|
|
| cpu-fs | Run Jan in CPU mode with default file system |
|
|
| cpu-s3fs | Run Jan in CPU mode with S3 file system |
|
|
| gpu-fs | Run Jan in GPU mode with default file system |
|
|
| gpu-s3fs | Run Jan in GPU mode with S3 file system |
|
|
|
|
#### Environment Variables
|
|
|
|
| Environment Variable | Description |
|
|
|--------------------------|------------------------------------------------------------|
|
|
| S3_BUCKET_NAME | S3 bucket name - leave blank for default file system |
|
|
| AWS_ACCESS_KEY_ID | AWS access key ID - leave blank for default file system |
|
|
| AWS_SECRET_ACCESS_KEY | AWS secret access key - leave blank for default file system|
|
|
| AWS_ENDPOINT | AWS endpoint URL - leave blank for default file system |
|
|
| AWS_REGION | AWS region - leave blank for default file system |
|
|
| API_BASE_URL | Jan Server URL, please modify it as your public ip address or domain name default http://localhost:1377 |
|
|
|
|
|
|
:::warning
|
|
|
|
If you are stuck in a broken build, go to the [Broken Build](/troubleshooting/#broken-build) section of Common Errors.
|
|
|
|
:::
|