jan/engine.md at bf1abd377c99a67e3e0a97bd05f6112dd0231c33 - jan - Gitea: Git with a cup of tea

Nicholai/jan

hiro f8d970401a chore: Update head metadata

2024-03-19 10:47:46 +07:00

2.1 KiB

Raw Blame History

title
Engine

<head> </head>

:::caution

Currently Under Development

:::

Overview

In the Jan application, engines serve as primary entities with the following capabilities:

Engine will be installed through inference-extensions.
Models will depend on engines to do inference.
Engine configuration and required metadata will be stored in a json file.

Folder Structure

Default parameters for engines are stored in JSON files located in the /engines folder.
These parameter files are named uniquely with engine_id.
Engines are referenced directly using engine_id in the model.json file.

jan/
engines/
nitro.json
openai.json
.....

Engine Default Parameter Files

Each inference engine requires default parameters to function in cases where user-provided parameters are absent.
These parameters are stored in JSON files, structured as simple key-value pairs.

Example

Here is an example of an engine file for engine_id nitro:

{
    "ctx_len": 512,
    "ngl": 100,
    "embedding": false,
    "n_parallel": 1,
    "cont_batching": false
    "prompt_template": "<|im_start|>system\n{system_message}<|im_end|>\n<|im_start|>user\n{prompt}<|im_end|>\n<|im_start|>assistant"
}

For detailed engine parameters, refer to: Nitro's Model Settings

Adding an Engine

Engine parameter files are automatically generated upon installing an inference-extension in the Jan application.