docs: add docs for engine, makeup for models docs
This commit is contained in:
parent
5403d9a8e5
commit
7a64b45365
@ -1,6 +1,59 @@
|
||||
---
|
||||
title: Engine
|
||||
title: Engine Specifications
|
||||
slug: /specs/engine
|
||||
---
|
||||
|
||||
# Hello world
|
||||
:::caution
|
||||
|
||||
Currently Under Development
|
||||
|
||||
:::
|
||||
|
||||
## Overview
|
||||
|
||||
In the Jan application, engines serve as primary entities with the following capabilities:
|
||||
|
||||
- Ability to add or remove engines via the installation of extensions.
|
||||
- Dependence on engines for conducting [inference](https://en.wikipedia.org/wiki/Inference_engine).
|
||||
|
||||
## Folder Structure
|
||||
|
||||
- Default parameters for engines are stored in JSON files located in the `/engines` folder.
|
||||
- These parameter files are named uniquely with `engine_id`.
|
||||
- Engines are referenced directly using `engine_id` in the `model.json` file.
|
||||
|
||||
```yaml
|
||||
jan/
|
||||
engines/
|
||||
nitro.json
|
||||
openai.json
|
||||
.....
|
||||
```
|
||||
|
||||
## Engine Default Parameter Files
|
||||
|
||||
- Each inference engine requires default parameters to function in cases where user-provided parameters are absent.
|
||||
- These parameters are stored in JSON files, structured as simple key-value pairs.
|
||||
|
||||
### Example
|
||||
|
||||
Here is an example of an engine file for `engine_id` `nitro`:
|
||||
|
||||
```js
|
||||
{
|
||||
"llama_model_path": "/model/llama-2-7b-model.gguf",
|
||||
"ctx_len": 512,
|
||||
"ngl": 100,
|
||||
"embedding": false,
|
||||
"n_parallel": 1,
|
||||
"cont_batching": false
|
||||
}
|
||||
```
|
||||
|
||||
For detailed engine parameters, refer to: [Nitro's Model Settings](https://nitro.jan.ai/features/load-unload#table-of-parameters)
|
||||
|
||||
## Adding an Engine
|
||||
|
||||
- Engine parameter files are automatically generated upon installing an `inference-extension` in the Jan application.
|
||||
|
||||
---
|
||||
|
||||
@ -64,7 +64,7 @@ Here's a standard example `model.json` for a GGUF model.
|
||||
"description": null, // Defaults to null
|
||||
"state": enum[null, "downloading", "ready", "starting", "stopping", ...]
|
||||
"format": "ggufv3", // Defaults to "ggufv3"
|
||||
"engine_parameters": { // Engine
|
||||
"engine_parameters": { // Engine parameters inside model.json can override the value inside the base engine.json
|
||||
"ctx_len": 2048,
|
||||
"ngl": 100,
|
||||
"embedding": true,
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user