Louis d85d02693b
feat: Nitro-Tensorrt-LLM Extension (#2280)
* feat: tensorrt-llm-extension

* fix: loading

* feat: add download tensorrt llm runner

Signed-off-by: James <james@jan.ai>

* feat: update to rollupjs instead of webpack for monitoring extension

Signed-off-by: James <james@jan.ai>

* feat: move update nvidia info to monitor extension

Signed-off-by: James <james@jan.ai>

* allow download tensorrt

Signed-off-by: James <james@jan.ai>

* update

Signed-off-by: James <james@jan.ai>

* allow download tensor rt based on gpu setting

Signed-off-by: James <james@jan.ai>

* update downloaded models

Signed-off-by: James <james@jan.ai>

* feat: add extension compatibility

* dynamic tensor rt engines

Signed-off-by: James <james@jan.ai>

* update models

Signed-off-by: James <james@jan.ai>

* chore: remove ts-ignore

* feat: getting installation state from extension

Signed-off-by: James <james@jan.ai>

* chore: adding type for decompress

Signed-off-by: James <james@jan.ai>

* feat: update according Louis's comment

Signed-off-by: James <james@jan.ai>

* feat: add progress for installing extension

Signed-off-by: James <james@jan.ai>

* chore: remove args from extension installation

* fix: model download does not work properly

* fix: do not allow user to stop tensorrtllm inference

* fix: extension installed style

* fix: download tensorrt does not update state

Signed-off-by: James <james@jan.ai>

* chore: replace int4 by fl16

* feat: modal for installing extension

Signed-off-by: James <james@jan.ai>

* fix: start download immediately after press install

Signed-off-by: James <james@jan.ai>

* fix: error switching between engines

* feat: rename inference provider to ai engine and refactor to core

* fix: missing ulid

* fix: core bundler

* feat: add cancel extension installing

Signed-off-by: James <james@jan.ai>

* remove mocking for mac

Signed-off-by: James <james@jan.ai>

* fix: show models only when extension is ready

* add tensorrt badge for model

Signed-off-by: James <james@jan.ai>

* fix: copy

* fix: add compatible check (#2342)

* fix: add compatible check

Signed-off-by: James <james@jan.ai>

* fix: copy

* fix: font

* fix: copy

* fix: broken monitoring extension

* chore: bump engine

* fix: copy

* fix: model copy

* fix: copy

* fix: model json

---------

Signed-off-by: James <james@jan.ai>
Co-authored-by: James <james@jan.ai>
Co-authored-by: Louis <louis@jan.ai>

* fix: vulkan support

* fix: installation button padding

* fix: empty script

* fix: remove hard code string

---------

Signed-off-by: James <james@jan.ai>
Co-authored-by: James <james@jan.ai>
Co-authored-by: NamH <NamNh0122@gmail.com>
2024-03-14 14:07:22 +07:00
..
2024-02-15 08:38:05 +07:00
2024-02-15 08:38:05 +07:00

@janhq/core

This module includes functions for communicating with core APIs, registering app extensions, and exporting type definitions.

Usage

Import the package

// Web / extension runtime
import * as core from "@janhq/core";

// Node runtime
import * as node from "@janhq/core/node";

Build an Extension

  1. Download an extension template, for example, https://github.com/janhq/extension-template.

  2. Update the source code:

    1. Open index.ts in your code editor.
    2. Rename the extension class from SampleExtension to your preferred extension name.
    3. Import modules from the core package.
      import * as core from "@janhq/core";
      
    4. In the onLoad() method, add your code:
      // Example of listening to app events and providing customized inference logic:
      import * as core from "@janhq/core";
      
      export default class MyExtension extends BaseExtension {
        // On extension load
        onLoad() {
          core.events.on(MessageEvent.OnMessageSent, (data) => MyExtension.inference(data, this));
        }
      
        // Customized inference logic
        private static inference(incomingMessage: MessageRequestData) {
      
          // Prepare customized message content
          const content: ThreadContent = {
            type: ContentType.Text,
            text: {
              value: "I'm Jan Assistant!",
              annotations: [],
            },
          };
      
          // Modify message and send out
          const outGoingMessage: ThreadMessage = {
            ...incomingMessage,
            content
          };
        }
      }
      
  3. Build the extension:

    1. Navigate to the extension directory.
    2. Install dependencies.
      yarn install
      
    3. Compile the source code. The following command keeps running in the terminal and rebuilds the extension when you modify the source code.
      yarn build
      
    4. Select the generated .tgz from Jan > Settings > Extension > Manual Installation.