* feat: tensorrt-llm-extension * fix: loading * feat: add download tensorrt llm runner Signed-off-by: James <james@jan.ai> * feat: update to rollupjs instead of webpack for monitoring extension Signed-off-by: James <james@jan.ai> * feat: move update nvidia info to monitor extension Signed-off-by: James <james@jan.ai> * allow download tensorrt Signed-off-by: James <james@jan.ai> * update Signed-off-by: James <james@jan.ai> * allow download tensor rt based on gpu setting Signed-off-by: James <james@jan.ai> * update downloaded models Signed-off-by: James <james@jan.ai> * feat: add extension compatibility * dynamic tensor rt engines Signed-off-by: James <james@jan.ai> * update models Signed-off-by: James <james@jan.ai> * chore: remove ts-ignore * feat: getting installation state from extension Signed-off-by: James <james@jan.ai> * chore: adding type for decompress Signed-off-by: James <james@jan.ai> * feat: update according Louis's comment Signed-off-by: James <james@jan.ai> * feat: add progress for installing extension Signed-off-by: James <james@jan.ai> * chore: remove args from extension installation * fix: model download does not work properly * fix: do not allow user to stop tensorrtllm inference * fix: extension installed style * fix: download tensorrt does not update state Signed-off-by: James <james@jan.ai> * chore: replace int4 by fl16 * feat: modal for installing extension Signed-off-by: James <james@jan.ai> * fix: start download immediately after press install Signed-off-by: James <james@jan.ai> * fix: error switching between engines * feat: rename inference provider to ai engine and refactor to core * fix: missing ulid * fix: core bundler * feat: add cancel extension installing Signed-off-by: James <james@jan.ai> * remove mocking for mac Signed-off-by: James <james@jan.ai> * fix: show models only when extension is ready * add tensorrt badge for model Signed-off-by: James <james@jan.ai> * fix: copy * fix: add compatible check (#2342) * fix: add compatible check Signed-off-by: James <james@jan.ai> * fix: copy * fix: font * fix: copy * fix: broken monitoring extension * chore: bump engine * fix: copy * fix: model copy * fix: copy * fix: model json --------- Signed-off-by: James <james@jan.ai> Co-authored-by: James <james@jan.ai> Co-authored-by: Louis <louis@jan.ai> * fix: vulkan support * fix: installation button padding * fix: empty script * fix: remove hard code string --------- Signed-off-by: James <james@jan.ai> Co-authored-by: James <james@jan.ai> Co-authored-by: NamH <NamNh0122@gmail.com>
Jan inference plugin
Created using Jan app example
Create a Jan Plugin using Typescript
Use this template to bootstrap the creation of a TypeScript Jan plugin. 🚀
Create Your Own Plugin
To create your own plugin, you can use this repository as a template! Just follow the below instructions:
- Click the Use this template button at the top of the repository
- Select Create a new repository
- Select an owner and name for your new repository
- Click Create repository
- Clone your new repository
Initial Setup
After you've cloned the repository to your local machine or codespace, you'll need to perform some initial setup steps before you can develop your plugin.
Note
You'll need to have a reasonably modern version of Node.js handy. If you are using a version manager like
nodenvornvm, you can runnodenv installin the root of your repository to install the version specified inpackage.json. Otherwise, 20.x or later should work!
-
🛠️ Install the dependencies
npm install -
🏗️ Package the TypeScript for distribution
npm run bundle -
✅ Check your artifact
There will be a tgz file in your plugin directory now
Update the Plugin Metadata
The package.json file defines metadata about your plugin, such as
plugin name, main entry, description and version.
When you copy this repository, update package.json with the name, description for your plugin.
Update the Plugin Code
The src/ directory is the heart of your plugin! This contains the
source code that will be run when your plugin extension functions are invoked. You can replace the
contents of this directory with your own code.
There are a few things to keep in mind when writing your plugin code:
-
Most Jan Plugin Extension functions are processed asynchronously. In
index.ts, you will see that the extension function will return aPromise<any>.import { core } from '@janhq/core' function onStart(): Promise<any> { return core.invokePluginFunc(MODULE_PATH, 'run', 0) }For more information about the Jan Plugin Core module, see the documentation.
So, what are you waiting for? Go ahead and start customizing your plugin!