2.3 KiB
2.3 KiB
ADR #001: Jan deployable cloud-native
Changelog
- 23.10.03: Initial unfinished draft
Authors
- @nam-john-ho
- @louis
Context
Status Quo
- User doesn't have a local GPU machine but wants to run Jan on a rented server
- User wants a quick, fast way to experiment with Jan on a rented GPU
- https://github.com/janhq/jan/issues/255
Decision
- This ADR aims to outline design decisions for deploying Jan in cloud native environments such as: Runpod, AWS, Azure, GCP in a fast and simple way.
- The current code-base should not change too much.
- The current plugins must be reusable across enviroments (Desktop, Cloud-native).
- Simple authentication (username/password) must be supported.
Key Design Decisions
Why middleware
- The /web codebase needs to operate in both browser and electron environments
- The /web codebase needs to route plugin routes accordingly, either to /server or /electron
- Middleware takes care of this
- We will have a /server codebase that takes care of routing to plugins
Unsuitable Alternatives
- Not possible to just run electron headless
- /web is on a different chromium window
- Does not have all the electron handlers
- Does not have the IPC handler
Detailed Design
FE
- Middleware:
- Httpserver: TBD
- Custom build for httpweb/electron: TBD
- IPC
- Electron imports in plugins
Devops:
- Allow to pass username/password as environment variables
- Assign a Public IP to the instance
- Customize instance types on-demand.
- Runpod: TBD
- AWS: TBD
- Azure: TBD
- GCP: TBD
Alternative Approaches
Separated server process runs along side with electron. 6005409a94
Considerations
- In the first version, all current plugins will be loaded in the web version by default, and users will not be allowed to add, remove, or update plugins.
References
- https://www.runpod.io/console/templates
- https://repost.aws/articles/ARQ0Tz9eorSL6EAus7XPMG-Q/how-to-install-textgen-webui-on-aws
- https://www.youtube.com/watch?v=_59AsSyMERQ
- https://gpus.llm-utils.org/running-llama-2-on-runpod-with-oobaboogas-text-generation-webui/
- https://medium.com/@jarimh1984/installing-oobabooga-and-oobabooga-api-to-runpod-cloud-step-by-step-tutorial-47457974dfa5
