jan/adr/adr-001-jan-deployable-cloud-native.md

ADR #001: Jan deployable cloud-native

Changelog

• 23.10.03: Initial unfinished draft

Authors

• @nam-john-ho
• @louis

Context

Status Quo

• User doesn't have a local GPU machine but wants to run Jan on a rented server
• User wants a quick, fast way to experiment with Jan on a rented GPU
• https://github.com/janhq/jan/issues/255

Decision

• This ADR aims to outline design decisions for deploying Jan in cloud native environments such as: Runpod, AWS, Azure, GCP in a fast and simple way.
• The current code-base should not change too much.
• The current plugins must be reusable across enviroments (Desktop, Cloud-native).
• Simple authentication (username/password) must be supported.

Key Design Decisions

Introduce 2 components in Jan:

• Middleware: responsible for routing the user interface to the appropriate platform (Electron/WebApp) that is built when packaged.
• Http server: a http server on cloud environment which interacts with plugin directly.

Detailed Design

FE

• Middleware: TBD
• Httpserver: TBD
• Custom build for httpweb/electron: TBD

Devops:

• Allow to pass username/password as environment variables
• Assign a Public IP to the instance
• Customize instance types on-demand.

• Runpod: TBD
• AWS: TBD
• Azure: TBD
• GCP: TBD

Alternative Approaches

Separated server process runs along side with electron. 6005409a94

Considerations

References

• https://www.runpod.io/console/templates
• https://repost.aws/articles/ARQ0Tz9eorSL6EAus7XPMG-Q/how-to-install-textgen-webui-on-aws
• https://www.youtube.com/watch?v=_59AsSyMERQ
• https://gpus.llm-utils.org/running-llama-2-on-runpod-with-oobaboogas-text-generation-webui/
• https://medium.com/@jarimh1984/installing-oobabooga-and-oobabooga-api-to-runpod-cloud-step-by-step-tutorial-47457974dfa5