excalidraw/docs/architecture/brownfield-architecture.md
2025-09-22 21:47:06 -06:00

26 KiB
Raw Blame History

Excalidraw Monorepo Brownfield Architecture Document

Introduction

This document captures the current, real-world state of the Excalidraw monorepo in this workspace. It is intentionally pragmatic: it highlights how the system is wired today, the trade-offs that exist, and the external services the app relies upon so that future contributors—human or AI—can work effectively without reverseengineering the project from scratch.

Document Scope

Comprehensive documentation of the entire repository (full brownfield analysis; no PRD scoping provided).

Change Log

Date Version Description Author
2025-09-22 1.1 Full repo sweep & accuracy refresh Winston
2025-09-21 1.0 Initial brownfield analysis BMad Master

Quick Reference — Key Files and Entry Points

  • Monorepo root: package.json, yarn.lock, package-lock.json (Yarn workspaces with legacy npm lockfile still present).
  • Web app entry: excalidraw-app/index.tsx, excalidraw-app/App.tsx, excalidraw-app/index.html.
  • Core editor package: packages/excalidraw/index.tsx, packages/excalidraw/components/*, packages/excalidraw/scene/*.
  • Build config: excalidraw-app/vite.config.mts, root vitest.config.mts, scripts/buildPackage.js (esbuild bundler for packages).
  • Env baselines: .env.development, .env.production, plus optional .env.local overlays.
  • Tests setup: setupTests.ts, vitest.config.mts.
  • Docker delivery: Dockerfile, docker-compose.yml.
  • Deployment settings: root vercel.json, public/_headers.
  • External rules: firebase-project/* (Firestore + Storage rules for libraries/collab).

Environment Profiles

Mode Source file Notable Overrides
Dev .env.development Local WS server, AI backend http://localhost:3015, plus exports to local Plus host.
Prod .env.production OSS production endpoints, Excalidraw Plus host, AI SaaS backend, enforced lint/PWA defaults.
Docker Dockerfile args VITE_APP_DISABLE_SENTRY=true, build into nginx static image.
CI GitHub Actions YAML Runs yarn test:all, release, docker, Sentry release, Crowdin sync.

High Level Architecture

Technical Summary

  • Monorepo using Yarn v1 workspaces (excalidraw-app, packages/*, examples/*, dev-docs).
  • Frontend: React 19 (runtime) + TypeScript 4.9; state via custom Jotai store (excalidraw-app/app-jotai.ts).
  • Bundling: Vite 5.x for excalidraw-app; esbuild for package builds; custom font pipeline.
  • Collaboration: Socket.IO client with custom Portal, backed by Firebase RTDB/Storage & VITE_APP_WS_SERVER_URL collab server.
  • Persistence: LocalStorage/IndexedDB for offline; Firebase storage for binary files; VITE_APP_BACKEND_V2_* REST endpoints for share links.
  • AI Add-ons: Optional Diagram-to-Code & Text-to-Diagram flows via VITE_APP_AI_BACKEND with hard rate limiting.
  • Telemetry: Sentry browser integration gated by env; selective analytics via packages/excalidraw/analytics.ts.
  • Docs: Docusaurus site in dev-docs; run independently on port 3003.

Actual Tech Stack (from manifests)

Category Technology / Version Notes
Runtime Node.js 18 22 Enforced via engines.
Framework React 19.0.0 Editor + app; types pinned to 19 as well.
Language TypeScript 4.9.4 Shared across packages (lagging behind React 19 features).
Bundler (app) Vite 5.0.12 + @vitejs/plugin-react Custom aliases map to package sources.
Bundler (pkgs) esbuild + esbuild-sass-plugin scripts/buildPackage.js.
State Jotai 2.11 (custom store) app-jotai.ts wraps primitives.
Testing Vitest 3.0.6 + jsdom 22 Coverage thresholds enforced (60/63/70).
Lint/Format ESLint (@excalidraw/eslint-config), Prettier 2.6 CLI scripts test:code, test:other.
Styling SCSS, CSS modules, custom fonts (Virgil, Cascadia, Assistant).
Collaboration Socket.IO client 4.7, Firebase 11.3 External collab server + GCP Storage.
Build Artifacts Docker (nginx 1.27) Dockerfile builds static site.
Deployment Vercel routes (vercel.json), GitHub Actions Sentry release, docker publish.

Repository Structure Reality Check

excalidraw/
├─ excalidraw-app/            # Vite-powered OSS app shell and integrations
│  ├─ components/             # UI surface, menus, dialogs, AI hooks
│  ├─ collab/                 # Portal + Socket.IO + Firebase glue
│  ├─ data/                   # Share-link API wrappers, storage adapters
│  ├─ app-jotai.ts            # Central Jotai store export
│  ├─ App.tsx                 # Composes editor + integrations
│  ├─ vite.config.mts         # Alias map back into packages/*
│  └─ index.html              # EJS template, theme bootstrap, meta
├─ packages/
│  ├─ excalidraw/             # Published editor package (@excalidraw/excalidraw)
│  │  ├─ components/, scene/, renderer/, data/, hooks/, tests/
│  │  ├─ analytics.ts, workers.ts, polyfill.ts
│  │  └─ index.tsx, index-node.ts
│  ├─ element/                # Geometry + binding logic
│  ├─ common/                 # Shared constants, utilities, event emitters
│  ├─ math/                   # Vector & matrix helpers
│  └─ utils/                  # Canvas helpers, bbox, visual debug
├─ examples/
│  ├─ with-nextjs/            # App Router integration example
│  └─ with-script-in-browser/ # Script-tag embedding demo
├─ dev-docs/                  # Docusaurus developer documentation site
├─ docs/architecture/         # Brownfield + standards docs
├─ firebase-project/          # Firestore/Storage rules used by the live app
├─ scripts/                   # Build/release tooling, font converters, locales
├─ public/                    # Static assets, fonts, legacy SW killer
├─ Dockerfile, docker-compose.yml
├─ vitest.config.mts, setupTests.ts
└─ .github/workflows/         # CI definitions

Workspace Overview

Workspace Purpose
excalidraw-app OSS web experience & integration playground
packages/common Shared constants, colors, emitters
packages/element Shape modeling & delta math
packages/excalidraw Public React editor package + plugins
packages/math Geometry helpers (vectors, curves, intersections)
packages/utils Canvas/shape utilities, debug overlays
examples/* Integration samples (Next.js, vanilla)
dev-docs Developer docs (Docusaurus)

Source Tree and Module Organization

excalidraw-app (OSS web app shell)

  • Purpose: Hosts the public Excalidraw experience, wires the core editor with collaboration, AI, sharing, and Plus upsell flows.
  • State management: app-jotai.ts exports a preconfigured Jotai store; helpers like useAtomWithInitialValue wrap default hydration for atoms.
  • Composition (App.tsx):
    • Imports bulk of @excalidraw/excalidraw components, CommandPalette, share dialogs, etc.
    • Mounts Collab Portal, file manager, Share dialog, AI plugin, Excalidraw Plus iframe export.
    • Handles theme bootstrapping, PWA install prompt, analytics gating (trackEvent).
  • Collaboration (collab/):
    • Collab.tsx manages Socket.IO lifecycle, Firebase uploads, room metadata, idle states, pointer broadcasts; expects VITE_APP_WS_SERVER_URL & Firebase config.
    • Portal.tsx coordinates broadcast vs local render, replicating CRA patterns.
    • CollabError.tsx surfaces connectivity & persistence issues via Jotai atoms.
  • Data layer (data/):
    • index.ts: share link encode/decode, REST calls to VITE_APP_BACKEND_V2_*, encryption helpers, random room ID generation.
    • LocalData.ts: IndexedDB adapter for library items; gracefully migrates old LocalStorage via LibraryLocalStorageMigrationAdapter.
    • FileManager.ts: Encodes binary files (images) for Firebase storage and ensures TTL for deleted files.
    • tabSync.ts: Cross-tab consistency; warns when another tab has fresher data.
  • AI features (components/AI.tsx):
    • Wraps DiagramToCodePlugin and TTDDialog from the editor package.
    • Hits VITE_APP_AI_BACKEND endpoints with strict rate limit handling (HTTP 429 => friendly HTML error + Plus upsell).
  • Configuration:
    • vite.config.mts sets up aliasing back into package source directories, configures fonts chunking, Workbox PWA caching, ESLint overlay toggles, WOFF2 relocation plugin, and sitemap generation.
    • index.html uses EJS placeholders to inject PROD font preload vs dev asset path fallback, handles dark mode preflight, and cookie-based redirects to Excalidraw Plus.
    • .env.* lists all required VITE_APP_* flags controlling PWA, lint overlay, overlays, Firebase, AI, ws server, tracking.
  • Sentry (sentry.ts): environment detection by hostname, sanitized console instrumentation, release annotation via VITE_APP_GIT_SHA.
  • Fonts: public/Assistant-Regular.woff2, Virgil.woff2, Cascadia.woff2 served with custom headers and cached long-term.
  • Custom stats/debug: CustomStats.tsx, components/DebugCanvas, isVisualDebuggerEnabled toggled via local storage.

packages/excalidraw (published editor)

  • Exports: Excalidraw React component, imperative API types, DiagramToCodePlugin, TTDDialog, LiveCollaborationTrigger, exportToBlob, etc.
  • Structure:
    • components/: core UI (toolbars, dialogs, command palette, stats, library, context menu).
    • scene/: rendering pipeline, canvas layers, export helpers (scene/export.ts), scrollbars, zoom management.
    • renderer/: canvas primitives, roundRect, background tiling, frame rendering, pointer overlay.
    • data/: serialization/deserialization (restore.ts), encryption, blob helpers, library sync & merge strategies (data/library.ts).
    • hooks/: internal hooks for selection, pointer, library, keyboard.
    • workers.ts: pooled web worker manager with TTL; throws explicit WorkerInTheMainChunkError if bundler inlines workers by mistake.
    • analytics.ts: extremely restrictive tracking (only whitelisted categories) to avoid privacy issues.
    • polyfill.ts: ensures missing DOM APIs (Intl, PointerEvent) exist in environments (used both in tests and runtime).
    • tests/: Vitest suites for charts, clipboard, mermaid import, etc., using vitest-canvas-mock / fake IndexedDB.
    • index-node.ts: Node script showcasing headless export via canvas bindings (registering Virgil/Cascadia fonts) — used for docs automation.
  • Build artifacts: dist/dev and dist/prod output ESM bundles, CSS, type declarations; exports map ensures type resolution for sub-path imports.
  • Dependencies: Jotai (mirrored from app), Radix UI (popover/tabs), perfect-freehand, roughjs, fractional-indexing, @excalidraw/laser-pointer, etc.
  • Technical quirks:
    • React peer dependency declared for 17/18/19; ensures host apps remain in control.
    • env.cjs used by buildPackage.js to inject .env values into bundles.

packages/element, packages/common, packages/math, packages/utils

  • common: constants (colors, themes, stats panels), Emitter, queue, promise pool, throttle/debounce wrappers, pointer utilities, environment detection.
  • element: shape definitions, binding logic, history diffs, arrow editing, delta calculations, lasso, lasers, frames, WYSIWYG integration; exports reused by tests and plugin authors.
  • math: low-level vector math, intersection algorithms, geometry utilities supporting snapping/frames.
  • utils: bounding box calculations, export helpers, debug draws, shape utilities used by renderer and tests.

examples

  • with-nextjs: demonstrates embedding the editor in Next.js App Router; includes dynamic import patterns and custom asset path configuration.
  • with-script-in-browser: minimal CDN script tag usage; used for manual testing and docs.
  • Scripts expect yarn build:packages prior to running to ensure compiled artifacts exist.

dev-docs

  • Docusaurus site for contributor docs; uses the published @excalidraw/excalidraw package directly.
  • Commands: yarn --cwd dev-docs start, yarn build for static output, yarn typecheck with dedicated TS config.
  • Deployed separately (Vercel config lives within this workspace).

firebase-project

  • firebase.json, firestore.rules, storage.rules, firestore.indexes.json govern OSS Firebase project (libraries & collab rooms).
  • Storage prefixes align with FIREBASE_STORAGE_PREFIXES in app constants.
  • No emulators provided; local dev relies on remote dev project credentials embedded in .env.development.

scripts

  • buildPackage.js: esbuild driver; bundles dev/prod ESM output, handles Sass, injects env variables, writes chunked bundles.
  • release.js: interactive multi-package release (test/next/latest tags), updates package versions, rebuilds packages, optionally commits & publishes to npm.
  • build-locales-coverage.js, locales-coverage-description.js: translation health metrics (Crowdin integration).
  • build-node.js, build-version.js: orchestrate app build metadata (Git SHA) and distribution version stamping.
  • woff2/ utilities: transform font assets for Web.

Other notable areas

  • public/service-worker.js: “self-destruct” service worker to retire old CRA installs when migrating to Vite; safe to remove once adoption confirmed.
  • public/_headers: CORS & cache rules for fonts (Excalidraws CDN expects permissive font access).
  • .bmad-core/: project governance for Codex CLI — do not alter without regeneration.
  • node_modules/: currently present in workspace (not checked into git by default but can inflate AI context; treat as generated).

Data Models and Persistence

Core Runtime Types

  • ExcalidrawElement (and OrderedExcalidrawElement): canonical shape model (id, versioning, roughness, seeds, z-order). Stored in exported .excalidraw JSON and broadcast over collaboration.
  • AppState: large shape describing editor UI state (tool selection, theme, zoom, frames, binding toggles). getDefaultAppState() seeds defaults (note: currentItemRoundness forced "round" outside tests).
  • BinaryFileData: metadata for embedded images/files; stored separately and referenced by FileId.
  • LibraryItems: array of reusable element groups persisted via IndexedDB + optional remote sources.

Client Storage & Sync

  • LocalStorage keys (STORAGE_KEYS in app_constants.ts): interactive state, offline scenes, theme, debug toggles; version keys guard migrations.
  • IndexedDB: LibraryIndexedDBAdapter stores library items; migrations fall back to LocalStorage for legacy data.
  • Cross-tab: tabSync.ts compares version stamps to avoid silent clobbering when multiple tabs are open.

Remote Persistence

  • Share Link Backend (json.excalidraw.com):
    • GET VITE_APP_BACKEND_V2_GET_URL: fetch scene by share code.
    • POST VITE_APP_BACKEND_V2_POST_URL: upload encrypted scene; uses compressData + AES-GCM (encryptData).
  • Firebase:
    • Storage buckets store binary files per room/share (files/rooms/<id>, files/shareLinks/<id>).
    • saveFilesToFirebase batches uploads after filtering duplicates; rate limited via FILE_UPLOAD_MAX_BYTES (4MiB) and TTL for deleted records.
  • Collaboration server:
    • Socket.IO namespace provides WS_EVENTS/WS_SUBTYPES channels for scene init, incremental updates, cursor positions, idle status, viewport bounds, and user follow.
    • Encryption key exchanged via URL hash; enforcement done client-side (RE_COLLAB_LINK).
  • Libraries API:
    • Public library metadata fetched from https://libraries.excalidraw.com and associated Cloud Function backend.
    • ALLOWED_LIBRARY_URLS restricts remote imports for security.

AI Integrations

  • Diagram-to-code: sends image (JPEG dataURL) + text to /v1/ai/diagram-to-code/generate; expects HTML snippet back.
  • Text-to-diagram: posts prompt to /v1/ai/text-to-diagram/generate; returns generatedResponse plus rate limit headers.
  • Error handling returns HTML for 429 to display inside plugin iframe.

Telemetry & Observability

  • analytics.ts only allows command_palette and export categories; respects VITE_APP_ENABLE_TRACKING and dev/test environments.
  • Sentry config ignores known browser noise (IndexedDB closing, SW fetch failures, quota errors) and rewrites console errors with stack traces.
  • Debug canvases and custom stats accessible through UI toggles; test harness logs act() misuse in yellow for easier triage.

Internationalization

  • Translation catalog: packages/excalidraw/locales/*.json (Crowdin pipeline configured via crowdin.yml).
  • app-language/ handles detection (language-detector using i18next-browser-languagedetector), state atoms, and runtime toggles.

Node/Server Usage

  • packages/excalidraw/index-node.ts demonstrates headless rendering by bundling canvas and fonts; useful for PDF/image exports in server contexts.

Build, Tooling, and Developer Workflow

Core Commands

Command Description
yarn install Installs workspace dependencies (respects resolutions).
yarn start Starts Vite dev server on VITE_APP_PORT (defaults 3000).
yarn build:packages Builds @excalidraw/{common,math,element,excalidraw} via esbuild.
yarn build Builds excalidraw-app (depends on built packages) and version metadata.
yarn test / yarn test:app Runs Vitest with jsdom.
yarn test:all tsc + ESLint + Prettier check + Vitest (CI default).
yarn fix Prettier write + ESLint --fix.
yarn build:app:docker Vite production build with Sentry disabled (for Dockerfile).
`yarn release[:test :next
docker compose up Builds nginx image from Dockerfile, serves built app on host 3000.
yarn --cwd dev-docs start Serve Docusaurus docs on port 3003.

Testing & Quality

  • Vitest: jsdom environment, setupTests.ts loads canvas mock, fake IndexedDB, fonts, reroutes setPointerCapture, adds matchMedia stub.
  • Coverage thresholds: lines ≥60%, branches ≥70%, functions ≥63% (enforced in config; failing thresholds fail CI).
  • Linting: yarn test:code uses ESLint with custom config; yarn test:other ensures formatting via Prettier.
  • Type safety: yarn test:typecheck runs tsc in build mode across workspaces; pinned to TS 4.9 (React 19 inference gaps exist).
  • Unit focus: heavy emphasis on deterministic snapshot tests around elements, clipboard, mermaid import; integration/E2E tests absent.

Build & Release Flow

  1. yarn build:packages (esbuild) produces dist/dev|prod bundles with inlined env definitions.
  2. yarn build triggers Vite build with Workbox PWA, fonts relocation, sitemap rewrite.
  3. scripts/build-version.js writes git SHA to excalidraw-app/build/version.json & populates window.__EXCALIDRAW_SHA__.
  4. Docker image built from multi-stage Dockerfile (node builder → nginx static).
  5. vercel.json controls headers, redirects, and output directory for hosting (Excalidraw uses Vercel for OSS site).
  6. Release process: yarn release --tag=<test|next|latest> [--version=x.y.z] updates package versions, rebuilds, optionally publishes to npm and commits with 🎉 message; manual prompts remain (tech debt for automation).

Developer Ergonomics

  • VITE_APP_ENABLE_ESLINT toggles dev overlay; disable if overlay is noisy.
  • VITE_APP_DEV_DISABLE_LIVE_RELOAD modifies WebSocket constructor to silence HMR — critical when debugging service workers.
  • VITE_APP_ENABLE_PWA enables Workbox precaching for local testing (disabled by default).
  • VITE_APP_COLLAPSE_OVERLAY controls Vite overlay collapsed state (useful with multi-agent workflows).
  • Font bundling: Workbox deliberately omits locale chunks & fonts to rely on CDN caching; manual chunk naming ensures translation JS loads lazily.

Documentation Tooling

  • Docusaurus uses older React 18 & @excalidraw/excalidraw@0.18.0; keep in sync when bumping core package to avoid breakage.
  • yarn --cwd dev-docs write-translations integrates with Crowdin for docs localization.

Technical Debt, Risks, and Gotchas

  • Dual lockfiles: package-lock.json lingers alongside Yarn; avoid npm install to prevent divergence.
  • TypeScript version: 4.9 lacks latest JSX transforms; React 19 features may require manual typing or upgrade plan.
  • Node_modules footprint: directory currently present (likely untracked) — ensure CI clean installs.
  • Workers & bundlers: external consumers must ensure worker files remain separate; WorkerInTheMainChunkError surfaces if bundlers inline them (not obvious until runtime).
  • External service reliance: Collaboration requires running excalidraw-room server + Firebase credentials; without them, collab buttons fail silently except for toast.
  • AI backend: Hard-coded rate limits; endpoints unreachable in offline dev unless local backend (3015) provided, causing plugin UI errors.
  • Legacy service worker: public/service-worker.js only unregisters CRA workers. Removing prematurely reintroduces white-screen issues for old caches.
  • Release automation: manual prompts for publishing/committing; automating or documenting CLI usage is essential to avoid half-published versions.
  • Fonts pipeline: WOFF2 names assume certain hyphenation; adding new fonts requires updating vercel.json headers + public/_headers to maintain CORS.
  • Env sprawl: dozens of VITE_APP_* toggles; missing any yields runtime errors (Sentry, overlays, collab). Use .env.example (not provided) or replicate .env.development.

Integration Points and External Services

Service / Endpoint Purpose Configuration Integration Notes
VITE_APP_WS_SERVER_URL (Socket.IO) Real-time collaboration .env.* Requires external excalidraw-room server; handles SCENE_*, cursor, idle updates.
VITE_APP_BACKEND_V2_* (json.excalidraw.com) Shareable scene persistence .env.* AES-encrypted payloads; failure surfaces via ShareableLinkDialog.
Firebase (Firestore & Storage) Library sync + binary file storage .env.* + firebase-project rules firebase SDK 11.3; uses Storage JSON config for dev/prod.
VITE_APP_LIBRARY_URL / Cloud Functions Public shape libraries .env.* Controlled via whitelist; merges using mergeLibraryItems.
VITE_APP_AI_BACKEND Diagram-to-code & text-to-diagram .env.* Optional; rate limits 429; returns HTML/JSON.
Sentry (@sentry/browser) Error reporting sentry.ts Disabled in dev & Docker builds; sanitized console errors.
@excalidraw/random-username Anonymous usernames autop-run Used in collab; ensures deterministic collaborator names.
Crowdin (crowdin.yml) i18n pipeline root config Maps en.json%locale%.json for translations.
Excalidraw Plus (links & exports) Upsell & export integration VITE_APP_PLUS_* Buttons export to Plus; ExcalidrawPlusIframeExport handles iframe handshake.

Development & Deployment Notes

  • Asset Hosting: Fonts relocated into /fonts/<family>/ with Workbox caching rules; vercel.json ensures proper CORS & caching for fonts and WOFF2.
  • PWA: vite-plugin-pwa configured to ignore locale chunks; runtime caching for fonts, locales, chunk JS. PWA disabled in dev unless VITE_APP_ENABLE_PWA=true.
  • Theme handling: index.html inlines theme bootstrap script to avoid white flash; theme stored in LocalStorage, respects “system” option.
  • AI plugin UI: Renders inside plugin panel; errors returned as HTML are directly displayed — ensure backend HTML escapes content.
  • Docker: Compose mounts repository into /opt/node_app/app with delegated volume but intentionally mounts node_modules to a named volume (notused) to avoid host conflict; run yarn build:app:docker prior to compose.
  • Docs: dev-docs not part of main build; deploy separately (Vercel project expects static output in dev-docs/build).

Testing Reality & Observability

  • Unit coverage moderate; integration with external services largely unmocked (Firebase interactions bypassed in tests with fakes or AbortError).
  • DebugCanvas + visualdebug.ts support runtime visual debugging of hit-tests and bounding boxes (toggle via local storage or query params).
  • setupTests.ts intercepts console.error to highlight missing act() usage, preventing noisy logs in Vitest runs.
  • Worker pool extensively unit-tested; ensure tests trigger WorkerInTheMainChunkError when bundler misconfigured.
  • No automated end-to-end tests; manual validation relies on OSS community and Plus product pipeline.

Useful Commands & Scripts (Appendix)

# Install dependencies (clean)
yarn clean-install

# Build everything
yarn build:packages && yarn build

# Start dev server on custom port
yarn start -- --port 4000

# Run selective test suites
yarn test --run Tests/components.test.tsx

# Launch docs site
yarn --cwd dev-docs start

# Build Docker image
DOCKER_BUILDKIT=1 docker build -t excalidraw:oss .

# Run locales coverage report
node scripts/build-locales-coverage.js

# Publish prerelease (interactive)
yarn release --tag=next --non-interactive

Appendix — Reference Documents

  • Coding standards: docs/architecture/coding-standards.md
  • Tech stack primer: docs/architecture/tech-stack.md
  • Source tree guidance: docs/architecture/source-tree.md
  • Firestore/Storage rules: firebase-project/*
  • BMAD agent configuration: .bmad-core/