file-browser/README.md

# Nextcloud + Elasticsearch Discovery File Explorer

A discovery-first file explorer built with Next.js (App Router, TypeScript, Tailwind, shadcn UI) that connects to Nextcloud via WebDAV and indexes metadata/content into Elasticsearch (BM25 baseline, optional semantic hybrid search). Includes upload/download, folder tree browsing, global search, Sentry instrumentation, tags + history, and a Qdrant vector-store UI with UMAP + deck.gl visualization.

## TL;DR (Quick Start)

Prereqs:
- Node 18+ and npm
- Docker (for Elasticsearch/Kibana/Tika/Qdrant services)

Setup:
1) Configure environment
   - Copy `.env.example` to `.env.local` and fill in your values (Nextcloud creds + Elasticsearch endpoint).
   - A pre-populated `.env.local` is included for your provided Nextcloud host (update if needed).

2) Provision services (Elasticsearch, Tika, Qdrant)
   - See docs/docker/compose.samples.md for ready-to-use Compose samples and TrueNAS Scale notes.

3) Create Elasticsearch index
   - `npm run create:index`

4) Ingest Nextcloud into ES (BM25 baseline)
   - `npx tsx -r dotenv/config -r tsconfig-paths/register scripts/ingest-nextcloud.ts`

5) Run the app
   - `npm run dev` then open the reported URL (e.g., http://localhost:3000)

## Why this exists

Nextcloud is the system of record for files; this app provides a discovery UX (fast search, filters, previews later) by indexing normalized documents into Elasticsearch. Apache Tika can extract plain text for rich BM25 search, and optional OpenAI-compatible embeddings enable semantic hybrid search. Qdrant support enables browsing and visualizing collections of vectors, including UMAP-based projection and interactive selection.

## Architecture

- UI: Next.js App Router (TypeScript), Tailwind CSS, shadcn UI
- Data
  - Nextcloud WebDAV for browse/upload/download
  - Elasticsearch for indexing and search (BM25 baseline)
  - Apache Tika server for content extraction (optional but recommended)
  - Qdrant (vector database) for collections + embeddings visualization
  - OpenAI-compatible embeddings (optional) for dense vectors/hybrid queries
- Observability: Sentry (client/server/edge) with logs and spans

## Features

- Browse Nextcloud directories via a lazy-loading sidebar tree
- View files with size/type and actions in a table
- Upload files into the current folder
- Download/proxy via Next.js route
- Global search box (BM25 baseline) with optional “Semantic” toggle for hybrid search (when embeddings are configured)
- Markdown editing (CodeMirror 6) via GET/PUT content route
- Tags management: update tags and view tag history (append-only) for a file
- Optimistic UX: rename/copy/delete with optimistic updates and rollback on error
- Qdrant page: browse collections and points; UMAP + deck.gl scatter visualization with lasso selection and drill-through placeholder

## Project Layout (highlights)

- src/lib
  - env.ts: zod-validated env loader with flags
  - paths.ts: normalization/helpers for DAV paths
  - elasticsearch.ts: ES client, ensureIndex, bm25Search, knnSearch, hybridSearch, helpers
  - webdav.ts: Nextcloud WebDAV wrapper (list/create/upload/download/stat/move/copy/delete/read/write) with spans
  - embeddings.ts: OpenAI-compatible embeddings client
  - qdrant.ts: Qdrant client wrapper (list collections, scroll/list points)
- src/app/api
  - folders/list, folders/create
  - files/list, files/upload, files/download
  - files/rename (MOVE), files/copy (COPY), files/delete (DELETE)
  - files/content (GET/PUT) for Markdown/text editing
  - files/tags (POST update tags), files/tags/history (GET)
  - qdrant/collections (GET), qdrant/points (GET)
  - search/query (BM25 and optional hybrid)
- UI components/pages
  - editor/markdown-editor.tsx
  - files/file-row-actions.tsx, files/file-table.tsx, files/tags-dialog.tsx
  - qdrant/page.tsx with embedding-scatter.tsx (UMAP + deck.gl)
  - app/loading.tsx (skeletons)
- scripts
  - create-index.ts: creates ES index + alias
  - ingest-nextcloud.ts: crawl Nextcloud → optional Tika → index into ES
- docs
  - docs/elasticsearch/mappings.json: canonical baseline mapping
  - docs/docker/compose.samples.md: Compose samples & TrueNAS Scale notes
- Sentry init
  - instrumentation-client.ts, sentry.server.config.ts, sentry.edge.config.ts

## Environment Variables

Populate `.env.local` (not committed). See `.env.example` for full list.

Required:
- NEXTCLOUD_BASE_URL: e.g. https://your-nextcloud.example.com
- NEXTCLOUD_USERNAME: WebDAV user
- NEXTCLOUD_APP_PASSWORD: App password generated in Nextcloud (not login password)
- NEXTCLOUD_ROOT_PATH: e.g. `/remote.php/dav/files/admin`
- ELASTICSEARCH_URL: e.g. https://elastic.fortura.cc (your testing cluster) or http://localhost:9200
- ELASTICSEARCH_INDEX: default `files`
- ELASTICSEARCH_ALIAS: default `files_current`

Optional:
- TIKA_BASE_URL: e.g. http://localhost:9998 (if using Apache Tika for extraction)
- QDRANT_URL, QDRANT_API_KEY: for Qdrant collections/points and visualization
- SENTRY_DSN: if provided, Sentry is enabled
- OPENAI_API_BASE, OPENAI_API_KEY, OPENAI_EMBEDDING_MODEL, EMBEDDING_DIM: Enable embeddings + semantic hybrid

## Dependencies and Local Services (Docker)

See docs/docker/compose.samples.md for ready-to-use services:
- Elasticsearch (single-node) + Kibana
- Apache Tika
- Qdrant

Harden for production (auth, TLS, resource limits). Update `.env.local` to point to your services.

## Install & Run

Install dependencies:

```bash
npm install
```

Run development server:

```bash
npm run dev
# Next.js will print the URL, for example http://localhost:3000
```

Build and start:

```bash
npm run build
npm start
```

## Initialize Elasticsearch

Create the index and alias (files/files_current):

```bash
npm run create:index
# internally: tsx -r dotenv/config -r tsconfig-paths/register scripts/create-index.ts
```

To recreate:
```bash
npm run create:index -- --recreate
```

## Ingest Nextcloud → Elasticsearch

Crawl Nextcloud via WebDAV, optionally extract content with Tika, index into ES:

```bash
npx tsx -r dotenv/config -r tsconfig-paths/register scripts/ingest-nextcloud.ts
```

Restrict to a subtree:
```bash
npx tsx -r dotenv/config -r tsconfig-paths/register scripts/ingest-nextcloud.ts -- --root=/remote.php/dav/files/admin/SomeFolder
```

After ingestion, try searching in the UI (BM25).

## Qdrant: Collections and Embeddings Visualization

- Navigate to `/qdrant` to browse collections and points.
- Toggle “Vectors: on” to retrieve vectors and render the UMAP + deck.gl scatter.
- Use click to open a point (drill-through placeholder) and lasso select to select a set of points.
- Configure:
  - `QDRANT_URL` (e.g., https://vectors.biohazardvfx.com)
  - `QDRANT_API_KEY` (if required by the service)

## Tags & History

- Open the “Tags” action on a file to edit tags (comma-separated).
- Tag changes are persisted to ES (tags field) and appended as events in the `files_events` index.
- Use “History” in the dialog to see the latest tag updates for the file.

## Optional: Semantic Hybrid Search

To enable, configure embeddings:
- OPENAI_API_BASE (e.g., https://api.openai.com/v1 or your compatible endpoint)
- OPENAI_API_KEY
- OPENAI_EMBEDDING_MODEL (e.g., text-embedding-3-large)
- EMBEDDING_DIM (e.g., 1536)

Current code path supports:
- Hybrid search in API when `semantic: true` in requests; UI has a “Semantic” toggle.
- A future backfill script can be added to compute vectors for existing docs (planned).

## Sentry

- Files:
  - instrumentation-client.ts (client init)
  - sentry.server.config.ts (node runtime)
  - sentry.edge.config.ts (edge runtime)
- Enable by setting SENTRY_DSN in `.env.local`.
- Logging enabled (`consoleLoggingIntegration`), capturing console.log/warn/error.
- Spans instrument WebDAV and ES operations with meaningful op/name and attributes.

## API Endpoints

- GET `/api/folders/list?path=/abs/path` → { folders[], files[] }
- POST `/api/folders/create` body: `{ path: "/abs/path" }` → `{ ok: true }`
- GET `/api/files/list?path=/abs/path&page=1&perPage=50` → `{ total, page, perPage, items[] }`
- POST `/api/files/upload` (multipart) form-data: `file`, `destPath`
- GET `/api/files/download?path=/abs/path` → stream download
- POST `/api/files/rename` → `{ from, to }`
- POST `/api/files/copy` → `{ from, to }`
- POST `/api/files/delete` → `{ path }`
- GET/PUT `/api/files/content` → `{ path }` and `{ path, content, mimeType? }`
- POST `/api/files/tags` → `{ path, tags[] }`
- GET `/api/files/tags/history?path=...` → history events
- GET `/api/qdrant/collections` → Qdrant collections
- GET `/api/qdrant/points?collection=...` → Qdrant points (with/without vectors)
- POST `/api/search/query` → body: `{ q, filters?, sort?, page?, perPage?, semantic? }`

## UI Usage

- Navigate folders via the left sidebar. Root defaults to `NEXTCLOUD_ROOT_PATH`.
- Breadcrumbs show the current path; click to navigate.
- Global search queries ES (BM25). Toggle “Semantic” to blend vector similarity (when enabled).
- Use “Upload” to send a file to the current folder.
- Use file actions (row menu) to Edit content, Download, Rename/Move, Copy, Delete, and Tags.
- Open `/qdrant` for vector collections; toggle vectors to visualize embeddings.

## Testing

- Unit tests (Vitest):
  - `npm run test` or `npm run test:unit`
  - Example: tests/unit/tags-dialog.test.ts
- E2E (Playwright):
  - `npm run test:e2e` (requires `npx playwright install` first)
- Integration tests: planned for API route handlers with mocks.

## Known Caveats / TODO

- Breadcrumb segments may include technical path prefixes if navigating above the user root (optional improvement).
- Embeddings backfill script & “Find similar” API/UI are planned.
- deck.gl typing simplifications used for portability; visualization is functional but can be enhanced with richer interactions.
- Tests (integration/E2E) and additional UX polish can be expanded.

## Development Notes

- Code style: TypeScript, ESLint (Next config), Tailwind v4.
- shadcn components added (Slate), most primitives included.
- ES Types via `@elastic/elasticsearch` estypes.
- Stream interop handled for Node → Web streams in downloads and uploads.
- Zod-based env loader normalizes URLs and validates required vars.

## Troubleshooting

- Search 500s:
  - Ensure Elasticsearch is running and reachable at `ELASTICSEARCH_URL`.
  - Run `npm run create:index` to create the index and alias.
  - If using Tika for content, ensure `TIKA_BASE_URL` is set and service is healthy (optional).
- WebDAV failures:
  - Verify `NEXTCLOUD_BASE_URL`, `NEXTCLOUD_USERNAME`, `NEXTCLOUD_APP_PASSWORD`, and `NEXTCLOUD_ROOT_PATH`.
  - Confirm the user has permission for the target path.
- Qdrant:
  - Ensure `QDRANT_URL` and `QDRANT_API_KEY` (if required) are set.
  - Toggle “Vectors: on” to include vectors in point queries for visualization.
- Sentry issues:
  - Ensure `SENTRY_DSN` is set; check networking/outbound restrictions.

## Scripts

- `npm run create:index`: Create/recreate Elasticsearch index and alias
- `scripts/ingest-nextcloud.ts`: Crawl Nextcloud → Tika (optional) → Elasticsearch
- `npm run test`: Run unit tests (Vitest)
- `npm run test:e2e`: Run Playwright E2E tests

## Security

- Keep `.env.local` out of source control (already in .gitignore).
- Use Nextcloud App Passwords, not login passwords.
- Harden Elasticsearch/Kibana/Tika/Qdrant for production (auth, TLS, resource limits).

---

Built per implementation_plan.md with a deterministic order of rollout and Sentry instrumentation aligned to organizational rules.