obsidian-qdrant/README.md

Qdrant Semantic Search for Obsidian

A powerful Obsidian plugin that indexes your entire vault into Qdrant for semantic search, using Ollama or OpenAI for text embeddings, with support for PDF and image text extraction via Text Extractor plugin.

Features

• Semantic Search: Find content by meaning, not just keywords
• Multiple Embedding Providers:
  • Ollama (local, free) - default
  • OpenAI (cloud, paid)
• Rich Content Support:
  • Markdown files with frontmatter parsing
  • Code files with syntax highlighting
  • PDFs via Text Extractor plugin
  • Images with OCR via Text Extractor plugin
• Hybrid Chunking: Smart text splitting on headings with size-based fallback
• Real-time Indexing: Automatic indexing of file changes
• Graph Visualization: View document relationships (planned)
• Comprehensive Settings: Full control over indexing and search behavior

Installation

Prerequisites

1. Qdrant: You need a Qdrant instance running

   • Local: docker run -p 6333:6333 qdrant/qdrant
   • Cloud: Sign up at Qdrant Cloud

2. Ollama (recommended for local embeddings):

   # Install Ollama
   curl -fsSL https://ollama.ai/install.sh | sh
   
   # Pull an embedding model
   ollama pull nomic-embed-text
   

3. Text Extractor Plugin (optional, for PDF/image support):

   • Install from Community Plugins
   • Enables PDF text extraction and image OCR

Plugin Installation

1. Download the latest release from GitHub
2. Extract main.js, manifest.json, and styles.css to your vault's .obsidian/plugins/obsidian-qdrant/ folder
3. Enable the plugin in Settings → Community plugins

Configuration

Basic Setup

1. Open Settings → Community plugins → Qdrant Semantic Search
2. Configure your Qdrant connection:
   • URL: http://localhost:6333 (local) or your Qdrant Cloud URL
   • API Key: Leave empty for local, add your key for cloud
3. Choose your embedding provider:
   • Ollama: Set model name (e.g., nomic-embed-text)
   • OpenAI: Add your API key and select model

Advanced Settings

Indexing Configuration

• Include Patterns: File types to index (default: *.md, *.txt, *.pdf, *.png, *.jpg)
• Exclude Patterns: File patterns to skip
• Max File Size: Skip files larger than this (default: 10MB)
• Ignored Folders: Folders to skip (default: .obsidian, .git, node_modules)

Chunking Settings

• Target Tokens: Ideal chunk size (default: 500)
• Overlap Tokens: Overlap between chunks (default: 100)
• Max Tokens: Hard limit per chunk (default: 800)

Graph Visualization

• Enable Graph View: Show document relationships
• Similarity Threshold: Minimum similarity for edges (default: 0.7)
• Max Nodes: Maximum nodes to display (default: 100)

Usage

Commands

• Semantic search: Open the search modal
• Index current file: Index the currently open file
• Full reindex vault: Reindex all files
• Clear index: Remove all indexed data
• Open graph view: Show document relationships (when implemented)

Search Interface

1. Use Ctrl+P (or Cmd+P on Mac) to open Command Palette
2. Type "Semantic search" and press Enter
3. Enter your search query
4. Browse results with keyboard navigation:
   • Arrow keys: Navigate results
   • Enter: Open selected result
   • Escape: Close search

Status Bar

The plugin shows indexing progress in the status bar:

• Ready: System is ready
• Indexing X%: Shows progress during full reindex
• Error: Click to see error details

Architecture

Components

• Extractors: Parse different file types (markdown, code, PDFs, images)
• Chunkers: Split text into semantic chunks
• Embedding Providers: Generate vector embeddings
• Qdrant Client: Store and search vectors
• Indexing Queue: Manage background indexing
• Search UI: Provide search interface

Data Flow

1. File Change → File Watcher → Indexing Queue
2. Extract → Chunk → Embed → Store in Qdrant
3. Search Query → Embed → Search Qdrant → Display Results

Collection Schema

Each vault gets a collection named vault_<sanitized_name>_<model>. Points contain:

• Vector: Embedding from your chosen model
• Payload: Rich metadata (path, title, tags, chunk info, etc.)

Troubleshooting

Common Issues

"Indexing system not ready"

• Check Qdrant connection in settings
• Verify embedding provider configuration
• Check console for error messages

"No results found"

• Ensure files are indexed (check status bar)
• Try a full reindex
• Verify your search query isn't too specific

"Ollama connection failed"

• Ensure Ollama is running: ollama serve
• Check model is installed: ollama list
• Verify URL in settings (default: http://localhost:11434)

"OpenAI connection failed"

• Verify API key is correct
• Check you have credits/quota
• Ensure model name is valid

Performance Tips

1. Batch Size: Increase for faster indexing (if you have memory)
2. Concurrency: Higher values for faster processing (but may overwhelm services)
3. File Filters: Exclude unnecessary files to speed up indexing
4. Chunk Size: Larger chunks = fewer vectors but less precise search

Debugging

Enable developer console (Ctrl+Shift+I) to see detailed logs:

• Indexing progress and errors
• Search query processing
• Qdrant API calls
• Embedding generation

Development

Building from Source

git clone <repository>
cd obsidian-qdrant
npm install
npm run dev    # Watch mode
npm run build  # Production build

Project Structure

src/
├── types.ts              # TypeScript interfaces
├── settings.ts           # Settings and defaults
├── main.ts              # Plugin entry point
├── qdrant/              # Qdrant client and collection management
├── embeddings/          # Embedding providers (Ollama, OpenAI)
├── extractors/          # Content extractors
├── chunking/            # Text chunking logic
├── indexing/            # Indexing orchestration
├── search/              # Search UI components
├── graph/               # Graph visualization
└── ui/                  # Settings UI

Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Add tests if applicable
5. Submit a pull request

License

MIT License - see LICENSE file for details.

Acknowledgments

• Qdrant for the vector database
• Ollama for local embeddings
• OpenAI for cloud embeddings
• Text Extractor for PDF/image support
• Obsidian for the amazing note-taking platform

Roadmap

• Graph visualization with D3.js
• Hybrid search (dense + sparse vectors)
• More embedding providers (Cohere, Mistral, etc.)
• Advanced filtering in search
• Search result ranking improvements
• Mobile support
• Plugin API for other plugins
• Export/import index data
• Search analytics and insights