371 lines
11 KiB
Markdown
371 lines
11 KiB
Markdown
# AutoQA Migration Testing Guide
|
|
|
|
🚀 Comprehensive guide for running migration tests with AutoQA to verify data persistence across Jan application upgrades.
|
|
|
|
## Table of Contents
|
|
|
|
1. [Overview](#overview)
|
|
2. [Prerequisites](#prerequisites)
|
|
3. [Basic Workflow (Base Test Cases)](#basic-workflow-base-test-cases)
|
|
4. [Migration Testing](#migration-testing)
|
|
5. [Migration Test Cases](#migration-test-cases)
|
|
6. [Running Migration Tests](#running-migration-tests)
|
|
7. [Advanced Configuration](#advanced-configuration)
|
|
8. [Troubleshooting](#troubleshooting)
|
|
9. [Examples](#examples)
|
|
|
|
## Overview
|
|
|
|
AutoQA provides comprehensive testing capabilities for the Jan application, including:
|
|
|
|
- **Base Test Cases**: Standard functionality testing (assistants, models, extensions, etc.)
|
|
- **Migration Testing**: Verify data persistence and functionality across application upgrades
|
|
- **Batch Mode**: Run multiple test phases efficiently
|
|
- **Screen Recording**: Capture test execution for debugging
|
|
- **ReportPortal Integration**: Upload test results and artifacts
|
|
|
|
## Prerequisites
|
|
|
|
Before running migration tests, ensure you have:
|
|
|
|
1. **Python Environment**: Python 3.8+ with required packages
|
|
2. **Jan Installers**: Both old and new version installers
|
|
3. **Test Environment**: Clean system or virtual machine
|
|
4. **Dependencies**: All AutoQA requirements installed
|
|
|
|
```bash
|
|
# Install dependencies
|
|
pip install -r requirements.txt
|
|
|
|
```
|
|
|
|
## Basic Workflow (Base Test Cases)
|
|
|
|
### Running Standard Tests
|
|
|
|
Base test cases verify core Jan functionality without version upgrades:
|
|
|
|
```bash
|
|
# Run all base tests
|
|
python main.py
|
|
|
|
# Run specific test directory
|
|
python main.py --tests-dir "tests/base"
|
|
|
|
# Run with custom configuration
|
|
python main.py \
|
|
--max-turns 50
|
|
```
|
|
|
|
### Available Base Test Cases
|
|
|
|
| Test Case | File | Description |
|
|
|-----------|------|-------------|
|
|
| Default Assistant | `tests/base/default-jan-assistant.txt` | Verify Jan default assistant exists |
|
|
| Extensions | `tests/base/extensions.txt` | Check available extensions |
|
|
| Hardware Info | `tests/base/hardware-info.txt` | Verify hardware information display |
|
|
| Model Providers | `tests/base/providers-available.txt` | Check model provider availability |
|
|
| User Chat | `tests/base/user-start-chatting.txt` | Test basic chat functionality |
|
|
| MCP Server | `tests/base/enable-mcp-server.txt` | Test experimental features |
|
|
|
|
## Migration Testing
|
|
|
|
Migration testing verifies that user data and configurations persist correctly when upgrading Jan from one version to another.
|
|
|
|
### Migration Test Flow
|
|
|
|
```
|
|
1. Install OLD version → Run SETUP tests
|
|
2. Install NEW version → Run VERIFICATION tests
|
|
3. Compare results and verify persistence
|
|
```
|
|
|
|
### Migration Test Approaches
|
|
|
|
#### Individual Mode
|
|
- Runs one test case at a time
|
|
- More granular debugging
|
|
- Better for development and troubleshooting
|
|
|
|
#### Batch Mode
|
|
- Runs all setup tests first, then upgrades, then all verification tests
|
|
- More realistic user experience
|
|
- Faster execution for multiple test cases
|
|
|
|
## Migration Test Cases
|
|
|
|
### Available Migration Test Cases
|
|
|
|
| Test Case Key | Name | Description | Setup Tests | Verification Tests |
|
|
|---------------|------|-------------|-------------|-------------------|
|
|
| `models` | Model Downloads Migration | Tests downloaded models persist after upgrade | `models/setup-download-models.txt` | `models/verify-model-persistence.txt` |
|
|
| `assistants` | Custom Assistants Migration | Tests custom assistants persist after upgrade | `assistants/setup-create-assistants.txt` | `assistants/verify-create-assistant-persistence.txt` |
|
|
| `assistants-complete` | Complete Assistants Migration | Tests both creation and chat functionality | Multiple setup tests | Multiple verification tests |
|
|
|
|
### Test Case Details
|
|
|
|
#### Models Migration Test
|
|
- **Setup**: Downloads models, configures settings, tests functionality
|
|
- **Verification**: Confirms models persist, settings maintained, functionality intact
|
|
|
|
#### Assistants Migration Test
|
|
- **Setup**: Creates custom assistants with specific configurations
|
|
- **Verification**: Confirms assistants persist with correct metadata and settings
|
|
|
|
#### Assistants Complete Migration Test
|
|
- **Setup**: Creates assistants AND tests chat functionality
|
|
- **Verification**: Confirms both creation and chat data persist correctly
|
|
|
|
## Running Migration Tests
|
|
|
|
### Basic Migration Test Command
|
|
|
|
```bash
|
|
python main.py \
|
|
--enable-migration-test \
|
|
--migration-test-case "assistants" \
|
|
--old-version "path/to/old/installer.exe" \
|
|
--new-version "path/to/new/installer.exe" \
|
|
--max-turns 65
|
|
```
|
|
|
|
### Batch Mode Migration Test
|
|
|
|
```bash
|
|
python main.py \
|
|
--enable-migration-test \
|
|
--migration-test-case "assistants-complete" \
|
|
--migration-batch-mode \
|
|
--old-version "path/to/old/installer.exe" \
|
|
--new-version "path/to/new/installer.exe" \
|
|
--max-turns 75
|
|
```
|
|
|
|
### Command Line Arguments
|
|
|
|
| Argument | Description | Required | Example |
|
|
|----------|-------------|----------|---------|
|
|
| `--enable-migration-test` | Enable migration testing mode | Yes | `--enable-migration-test` |
|
|
| `--migration-test-case` | Specific test case to run | Yes | `--migration-test-case "assistants"` |
|
|
| `--migration-batch-mode` | Use batch mode for multiple tests | No | `--migration-batch-mode` |
|
|
| `--old-version` | Path to old version installer | Yes | `--old-version "C:\path\to\old.exe"` |
|
|
| `--new-version` | Path to new version installer | Yes | `--new-version "C:\path\to\new.exe"` |
|
|
| `--max-turns` | Maximum turns per test phase | No | `--max-turns 75` |
|
|
|
|
### Environment Variables
|
|
|
|
You can also use environment variables for cleaner commands:
|
|
|
|
```bash
|
|
# Set environment variables
|
|
export OLD_VERSION="C:\path\to\old\installer.exe"
|
|
export NEW_VERSION="C:\path\to\new\installer.exe"
|
|
export MIGRATION_TEST_CASE="assistants"
|
|
export MAX_TURNS=65
|
|
|
|
# Run with environment variables
|
|
python main.py \
|
|
--enable-migration-test \
|
|
--migration-test-case "$MIGRATION_TEST_CASE" \
|
|
--old-version "$OLD_VERSION" \
|
|
--new-version "$NEW_VERSION" \
|
|
--max-turns "$MAX_TURNS"
|
|
```
|
|
|
|
## Advanced Configuration
|
|
|
|
### Custom Model Configuration
|
|
|
|
```bash
|
|
python main.py \
|
|
--enable-migration-test \
|
|
--migration-test-case "assistants" \
|
|
--old-version "path/to/old.exe" \
|
|
--new-version "path/to/new.exe" \
|
|
--model-name "gpt-4" \
|
|
--model-provider "openai" \
|
|
--model-base-url "https://api.openai.com/v1" \
|
|
--max-turns 80
|
|
```
|
|
|
|
### ReportPortal Integration
|
|
|
|
```bash
|
|
python main.py \
|
|
--enable-migration-test \
|
|
--migration-test-case "assistants" \
|
|
--old-version "path/to/old.exe" \
|
|
--new-version "path/to/new.exe" \
|
|
--enable-reportportal \
|
|
--rp-token "YOUR_TOKEN" \
|
|
--rp-project "jan_migration_tests" \
|
|
--max-turns 65
|
|
```
|
|
|
|
### Custom Test Directory
|
|
|
|
```bash
|
|
python main.py \
|
|
--enable-migration-test \
|
|
--migration-test-case "assistants" \
|
|
--old-version "path/to/old.exe" \
|
|
--new-version "path/to/new.exe" \
|
|
--tests-dir "custom_tests" \
|
|
--max-turns 65
|
|
```
|
|
|
|
## Examples
|
|
|
|
### Example 1: Basic Assistants Migration Test
|
|
|
|
```bash
|
|
# Test custom assistants persistence
|
|
python main.py \
|
|
--enable-migration-test \
|
|
--migration-test-case "assistants" \
|
|
--old-version "C:\Users\ziczac computer\Downloads\Jan_0.6.6_x64-setup.exe" \
|
|
--new-version "C:\Users\ziczac computer\Downloads\Jan_0.6.7_x64-setup.exe" \
|
|
--max-turns 65
|
|
```
|
|
|
|
**What this does:**
|
|
1. Installs Jan 0.6.6
|
|
2. Creates custom assistants (Python Tutor, Creative Writer)
|
|
3. Upgrades to Jan 0.6.7
|
|
4. Verifies assistants persist with correct settings
|
|
|
|
### Example 2: Complete Assistants Migration (Batch Mode)
|
|
|
|
```bash
|
|
# Test both creation and chat functionality
|
|
python main.py \
|
|
--enable-migration-test \
|
|
--migration-test-case "assistants-complete" \
|
|
--migration-batch-mode \
|
|
--old-version "C:\Users\ziczac computer\Downloads\Jan_0.6.6_x64-setup.exe" \
|
|
--new-version "C:\Users\ziczac computer\Downloads\Jan_0.6.7_x64-setup.exe" \
|
|
--max-turns 75
|
|
```
|
|
|
|
**What this does:**
|
|
1. Installs Jan 0.6.6
|
|
2. Creates custom assistants
|
|
3. Tests chat functionality with assistants
|
|
4. Upgrades to Jan 0.6.7
|
|
5. Verifies both creation and chat data persist
|
|
|
|
### Example 3: Models Migration Test
|
|
|
|
```bash
|
|
# Test model downloads and settings persistence
|
|
python main.py \
|
|
--enable-migration-test \
|
|
--migration-test-case "models" \
|
|
--old-version "C:\Users\ziczac computer\Downloads\Jan_0.6.6_x64-setup.exe" \
|
|
--new-version "C:\Users\ziczac computer\Downloads\Jan_0.6.7_x64-setup.exe" \
|
|
--max-turns 60
|
|
```
|
|
|
|
**What this does:**
|
|
1. Installs Jan 0.6.6
|
|
2. Downloads models (jan-nano-gguf, gemma-2-2b-instruct-gguf)
|
|
3. Configures model settings
|
|
4. Upgrades to Jan 0.6.7
|
|
5. Verifies models persist and settings maintained
|
|
|
|
## Troubleshooting
|
|
|
|
### Common Issues
|
|
|
|
#### 1. Installer Path Issues
|
|
```bash
|
|
# Use absolute paths with proper escaping
|
|
--old-version "C:\Users\ziczac computer\Downloads\Jan_0.6.6_x64-setup.exe"
|
|
--new-version "C:\Users\ziczac computer\Downloads\Jan_0.6.7_x64-setup.exe"
|
|
```
|
|
|
|
#### 2. Turn Limit Too Low
|
|
```bash
|
|
# Increase max turns for complex tests
|
|
--max-turns 75 # Instead of default 30
|
|
```
|
|
|
|
#### 3. Test Case Not Found
|
|
```bash
|
|
# Verify test case key exists
|
|
--migration-test-case "assistants" # Valid: models, assistants, assistants-complete
|
|
```
|
|
|
|
#### 4. Permission Issues
|
|
```bash
|
|
# Run as administrator on Windows
|
|
# Use sudo on Linux/macOS for system-wide installations
|
|
```
|
|
|
|
### Debug Mode
|
|
|
|
Enable detailed logging for troubleshooting:
|
|
|
|
```bash
|
|
# Set logging level
|
|
export PYTHONPATH=.
|
|
export LOG_LEVEL=DEBUG
|
|
|
|
# Run with verbose output
|
|
python main.py \
|
|
--enable-migration-test \
|
|
--migration-test-case "assistants" \
|
|
--old-version "path/to/old.exe" \
|
|
--new-version "path/to/new.exe" \
|
|
--max-turns 65
|
|
```
|
|
|
|
### Test Results
|
|
|
|
Migration tests generate detailed results:
|
|
|
|
- **Setup Phase Results**: Success/failure for each setup test
|
|
- **Upgrade Results**: Installation success status
|
|
- **Verification Phase Results**: Success/failure for each verification test
|
|
- **Overall Success**: Combined result from all phases
|
|
|
|
### Output Files
|
|
|
|
Tests generate several output files:
|
|
|
|
- **Trajectories**: `trajectories/` - Agent interaction logs
|
|
- **Recordings**: `recordings/` - Screen recordings (MP4)
|
|
- **Logs**: Console output with detailed execution information
|
|
|
|
## Best Practices
|
|
|
|
### 1. Test Environment
|
|
- Use clean virtual machines or fresh system installations
|
|
- Ensure sufficient disk space for installers and test data
|
|
- Close other applications during testing
|
|
|
|
### 2. Test Data
|
|
- Use realistic test data (assistant names, descriptions, instructions)
|
|
- Test with multiple models and configurations
|
|
- Verify edge cases and error conditions
|
|
|
|
### 3. Execution
|
|
- Start with individual mode for debugging
|
|
- Use batch mode for production testing
|
|
- Monitor system resources during execution
|
|
|
|
### 4. Validation
|
|
- Verify test results manually when possible
|
|
- Check generated artifacts (trajectories, recordings)
|
|
- Compare expected vs. actual behavior
|
|
|
|
## Next Steps
|
|
|
|
1. **Start Simple**: Begin with basic migration tests
|
|
2. **Add Complexity**: Gradually test more complex scenarios
|
|
3. **Automate**: Integrate into CI/CD pipelines
|
|
4. **Extend**: Add new test cases for specific features
|
|
5. **Optimize**: Refine test parameters and configurations
|
|
|
|
For more information, see the main [README.md](README.md) and explore the test files in the `tests/` directory.
|