jan/autoqa/MIGRATION_TESTING.md

AutoQA Migration Testing Guide

🚀 Comprehensive guide for running migration tests with AutoQA to verify data persistence across Jan application upgrades.

Table of Contents

1. Overview
2. Prerequisites
3. Basic Workflow (Base Test Cases)
4. Migration Testing
5. Migration Test Cases
6. Running Migration Tests
7. Advanced Configuration
8. Troubleshooting
9. Examples

Overview

AutoQA provides comprehensive testing capabilities for the Jan application, including:

• Base Test Cases: Standard functionality testing (assistants, models, extensions, etc.)
• Migration Testing: Verify data persistence and functionality across application upgrades
• Batch Mode: Run multiple test phases efficiently
• Screen Recording: Capture test execution for debugging
• ReportPortal Integration: Upload test results and artifacts

Prerequisites

Before running migration tests, ensure you have:

1. Python Environment: Python 3.8+ with required packages
2. Jan Installers: Both old and new version installers
3. Test Environment: Clean system or virtual machine
4. Dependencies: All AutoQA requirements installed

# Install dependencies
pip install -r requirements.txt

Basic Workflow (Base Test Cases)

Running Standard Tests

Base test cases verify core Jan functionality without version upgrades:

# Run all base tests
python main.py

# Run specific test directory
python main.py --tests-dir "tests/base"

# Run with custom configuration
python main.py \
  --max-turns 50 

Available Base Test Cases

Test Case	File	Description
Default Assistant	tests/base/default-jan-assistant.txt	Verify Jan default assistant exists
Extensions	tests/base/extensions.txt	Check available extensions
Hardware Info	tests/base/hardware-info.txt	Verify hardware information display
Model Providers	tests/base/providers-available.txt	Check model provider availability
User Chat	tests/base/user-start-chatting.txt	Test basic chat functionality
MCP Server	tests/base/enable-mcp-server.txt	Test experimental features

Migration Testing

Migration testing verifies that user data and configurations persist correctly when upgrading Jan from one version to another.

Migration Test Flow

1. Install OLD version → Run SETUP tests
2. Install NEW version → Run VERIFICATION tests
3. Compare results and verify persistence

Migration Test Approaches

Individual Mode

• Runs one test case at a time
• More granular debugging
• Better for development and troubleshooting

Batch Mode

• Runs all setup tests first, then upgrades, then all verification tests
• More realistic user experience
• Faster execution for multiple test cases

Migration Test Cases

Available Migration Test Cases

Test Case Key	Name	Description	Setup Tests	Verification Tests
models	Model Downloads Migration	Tests downloaded models persist after upgrade	models/setup-download-models.txt	models/verify-model-persistence.txt
assistants	Custom Assistants Migration	Tests custom assistants persist after upgrade	assistants/setup-create-assistants.txt	assistants/verify-create-assistant-persistence.txt
assistants-complete	Complete Assistants Migration	Tests both creation and chat functionality	Multiple setup tests	Multiple verification tests

Test Case Details

Models Migration Test

• Setup: Downloads models, configures settings, tests functionality
• Verification: Confirms models persist, settings maintained, functionality intact

Assistants Migration Test

• Setup: Creates custom assistants with specific configurations
• Verification: Confirms assistants persist with correct metadata and settings

Assistants Complete Migration Test

• Setup: Creates assistants AND tests chat functionality
• Verification: Confirms both creation and chat data persist correctly

Running Migration Tests

Basic Migration Test Command

python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "path/to/old/installer.exe" \
  --new-version "path/to/new/installer.exe" \
  --max-turns 65

Batch Mode Migration Test

python main.py \
  --enable-migration-test \
  --migration-test-case "assistants-complete" \
  --migration-batch-mode \
  --old-version "path/to/old/installer.exe" \
  --new-version "path/to/new/installer.exe" \
  --max-turns 75

Command Line Arguments

Argument	Description	Required	Example
--enable-migration-test	Enable migration testing mode	Yes	--enable-migration-test
--migration-test-case	Specific test case to run	Yes	--migration-test-case "assistants"
--migration-batch-mode	Use batch mode for multiple tests	No	--migration-batch-mode
--old-version	Path to old version installer	Yes	--old-version "C:\path\to\old.exe"
--new-version	Path to new version installer	Yes	--new-version "C:\path\to\new.exe"
--max-turns	Maximum turns per test phase	No	--max-turns 75

Environment Variables

You can also use environment variables for cleaner commands:

# Set environment variables
export OLD_VERSION="C:\path\to\old\installer.exe"
export NEW_VERSION="C:\path\to\new\installer.exe"
export MIGRATION_TEST_CASE="assistants"
export MAX_TURNS=65

# Run with environment variables
python main.py \
  --enable-migration-test \
  --migration-test-case "$MIGRATION_TEST_CASE" \
  --old-version "$OLD_VERSION" \
  --new-version "$NEW_VERSION" \
  --max-turns "$MAX_TURNS"

Advanced Configuration

Custom Model Configuration

python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "path/to/old.exe" \
  --new-version "path/to/new.exe" \
  --model-name "gpt-4" \
  --model-provider "openai" \
  --model-base-url "https://api.openai.com/v1" \
  --max-turns 80

ReportPortal Integration

python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "path/to/old.exe" \
  --new-version "path/to/new.exe" \
  --enable-reportportal \
  --rp-token "YOUR_TOKEN" \
  --rp-project "jan_migration_tests" \
  --max-turns 65

Custom Test Directory

python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "path/to/old.exe" \
  --new-version "path/to/new.exe" \
  --tests-dir "custom_tests" \
  --max-turns 65

Examples

Example 1: Basic Assistants Migration Test

# Test custom assistants persistence
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "C:\Users\ziczac computer\Downloads\Jan_0.6.6_x64-setup.exe" \
  --new-version "C:\Users\ziczac computer\Downloads\Jan_0.6.7_x64-setup.exe" \
  --max-turns 65

What this does:

1. Installs Jan 0.6.6
2. Creates custom assistants (Python Tutor, Creative Writer)
3. Upgrades to Jan 0.6.7
4. Verifies assistants persist with correct settings

Example 2: Complete Assistants Migration (Batch Mode)

# Test both creation and chat functionality
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants-complete" \
  --migration-batch-mode \
  --old-version "C:\Users\ziczac computer\Downloads\Jan_0.6.6_x64-setup.exe" \
  --new-version "C:\Users\ziczac computer\Downloads\Jan_0.6.7_x64-setup.exe" \
  --max-turns 75

What this does:

1. Installs Jan 0.6.6
2. Creates custom assistants
3. Tests chat functionality with assistants
4. Upgrades to Jan 0.6.7
5. Verifies both creation and chat data persist

Example 3: Models Migration Test

# Test model downloads and settings persistence
python main.py \
  --enable-migration-test \
  --migration-test-case "models" \
  --old-version "C:\Users\ziczac computer\Downloads\Jan_0.6.6_x64-setup.exe" \
  --new-version "C:\Users\ziczac computer\Downloads\Jan_0.6.7_x64-setup.exe" \
  --max-turns 60

What this does:

1. Installs Jan 0.6.6
2. Downloads models (jan-nano-gguf, gemma-2-2b-instruct-gguf)
3. Configures model settings
4. Upgrades to Jan 0.6.7
5. Verifies models persist and settings maintained

Troubleshooting

Common Issues

1. Installer Path Issues

# Use absolute paths with proper escaping
--old-version "C:\Users\ziczac computer\Downloads\Jan_0.6.6_x64-setup.exe"
--new-version "C:\Users\ziczac computer\Downloads\Jan_0.6.7_x64-setup.exe"

2. Turn Limit Too Low

# Increase max turns for complex tests
--max-turns 75  # Instead of default 30

3. Test Case Not Found

# Verify test case key exists
--migration-test-case "assistants"  # Valid: models, assistants, assistants-complete

4. Permission Issues

# Run as administrator on Windows
# Use sudo on Linux/macOS for system-wide installations

Debug Mode

Enable detailed logging for troubleshooting:

# Set logging level
export PYTHONPATH=.
export LOG_LEVEL=DEBUG

# Run with verbose output
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "path/to/old.exe" \
  --new-version "path/to/new.exe" \
  --max-turns 65

Test Results

Migration tests generate detailed results:

• Setup Phase Results: Success/failure for each setup test
• Upgrade Results: Installation success status
• Verification Phase Results: Success/failure for each verification test
• Overall Success: Combined result from all phases

Output Files

Tests generate several output files:

• Trajectories: trajectories/ - Agent interaction logs
• Recordings: recordings/ - Screen recordings (MP4)
• Logs: Console output with detailed execution information

Best Practices

1. Test Environment

• Use clean virtual machines or fresh system installations
• Ensure sufficient disk space for installers and test data
• Close other applications during testing

2. Test Data

• Use realistic test data (assistant names, descriptions, instructions)
• Test with multiple models and configurations
• Verify edge cases and error conditions

3. Execution

• Start with individual mode for debugging
• Use batch mode for production testing
• Monitor system resources during execution

4. Validation

• Verify test results manually when possible
• Check generated artifacts (trajectories, recordings)
• Compare expected vs. actual behavior

Next Steps

1. Start Simple: Begin with basic migration tests
2. Add Complexity: Gradually test more complex scenarios
3. Automate: Integrate into CI/CD pipelines
4. Extend: Add new test cases for specific features
5. Optimize: Refine test parameters and configurations

For more information, see the main README.md and explore the test files in the tests/ directory.