jan/autoqa/QUICK_START.md

AutoQA Quick Start Guide

🚀 Get started with AutoQA in minutes - from basic testing to migration verification.

Quick Start

1. Install Dependencies

# Install required packages
pip install -r requirements.txt

2. Basic Testing (No Migration)

# Run all base tests
python main.py

# Run specific test category
python main.py --tests-dir "tests/base"

# Custom configuration
python main.py \
  --max-turns 50

3. Migration Testing

# Basic migration test
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "C:\path\to\old\installer.exe" \
  --new-version "C:\path\to\new\installer.exe" \
  --max-turns 65

# Batch mode migration test
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants-complete" \
  --migration-batch-mode \
  --old-version "C:\path\to\old\installer.exe" \
  --new-version "C:\path\to\new\installer.exe" \
  --max-turns 75

4. Reliability Testing

# Development phase (5 runs)
python main.py \
  --enable-reliability-test \
  --reliability-phase development \
  --max-turns 40

# Deployment phase (20 runs)
python main.py \
  --enable-reliability-test \
  --reliability-phase deployment \
  --max-turns 40

# Custom number of runs
python main.py \
  --enable-reliability-test \
  --reliability-runs 10 \
  --max-turns 40

Test Types

Base Test Cases

• Default Assistant: Verify Jan default assistant exists
• Extensions: Check available extensions
• Hardware Info: Verify hardware information display
• Model Providers: Check model provider availability
• User Chat: Test basic chat functionality
• MCP Server: Test experimental features

Migration Test Cases

• models: Test downloaded models persist after upgrade
• assistants: Test custom assistants persist after upgrade
• assistants-complete: Test both creation and chat functionality

Reliability Testing

• Development Phase: Run test 5 times to verify basic stability (≥80% success rate)
• Deployment Phase: Run test 20 times to verify production readiness (≥90% success rate)
• Custom Runs: Specify custom number of runs for specific testing needs

Common Commands

Basic Workflow

# Run all tests
python main.py

# Run with ReportPortal
python main.py --enable-reportportal --rp-token "YOUR_TOKEN"

# Custom test directory
python main.py --tests-dir "my_tests"

# Skip computer server auto-start
python main.py --skip-server-start

Migration Workflow

# Test assistants migration
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "path/to/old.exe" \
  --new-version "path/to/new.exe"

# Test models migration
python main.py \
  --enable-migration-test \
  --migration-test-case "models" \
  --old-version "path/to/old.exe" \
  --new-version "path/to/new.exe"

# Test complete assistants migration (batch mode)
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants-complete" \
  --migration-batch-mode \
  --old-version "path/to/old.exe" \
  --new-version "path/to/new.exe"

# Test reliability - development phase
python main.py \
  --enable-reliability-test \
  --reliability-phase development \
  --max-turns 40

# Test reliability - deployment phase
python main.py \
  --enable-reliability-test \
  --reliability-phase deployment \
  --max-turns 40

## Configuration Options

### Essential Arguments
| Argument | Description | Default |
|----------|-------------|---------|
| `--max-turns` | Maximum turns per test | 30 |
| `--tests-dir` | Test files directory | `tests` |
| `--jan-app-path` | Jan executable path | auto-detected |
| `--model-name` | AI model name | UI-TARS-1.5-7B |

### Migration Arguments
| Argument | Description | Required |
|----------|-------------|----------|
| `--enable-migration-test` | Enable migration mode | Yes |
| `--migration-test-case` | Test case to run | Yes |
| `--migration-batch-mode` | Use batch mode | No |
| `--old-version` | Old installer path | Yes |
| `--new-version` | New installer path | Yes |

### ReportPortal Arguments
| Argument | Description | Required |
|----------|-------------|----------|
| `--enable-reportportal` | Enable RP integration | No |
| `--rp-token` | ReportPortal token | Yes (if RP enabled) |
| `--rp-endpoint` | RP endpoint URL | No |
| `--rp-project` | RP project name | No |

### Reliability Testing Arguments
| Argument | Description | Required |
|----------|-------------|----------|
| `--enable-reliability-test` | Enable reliability mode | Yes |
| `--reliability-phase` | Testing phase (development/deployment) | No |
| `--reliability-runs` | Custom number of runs | No |
| `--reliability-test-path` | Specific test file path | No |

## Environment Variables

```bash
# Set common variables
export MAX_TURNS=65
export MODEL_NAME="gpt-4"
export MODEL_BASE_URL="https://api.openai.com/v1"
export JAN_APP_PATH="C:\path\to\Jan.exe"

# Use in commands
python main.py --max-turns "$MAX_TURNS"

Examples

Example 1: Basic Testing

# Test core functionality
python main.py \
  --max-turns 40 \
  --tests-dir "tests/base"

Example 2: Simple Migration

# Test assistants persistence
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "Jan_0.6.6.exe" \
  --new-version "Jan_0.6.7.exe" \
  --max-turns 65

Example 3: Advanced Migration

# Test complete functionality with ReportPortal
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants-complete" \
  --migration-batch-mode \
  --old-version "Jan_0.6.6.exe" \
  --new-version "Jan_0.6.7.exe" \
  --max-turns 75 \
  --enable-reportportal \
  --rp-token "YOUR_TOKEN" \
  --rp-project "jan_migration_tests"

Example 4: Reliability Testing

# Test reliability with deployment phase
python main.py \
  --enable-reliability-test \
  --reliability-phase deployment \
  --reliability-test-path "tests/base/default-jan-assistant.txt" \
  --max-turns 50 \
  --enable-reportportal \
  --rp-token "YOUR_TOKEN" \
  --rp-project "jan_reliability_tests"

Troubleshooting

Common Issues

1. Path Issues: Use absolute paths with proper escaping
2. Turn Limits: Increase --max-turns for complex tests
3. Permissions: Run as administrator on Windows
4. Dependencies: Ensure all packages are installed

Debug Mode

# Enable verbose logging
export LOG_LEVEL=DEBUG
export PYTHONPATH=.

# Run with debug output
python main.py --enable-migration-test ...

Output Files

• Trajectories: trajectories/ - Agent interaction logs
• Recordings: recordings/ - Screen recordings (MP4)
• Console: Detailed execution logs

Next Steps

1. Start Simple: Run basic tests first
2. Add Migration: Test data persistence
3. Customize: Adjust parameters for your needs
4. Integrate: Add to CI/CD pipelines

For detailed documentation, see MIGRATION_TESTING.md and README.md.