jan/autoqa/QUICK_START.md

AutoQA Quick Start Guide

🚀 Get started with AutoQA in minutes - from basic testing to migration verification.

Quick Start

1. Install Dependencies

# Install required packages
pip install -r requirements.txt

2. Basic Testing (No Migration)

# Run all base tests
python main.py

# Run specific test category
python main.py --tests-dir "tests/base"

# Custom configuration
python main.py \
  --max-turns 50

3. Migration Testing

# Basic migration test
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "C:\path\to\old\installer.exe" \
  --new-version "C:\path\to\new\installer.exe" \
  --max-turns 65

# Batch mode migration test
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants-complete" \
  --migration-batch-mode \
  --old-version "C:\path\to\old\installer.exe" \
  --new-version "C:\path\to\new\installer.exe" \
  --max-turns 75

Test Types

Base Test Cases

• Default Assistant: Verify Jan default assistant exists
• Extensions: Check available extensions
• Hardware Info: Verify hardware information display
• Model Providers: Check model provider availability
• User Chat: Test basic chat functionality
• MCP Server: Test experimental features

Migration Test Cases

• models: Test downloaded models persist after upgrade
• assistants: Test custom assistants persist after upgrade
• assistants-complete: Test both creation and chat functionality

Common Commands

Basic Workflow

# Run all tests
python main.py

# Run with ReportPortal
python main.py --enable-reportportal --rp-token "YOUR_TOKEN"

# Custom test directory
python main.py --tests-dir "my_tests"

# Skip computer server auto-start
python main.py --skip-server-start

Migration Workflow

# Test assistants migration
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "path/to/old.exe" \
  --new-version "path/to/new.exe"

# Test models migration
python main.py \
  --enable-migration-test \
  --migration-test-case "models" \
  --old-version "path/to/old.exe" \
  --new-version "path/to/new.exe"

# Test complete assistants migration (batch mode)
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants-complete" \
  --migration-batch-mode \
  --old-version "path/to/old.exe" \
  --new-version "path/to/new.exe"

Configuration Options

Essential Arguments

Argument	Description	Default
--max-turns	Maximum turns per test	30
--tests-dir	Test files directory	tests
--jan-app-path	Jan executable path	auto-detected
--model-name	AI model name	UI-TARS-1.5-7B

Migration Arguments

Argument	Description	Required
--enable-migration-test	Enable migration mode	Yes
--migration-test-case	Test case to run	Yes
--migration-batch-mode	Use batch mode	No
--old-version	Old installer path	Yes
--new-version	New installer path	Yes

ReportPortal Arguments

Argument	Description	Required
--enable-reportportal	Enable RP integration	No
--rp-token	ReportPortal token	Yes (if RP enabled)
--rp-endpoint	RP endpoint URL	No
--rp-project	RP project name	No

Environment Variables

# Set common variables
export MAX_TURNS=65
export MODEL_NAME="gpt-4"
export MODEL_BASE_URL="https://api.openai.com/v1"
export JAN_APP_PATH="C:\path\to\Jan.exe"

# Use in commands
python main.py --max-turns "$MAX_TURNS"

Examples

Example 1: Basic Testing

# Test core functionality
python main.py \
  --max-turns 40 \
  --tests-dir "tests/base"

Example 2: Simple Migration

# Test assistants persistence
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants" \
  --old-version "Jan_0.6.6.exe" \
  --new-version "Jan_0.6.7.exe" \
  --max-turns 65

Example 3: Advanced Migration

# Test complete functionality with ReportPortal
python main.py \
  --enable-migration-test \
  --migration-test-case "assistants-complete" \
  --migration-batch-mode \
  --old-version "Jan_0.6.6.exe" \
  --new-version "Jan_0.6.7.exe" \
  --max-turns 75 \
  --enable-reportportal \
  --rp-token "YOUR_TOKEN" \
  --rp-project "jan_migration_tests"

Troubleshooting

Common Issues

1. Path Issues: Use absolute paths with proper escaping
2. Turn Limits: Increase --max-turns for complex tests
3. Permissions: Run as administrator on Windows
4. Dependencies: Ensure all packages are installed

Debug Mode

# Enable verbose logging
export LOG_LEVEL=DEBUG
export PYTHONPATH=.

# Run with debug output
python main.py --enable-migration-test ...

Output Files

• Trajectories: trajectories/ - Agent interaction logs
• Recordings: recordings/ - Screen recordings (MP4)
• Console: Detailed execution logs

Next Steps

1. Start Simple: Run basic tests first
2. Add Migration: Test data persistence
3. Customize: Adjust parameters for your needs
4. Integrate: Add to CI/CD pipelines

For detailed documentation, see MIGRATION_TESTING.md and README.md.