Ranex

ARBITER Test Integrity

What is ARBITER? A 3-layer system that ensures your tests actually catch bugs.

The Problem

Tests can pass even when they don't really test anything:

  • Tests that always pass (assert True)
  • Missing test coverage for requirements
  • Weak assertions that miss bugs

The 3-Layer Solution

┌────────────────────────────────────────────┐
│ Layer 1: CONTRACT                          │
│ "Are we testing everything?"               │
│                                            │
│ • Requirement-to-test mapping              │
│ • API endpoint coverage                    │
│ • 80% minimum coverage                     │
└────────────────────────────────────────────┘
                    ↓
┌────────────────────────────────────────────┐
│ Layer 2: ORACLE                            │
│ "Are our assertions meaningful?"           │
│                                            │
│ • Detects tautological assertions          │
│ • Identifies self-comparisons (x == x)     │
│ • Validates expected values                │
└────────────────────────────────────────────┘
                    ↓
┌────────────────────────────────────────────┐
│ Layer 3: MUTATION                          │
│ "Do our tests catch actual bugs?"          │
│                                            │
│ • Mutates production code                  │
│ • Runs tests against mutants               │
│ • 80% mutation score threshold             │
└────────────────────────────────────────────┘

Quick Start

# Run full ARBITER validation
ranex arbiter validate

# Run specific layer
ranex arbiter contract
ranex arbiter oracle
ranex arbiter mutation

Expected Output:

🛡️  ARBITER Test Integrity Validation
══════════════════════════════════════════════
  Target:    /path/to/project
  Threshold: 80%

📋 Layer 1: Contract Validation
   Coverage: 85%
   Status: ✅ PASS

🔮 Layer 2: Oracle Validation
   Assertions: 42 valid, 0 tautological
   Status: ✅ PASS

🧬 Layer 3: Mutation Testing
   Mutation Score: 82%
   Status: ✅ PASS

══════════════════════════════════════════════
Result: PASSED (All layers validated)

Commands

CommandDescription
ranex arbiter validateRun all 3 layers
ranex arbiter contractCheck test coverage
ranex arbiter oracleCheck assertion quality
ranex arbiter mutationRun mutation testing
ranex arbiter statusShow current validation status

Test Validation Examples

ARBITER SQL Injection test with detailed criteria validation

SQL Injection test validation - checking for parameterized queries and security patterns

ARBITER Blocking I/O test results

Blocking I/O detection - validates async patterns and proper library usage

Writing Good Tests

Add Requirement Markers

def test_user_login():
    """REQ-AUTH-001: Users can login with valid credentials"""
    result = login("user@example.com", "password123")
    assert result.success is True
    assert result.user_id is not None

Avoid Tautological Assertions

# ❌ Bad - always passes
assert True
assert x == x

# ✅ Good - tests actual behavior
assert response.status_code == 200
assert len(users) > 0

Next Steps