Skip to main content
hitspec diff compares two JSON test result files and highlights differences in pass/fail status and response times. This is useful for detecting regressions between builds, branches, or deployments.

​
Workflow

  1. Run tests and save results as JSON
  2. Make changes (deploy, merge PR, etc.)
  3. Run tests again and save new results
  4. Compare with hitspec diff
# Before changes
hitspec run tests/ --output json --output-file baseline.json

# ... make changes ...

# After changes
hitspec run tests/ --output json --output-file current.json

# Compare
hitspec diff baseline.json current.json

​
Output Formats

​
Console (default)

hitspec diff baseline.json current.json
Shows a color-coded summary with improved/regressed/unchanged counts and per-test duration changes.

​
JSON

hitspec diff baseline.json current.json --output json

​
HTML

hitspec diff baseline.json current.json --output html > diff-report.html
Generates a self-contained HTML report with a dark theme, summary cards, and a comparison table.

​
Threshold Checks

Fail the diff if any test regresses beyond a threshold: 
hitspec diff baseline.json current.json --threshold 10%
This exits with code 1 if any test’s response time increased by more than 10%. Useful as a CI/CD gate.

​
CI/CD Integration

# GitHub Actions
- name: Run baseline tests
  run: hitspec run tests/ --output json --output-file baseline.json
  continue-on-error: true

- name: Deploy staging
  run: ./deploy.sh staging

- name: Run current tests
  run: hitspec run tests/ --output json --output-file current.json

- name: Check for regressions
  run: hitspec diff baseline.json current.json --threshold 15%

​
What Gets Compared

MetricDescription
Pass/fail statusTests that changed from pass to fail (or vice versa)
DurationResponse time changes with percentage
New testsTests present only in the second file
Removed testsTests present only in the first file
A test is classified as:
  • Improved — went from fail to pass, or duration decreased by >10%
  • Regressed — went from pass to fail, or duration increased by >10%
  • Unchanged — same status and duration within 10%