Interactive Benchmarks - Claude Documentation

This document provides mappings and instructions for updating benchmark data from execution results.

Quick Reference: Updating Benchmarks

# Regenerate benchmark_data.json after any changes
python3 scripts/update_benchmarks.py

Key Files:

• scripts/update_benchmarks.py - Main generator script
• public/data/benchmarks/benchmark_data.json - Generated output (DO NOT EDIT)
• src/components/InteractiveBenchmark.astro - Display component

Naming Convention

IMPORTANT: Always use “AADC Python” (not “Python AADC”)

Actual vs Estimated Values

The system tracks whether data is actual (from execution log) or estimated (extrapolated):

Display Rules

1. Performance Comparison table:

   • Estimated values show * Estimate marker
   • Estimated rows have yellow left border
   • Hover shows source configuration (e.g., ”📊 10×100K available”)

2. Key Findings section:

   • Only actual values shown for apples-to-apples comparisons
   • Estimate count shown: “X actual, Y estimated *”
   • No speedup comparisons between actual and estimated

3. Missing implementations:

   • Show visual ”✗” icon
   • Suggest trying smaller configuration

Implementation in update_benchmarks.py

# Estimate detection
if not has_actual_data:
    impl["is_estimate"] = True
    impl["estimate_note"] = f"Estimated from {closest_trades}×{closest_scenarios}"
    impl["closest_trades"] = closest_trades
    impl["closest_scenarios"] = closest_scenarios

Dynamic Key Insights

Key Insights now update based on selected configuration (no longer static).

Category-Specific Insights Generated

Example Output (at 10×100K)

Performance: AADC Python (668.7 ms) is 9x faster than Enzyme-AD (6.27s)
             and 13x faster than CoDiPack (8.85s) at 100K scenarios

Open-Source Choice: If commercial AADC isn't an option, Enzyme-AD (6.27s)
                    is the fastest open-source alternative but 9x slower than AADC.

Kernel Call Explanation

AADC C++ and Python have very different batch models - the component now shows separate explanations:

Example at 1000 trades × 100K scenarios:

• Python: ~2 batches/trade × 999 trades = ~1,998 kernel calls
• C++: ~12,500 batches/trade × 999 trades = ~12.5M kernel calls

CRITICAL: Total Evaluation Time is the Production Metric

The total_eval_time_sec column is the MOST IMPORTANT metric for production benchmarks.

total_eval_time_sec = eval_time_sec + recording_time_sec

Why This Matters

Key Insight: JAX’s eval_time (6.5s) is only 2.4× slower than AADC, but total_eval_time is 6.7× slower (19s vs 2.85s) due to JIT compilation overhead.

Recording/Compilation Happens on Every Cold Start

• Process restarts - Kubernetes pod restarts, service deployments
• Model changes - Any modification to pricing logic
• Parameter changes - Different trade structures may require recompilation
• Scale changes - Different batch sizes may trigger recompilation

Small Benchmarks Hide the Problem

At 10 trades × 1K scenarios, JAX’s 12.5s compilation might seem acceptable. But this isn’t production-representative. In production:

• Trading systems re-record kernels frequently
• Intraday risk runs with changing parameters
• RFQs require instant pricing for new structures

Always use total_eval_time_sec for production performance comparisons. Always show both eval_time and total_eval_time in benchmark displays.

Important: Execution Log Column Mapping

Before using execution log data, be aware that different model types have different column counts:

Always use execution_log_clean_16Jan.csv which has been corrected with fix_column_mapping.py. See /home/natashamanito/Asian Options Benchmark/AI Asian Pricer/data/CLAUDE.md for full details.

New calculated column: total_eval_time_sec = eval_time_sec + recording_time_sec

• For AADC: This is the cold start / total wall time (eval excludes recording)
• For non-AADC: Same as eval_time (no recording)

Data Sources

There are two data files that must be kept in sync:

1. Scatter Plot Data: src/content/data/benchmark-scatter-data.md

   • Used by: BenchmarkScatterPlot.astro component
   • Displayed on: All interactive benchmark pages and the main benchmarks index

2. Asian Option Benchmark Stats: src/content/resources/benchmarks/asian-option-monte-carlo.md

   • Used by: accelerate-python-models.astro, CodeViewer component
   • Contains: benchmarkStats object with speedup comparisons and formatted values

Field Mappings

CSV Execution Fields → scatter-data.md Properties

Important Timing Relationship:

For JIT/Recording implementations (AADC, JAX, Enzyme):
  greeks_time (eval_time) = first_eval + steady_state (excludes recording/JIT)
  total eval = greeks_time + recording_time (or compile_time)

For traditional implementations (NumPy, PyTorch, Basic Python, etc.):
  greeks_time = total evaluation time

Note: The component uses both recording_time and compile_time fields interchangeably. Implementations with JIT compilation (JAX, Enzyme) have compile_time in scatter data.

CSV Execution Fields → asian-option-monte-carlo.md benchmarkStats

Configuration Fields

testConfig Format: "{num_trades} trades × {num_scenarios/1000}K scenarios × {num_timesteps} timesteps"

Computed Metrics (Formulas)

Speedup Calculations

# Speedup vs Baseline
speedup = baseline_time / implementation_time

# Format: "Xx" (e.g., "420x")
aadcVsBasicPython = basicPythonTime / aadcTime
aadcVsNumpy = numpyTime / aadcTime
aadcGreeksVsCpp = cppGreeksTime / aadcGreeksTime
cppVsBasicPython = basicPythonTime / cppTime
numpyVsBasicPython = basicPythonTime / numpyTime

Greeks Overhead Calculation

# Greeks overhead = how much slower Greeks is vs price-only
# Format: "+X%" (e.g., "+42%")

overhead_pct = ((greeks_time - priceonly_time) / priceonly_time) * 100

# Example:
# AADC: priceonly = 67s, greeks = 95s
# overhead = ((95 - 67) / 67) * 100 = 41.8% → "+42%"

Progress Bar Width (InteractiveBenchmark.astro)

The progress bar uses logarithmic scale to handle wide time ranges:

// All times in same unit (seconds)
const logMin = Math.log10(Math.max(minTime, 0.001));
const logMax = Math.log10(Math.max(maxTime, 0.001));
const logVal = Math.log10(Math.max(currentTime, 0.001));
const logRange = logMax - logMin;

// Colored bar represents time: slowest = 100% fill, fastest = smallest fill
const barWidthPercent = logRange > 0
  ? ((logVal - logMin) / logRange) * 100
  : 50;

Speedup vs Baseline (Table Column)

# Baseline = slowest implementation in the group (typically Basic Python or autodiff)
speedup = slowest_time / implementation_time

# Format: "X.Xx" (e.g., "420.0x", "1.0x" for baseline)

Scatter Plot Data Structure

Each implementation in benchmark-scatter-data.md:

- id: aadc_python              # Unique identifier
  name: "AADC Python"          # Display name
  shortName: "Py AADC"         # Short name for legends
  category: "ml-libraries"     # One of: aad-tools, ml-libraries, python-implementations, languages
  color: "#22d3ee"             # Hex color for chart
  language: "Python"           # Programming language

  # Metrics (from CSV)
  greeks_time: 0.348           # eval_time_sec (seconds)
  memory: 52                   # memory_mb
  code_lines: 176              # model_total_lines or delta from baseline
  compile_time: 0.095          # recording_time_sec or kernel_execution_time_sec
  num_greeks: 3                # Number of Greeks computed

  recommended: true            # Optional: highlight as recommended

Page-Specific Categories

aad-tools.astro

• Implementations: aadc_cpp, enzyme_ad, codipack, adept, cppad, autodiff
• Focus: C++ AAD libraries comparison
• Baseline: autodiff (slowest)

ml-libraries.astro

• Implementations: aadc_python, jax, pytorch, autograd
• Focus: Python ML frameworks with AD
• Baseline: autograd (slowest)

python-implementations.astro

• Implementations: aadc_python, basic_python, numpy_vectorized, numba_jit
• Focus: Python optimization approaches
• Baseline: basic_python (slowest)

languages.astro

• Implementations: aadc_cpp, aadc_python, cpp_optimized, julia, haskell
• Focus: Cross-language comparison
• Baseline: haskell (slowest)

production.astro

• Implementations: aadc_cpp, aadc_python, jax, julia
• Focus: Production-ready options
• Baseline: jax (slowest)

Updating accelerate-python-models

The resources/publications/accelerate-python-models.astro page should use the same data from asian-option-monte-carlo.md. Currently it does via:

const benchmark = await getEntry('resources', 'benchmarks/asian-option-monte-carlo');
const benchmarkStats = benchmark?.data?.benchmarkStats;

When benchmark data changes:

1. Update asian-option-monte-carlo.md with new benchmarkStats:

   • aadcVsBasicPython, aadcVsNumpy, aadcGreeksVsCpp
   • aadcTime, cppTime, basicPythonTime, numpyTime
   • aadcLines, basicPythonLines, linesAdded
   • aadOverhead, bumpOverhead, numpyGreeksOverhead
   • testConfig, threads

2. Update benchmark-scatter-data.md with new metrics:

   • greeks_time, memory, code_lines, compile_time

3. The accelerate-python-models page will automatically reflect the changes because it reads from asian-option-monte-carlo.md.

Displayed Metrics on accelerate-python-models

Step-by-Step: Processing New Benchmark Results

1. Parse CSV Execution Log

Extract these fields for each implementation:

model_name, language, eval_time_sec, memory_mb, model_total_lines,
recording_time_sec, num_trades, num_scenarios, num_timesteps, num_threads

2. Calculate Derived Metrics

# For each implementation pair
speedup = baseline_time / impl_time
overhead = ((greeks_time - price_time) / price_time) * 100
lines_added = aadc_lines - baseline_lines

3. Update benchmark-scatter-data.md

implementations:
  - id: aadc_python
    greeks_time: <eval_time_sec>
    memory: <memory_mb>
    code_lines: <model_total_lines>
    compile_time: <recording_time_sec>

4. Update asian-option-monte-carlo.md

benchmarkStats:
  aadcVsBasicPython: "<calculated>x"
  aadcTime: "<eval_time_sec>s"
  linesAdded: "+<calculated>"
  # ... etc

5. Verify accelerate-python-models Updates

Run npm run build and verify:

• Hero displays new speedup
• Performance table has correct numbers
• FAQ answers reflect new data

Validation Checklist

When updating benchmark data:

• All speedup values are calculated consistently (baseline / implementation)
• All overhead values use same formula: (greeks - price) / price * 100
• testConfig matches actual benchmark configuration
• linesAdded = AADC lines - baseline lines
• Times are formatted consistently (“Xs” for seconds, “~X hours” for long runs)
• Both benchmark-scatter-data.md and asian-option-monte-carlo.md are updated
• Run npm run build to verify no errors
• Visual check: scatter plots display correctly
• Visual check: accelerate-python-models shows updated numbers

Timing Breakdown Explanation

AADC / Kernel-Recording Implementations

For AADC and similar kernel-recording implementations, the timing fields have specific meanings:

IMPORTANT: eval_time_sec (displayed as “Greeks Time” in main table) does NOT include recording time. It measures only the kernel evaluation.

Key Insight:

• Cold Start = Recording + Evaluation (e.g., 48.8ms + 39.4ms = 88.2ms)
• Subsequent runs = Evaluation only (e.g., 39.4ms) — kernel is already recorded
• Recording overhead is amortized over many evaluations

Per-Kernel Execution Time Calculation

The “Per Kernel” column shows the time for a SINGLE kernel execution. This is calculated as:

Per Kernel = steady_state_time / num_kernel_calls_steady_state

Where num_kernel_calls depends on the implementation:

Example (10 trades × 1K scenarios):

Key insight: C++ AADC processes 8 scenarios per kernel call (AVX512), while Python AADC processes up to 65536 scenarios per call. This explains the large difference in per-kernel time but similar total evaluation time.

Full timing breakdown (Python AADC at 10 trades × 1K scenarios):

• Recording: 48.8 ms (one-time kernel build)
• Per Kernel: 3.94 ms (single kernel execution)
• Steady State: 35.5 ms (9 kernel calls × 3.94 ms)
• Cold Start: 88.2 ms (recording + steady state)

Traditional Implementations

For non-kernel-recording implementations:

Traditional implementations re-compute everything each evaluation - no amortization benefit.

Additional CSV Fields (Reference)

These fields are available but not currently mapped:

File Locations Summary

src/
├── content/
│   ├── data/
│   │   └── benchmark-scatter-data.md     # Scatter plot metrics
│   └── resources/
│       ├── benchmarks/
│       │   └── asian-option-monte-carlo.md  # benchmarkStats (single source)
│       └── articles/
│           └── accelerate-python-models.md  # References benchmarkStats
├── pages/
│   ├── technology/
│   │   └── benchmarks/
│   │       └── interactive-benchmarks/
│   │           ├── CLAUDE.md (this file)
│   │           ├── aad-tools.astro
│   │           ├── ml-libraries.astro
│   │           ├── python-implementations.astro
│   │           ├── languages.astro
│   │           └── production.astro
│   └── resources/
│       └── publications/
│           └── accelerate-python-models.astro  # Uses benchmarkStats
└── components/
    └── benchmark/
        └── BenchmarkScatterPlot.astro    # Uses scatter-data.md