Tested against what actually happened.
Each model is compared with observed data it was not fitted on — measured winds, insurance claims, reported emissions. We publish the weak results alongside the strong ones, and every test can be re-run from the script named under it.
We have not benchmarked against commercial climate-VaR products, whose models and outputs are not public.
Reads Like a Model Risk Management File, Because It Is One
Bank model-validation teams generally ask the same three questions of any model before it reaches production: is it conceptually sound, is it monitored on an ongoing basis, and has its outcome been tested against reality. The tests below are organised the same way, just not labelled that way until now:
Conceptual soundness
Every hazard function traces to a named public source — see the full methodology and the data sources listed below.
Ongoing monitoring
Model Lab's global sweep and the worldwide QA run are re-executed, not one-off — each ships with the script that produced it.
Outcomes analysis
Cyclone winds vs. measured stations, flood scores vs. NFIP claims, emissions vs. EPA GHGRP — hindcast against data the model was never fitted on.
Tropical cyclone wind
Truth: Measured hourly winds at 148 weather stations (NOAA ISD) within 150 km of 26 storms' tracks
Leave-one-storm-out calibration. Correlation r = 0.53, RMSE 7.7 m/s, gale false-alarm rate 26%. Skill is strongest in the North Atlantic (r = 0.69, 76 stations) and weakest in the West Pacific (r = 0.31).
Usable for site screening; basin-dependent.
scripts/validate_tc_wind.py
Flood exposure vs insurance claims
Truth: FEMA NFIP claims per km² of built-up land, 180 census tracts in Harris TX, East Baton Rouge LA and Charleston SC
The area model (built-up land × height above nearest drainage) is fitted on two counties and tested on the third: AUC 0.61 in Harris, 0.78 in East Baton Rouge, 0.73 in Charleston. The earlier single-point terrain score showed no skill in Harris County, so site-level flood losses now come from JRC river and Deltares coastal flood depth maps instead.
Moderate skill in three US counties; not yet evidence of global skill.
scripts/backtest_flood_area.py · scripts/backtest_flood_nfip.py
Emissions estimates
Truth: EPA GHGRP 2023 reported emissions for 163 US companies, matched to SEC revenue
Revenue-based estimates are best for oil & gas (62% within 2×) and weakest for utilities, where fuel mix dominates. The engine therefore uses reported or facility-level emissions (Climate TRACE, EPA GHGRP) wherever they exist and labels revenue-based estimates as such.
Screening-grade only — replace with reported data before any decision.
scripts/evaluate_emissions_models.py
Worldwide QA run
Truth: Manual review of site analyses and company asset maps on every inhabited continent
Sites range from Fiji and Dhaka to Lima, Lagos and Kedarnath. Failures found in earlier rounds (a pin on open water read as flooding, a lake counted as new flood water) were fixed in the engine and re-tested.
A sanity check, not a statistical benchmark.
scripts/world_qa.py
Historical disaster calibration
Truth: 16 sourced catastrophe losses, 1997–2022 (EM-DAT, Munich Re NatCatSERVICE, Swiss Re sigma, FAO, World Bank and national disaster agencies), each compared against a single representative company run
Each event's documented loss is compared against the engine's own hazard output for one representative company, normalised to loss-as-percent-of-revenue because historical figures are economy-wide and the engine runs at company level. Error climbs past 100% for most events — not because the hazard magnitudes are wrong, but because one proxy company cannot stand in for an entire economy's sector mix. It is a transparency check on the calibration module itself, never used to tune the model: run per-event from the same script shown below.
A deliberately blunt check. The measured-hazard results above are the stronger evidence; this one is published for the same reason they are — so a reviewer sees it, not only the results that flattered the model.
climate_risk_engine/src/cri/climate/scenarios/calibration.py · historical_events.py
Model Lab global sweep
Truth: 222 from-scratch physics studies at 37 cities across every inhabited climate zone, 6 hazards, checked against each site's own historical record where one exists
Model Lab builds a hazard model from first principles on request — picks the physics, pulls terrain and reanalysis data, runs it, verifies it, and hindcasts it against the site's own historical record — across flood, wind, fire, heat, cyclone and earthquake. Run at 37 cities from Reykjavik to Singapore, 19 of 222 studies failed their hindcast outright and are listed, not dropped; several of those are the hazard genuinely intensifying (recent years measurably exceeding what the early record predicted), reported as a finding rather than an error. One bug surfaced mid-sweep — a fire-weather run crashed on a hyper-arid site — and is fixed.
The newest and widest of these checks: real breadth, an honest failure rate, climate trends it caught rather than smoothed over. Full site-by-site map on the Updates page.
mac-test/mac_lab_e2e.py · sweep/sweep.jsonl
Data sources