Stats · Walchi Oracle

Visitors

Real visitors (30 days): 1357
Page views (30 days): 4233

Bots and scanners filtered out; IPv6 addresses grouped per household (/64).

Live models (full history)

What's actually deployed, scored over the whole seasonal history (Apr–Oct, ~10 years): the rule and the distilled logistic model. Both on the same day set.

Rule-based

Days scored: 1980
Forecast skill: +0.17

Wind days caught: 90%; Share of real wind days (≥ 1 h ≥ 8 kt) where the oracle did not say NO GO.
Wasted drives avoided: 27%; Share of real no-wind days that were correctly forecast as NO GO.

Peirce score (sensitivity + specificity − 1). 0 = no better than guessing, 1 = perfect. Positive = useful. (68 thunderstorm days, scored on thermal merit (LI decoupled).)

Advanced statistics — rule layer

3-class accuracy: 38%
Naive baseline: NO GO · 44%

Share of days where the forecast hit exactly the right bucket (session / marginal / no wind).

What a blunt "always the same" guess (the most common outcome) would score. The forecast has to beat this to be useful.

Confusion matrix — rule layer

Forecast ↓ Actual →	GO	MAYBE	NO GO
GO	352	336	378
MAYBE	145	155	260
NO GO	42	73	239

Rows = what was forecast, columns = what actually happened at the lake. The diagonal holds the hits.

Sensitivity and specificity are binary measures. The three buckets are collapsed to “wind day yes/no”: GO and MAYBE count as a wind day, NO GO as calm — for both the forecast and the actual outcome.

ML — logistic

Days scored: 1980
Forecast skill: +0.29

Wind days caught: 76%; Share of real wind days (≥ 1 h ≥ 8 kt) where the oracle did not say NO GO.
Wasted drives avoided: 53%; Share of real no-wind days that were correctly forecast as NO GO.

Same day set as above, but forecast = the experimental logistic ML classifier. Never drives the official verdict — comparison only. (68 thunderstorm days, scored on thermal merit (LI decoupled).)

Advanced statistics — ML logistic (parallel model)

3-class accuracy: 49%
Naive baseline: NO GO · 44%

Share of days where the forecast hit exactly the right bucket (session / marginal / no wind).

What a blunt "always the same" guess (the most common outcome) would score. The forecast has to beat this to be useful.

Confusion matrix — ML logistic (parallel model)

Forecast ↓ Actual →	GO	MAYBE	NO GO
GO	265	165	251
MAYBE	158	252	164
NO GO	116	147	462

Rows = what was forecast, columns = what actually happened at the lake. The diagonal holds the hits.

Sensitivity and specificity are binary measures. The three buckets are collapsed to “wind day yes/no”: GO and MAYBE count as a wind day, NO GO as calm — for both the forecast and the actual outcome.

Fair comparison (held out, 2023+)

A like-for-like three-way comparison on one shared day set: only the unseen test years from 2023 on, out-of-sample for all three. The rule plus the bundle's logistic and HGB models trained up to 2022 — so HGB isn't graded on its own training years.

Rule-based

Days scored: 715
Forecast skill: +0.15

Wind days caught: 91%; Share of real wind days (≥ 1 h ≥ 8 kt) where the oracle did not say NO GO.
Wasted drives avoided: 24%; Share of real no-wind days that were correctly forecast as NO GO.

Peirce score (sensitivity + specificity − 1). 0 = no better than guessing, 1 = perfect. Positive = useful.

Advanced statistics — rule layer

3-class accuracy: 35%
Naive baseline: NO GO · 44%

Share of days where the forecast hit exactly the right bucket (session / marginal / no wind).

What a blunt "always the same" guess (the most common outcome) would score. The forecast has to beat this to be useful.

Confusion matrix — rule layer

Forecast ↓ Actual →	GO	MAYBE	NO GO
GO	125	125	151
MAYBE	63	52	88
NO GO	14	22	75

Rows = what was forecast, columns = what actually happened at the lake. The diagonal holds the hits.

Sensitivity and specificity are binary measures. The three buckets are collapsed to “wind day yes/no”: GO and MAYBE count as a wind day, NO GO as calm — for both the forecast and the actual outcome.

ML — logistic

Days scored: 715
Forecast skill: +0.23

Wind days caught: 70%; Share of real wind days (≥ 1 h ≥ 8 kt) where the oracle did not say NO GO.
Wasted drives avoided: 54%; Share of real no-wind days that were correctly forecast as NO GO.

Same day set as above, but forecast = the experimental logistic ML classifier. Never drives the official verdict — comparison only.

Advanced statistics — ML logistic (parallel model)

3-class accuracy: 45%
Naive baseline: NO GO · 44%

Share of days where the forecast hit exactly the right bucket (session / marginal / no wind).

What a blunt "always the same" guess (the most common outcome) would score. The forecast has to beat this to be useful.

Confusion matrix — ML logistic (parallel model)

Forecast ↓ Actual →	GO	MAYBE	NO GO
GO	75	63	74
MAYBE	65	77	72
NO GO	62	59	168

Rows = what was forecast, columns = what actually happened at the lake. The diagonal holds the hits.

Sensitivity and specificity are binary measures. The three buckets are collapsed to “wind day yes/no”: GO and MAYBE count as a wind day, NO GO as calm — for both the forecast and the actual outcome.

ML — HGB (black box)

Days scored: 715
Forecast skill: +0.27

Wind days caught: 66%; Share of real wind days (≥ 1 h ≥ 8 kt) where the oracle did not say NO GO.
Wasted drives avoided: 61%; Share of real no-wind days that were correctly forecast as NO GO.

Forecast = HistGradientBoosting. Trained on the years up to 2022, so scored here only on the test years from 2023 on (a different, smaller day set than above). The strongest model in the comparison — it runs live as a black-box shadow model but never drives the official verdict.

Advanced statistics — HistGradientBoosting (black-box model)

3-class accuracy: 49%
Naive baseline: NO GO · 44%

Share of days where the forecast hit exactly the right bucket (session / marginal / no wind).

What a blunt "always the same" guess (the most common outcome) would score. The forecast has to beat this to be useful.

Confusion matrix — HistGradientBoosting (black-box model)

Forecast ↓ Actual →	GO	MAYBE	NO GO
GO	76	58	61
MAYBE	51	81	61
NO GO	75	60	192

Rows = what was forecast, columns = what actually happened at the lake. The diagonal holds the hits.

Sensitivity and specificity are binary measures. The three buckets are collapsed to “wind day yes/no”: GO and MAYBE count as a wind day, NO GO as calm — for both the forecast and the actual outcome.