Results

TEST window: 2022-W01 → 2023-W52 (104 weeks, held out from all model fitting). TEST window observed: 23 weeks with ≥1 incident (prevalence 22%), 1 week with ≥2 incidents (prevalence 1%).

Count task — best on tail-quantile loss

Model	RMSE	MAE	MAPE	pinball 0.5	pinball 0.8	pinball 0.9
naive_seasonal	0.4653	0.3096	30.00	0.1154	0.1865	0.1087
nb_lag	0.4524	0.3191	31.02	0.1154	0.1942	0.0827
nb_full	0.4502	0.3438	33.50	0.1154	0.1673	0.0817

The full-feature Negative-Binomial model (nb_full) dominates the upper quantiles (pinball@0.8 and pinball@0.9) where mass-shooting forecasting actually matters. RMSE / MAE / MAPE are near-tied across all three models — the target class is sparser than the autoregressive signal can resolve.

Binary ≥1 task — best on precision-recall + calibration

Prevalence: 22% (23 / 104 weeks).

Model	AUC-PR	AUC-ROC	Brier
nb_lag	0.237	0.557	0.179
logistic_any	0.271	0.553	0.177

The dedicated logistic classifier beats the NB-derived probability on the metrics that matter for incident alerting — precision-recall (fewer false positives) and Brier (calibration).

Binary ≥2 task — too rare to learn

Only 1 positive in 104 weeks. The dedicated logistic regresses; the lag-only NB's probability is the more useful tail signal.

Model	AUC-PR	AUC-ROC	Brier
nb_lag	0.0213	0.5534	0.00969
logistic_high	0.0130	0.2621	0.00963

Interpretation

The headline finding is modest improvement over baselines, consistent with the 80-event-count target class (104-week TEST window, 22% positive rate for the ≥1 binary). The full model adds meaningful lift on the upper-quantile loss (pinball 0.8: 0.167 vs 0.187 = 10.4% improvement over naive seasonal; pinball 0.9: 0.082 vs 0.109 = 24.8% improvement over naive seasonal) which is the regime where an alerting system would actually fire.

Public-coverage proxy and NICS lagged signals carry the bulk of the lift. The paper §6 (Discussion) breaks this down with feature importance from the NB GLM.

Downloads

results/metrics_count.csv — count task
results/metrics_any.csv — binary ≥1 task
results/metrics_high.csv — binary ≥2 task
results/predictions_test.csv — weekly predictions across the 104-week TEST window
results/model_coefficients.csv — NB GLM coefficients
results/cv_metrics.csv — rolling-origin inner cross-validation on TRAIN

Reproducibility

git clone https://git.catalystgroup.tech/herman/msf-forecast-pipeline
cd msf-forecast-pipeline
pip install -r requirements.txt
make all     # ≈ 2 minutes; writes results/*.csv + results/figures/*.png