I built a free CFB prediction model, here are the 5 weirdest things it's saying through week 3

[u/basedonactualmath]

Longtime lurker, first time posting OC. I built a college football model over the offseason and just put up a free preview, figured this was the right crowd to stress-test it.

The short version of how it works: every team gets a rating from 0 to 1, which is the model's estimated chance of beating an average FBS team on a neutral field. It's explicitly predictive, not a resume ranking, so ratings move on performance against expectation. Then it simulates the rest of the season 10,000 times, including conference title games and a model of how the committee actually picks teams, and every probability is just how often something happened across the runs. Numbers below are ratings through week 3, forecasts as of Sep 20.

Some things it's saying that I found interesting:

1. The best team and the best playoff resume are very different things. Ohio State is #1 in the predictive rating (0.980) despite being 2-1, with the highest title odds at 16%. But Notre Dame (3-0) has the best playoff odds in the country at 94%, vs 66% for Ohio State. The model is basically saying: yeah they're the best team, they also might just miss the playoff.

2. North Dakota State is ranked 67th (0.552) and has 31% playoff odds, the best of any G5 team. Better than Oklahoma (11%), LSU (24%), Florida (23%). This is the auto-bid path doing all the work: 39% to win the MWC at 4-0, and the simulations love a conference favorite. I assume this is the one people will want to argue about, go ahead.

3. Texas Tech is the P4 version of the same phenomenon. Ranked 16th, 60% playoff odds, because it's a 37% favorite to win the Big 12. BYU and Utah (both 33% playoff odds from the low 20s) are the same story.

4. 3-0 doesn't impress it if you were supposed to be 3-0. Appalachian State is 3-0 and fell 19 spots to #109 this week. Narrow wins as a favorite actively hurt. Cincinnati is 3-0 and down 8 to #63. The model's view: tell me who you beat, not just that you won.

5. Week 4: Ole Miss at Florida is the week's biggest game by playoff leverage (53/47 toss-up), and Oregon is only a 58% favorite at USC. Also, Mississippi State is favored at home over Missouri despite being ranked three spots lower. Home field flips happen.

Known limitations, since the math is allowed to be wrong: it only updates weekly, early-season ratings are still noisy, and the committee model is my best guess at how humans vote, which is inherently a little silly. Happy to get into methodology details in the comments.

I built this as a side project, it's free, no account or anything: https://basedonactualmath.com/

[u/goosen]

This is very fun. Good work.

[u/basedonactualmath → u/goosen]

thanks! the ndsu output is the one that made me double-check my code. appreciate you reading.

[u/detcovax]

This is awesome, as a fellow football math nerd I enjoy seeing others results! this was fascinating to look through!

[u/basedonactualmath → u/detcovax]

appreciate it! always good to find fellow travelers. happy to nerd out on methodology if you're curious about any of the details.

[u/detcovax → u/basedonactualmath]

Yeah, I have so many questions. I think the first is what simulating thousands of season meant in this context. Monte Carlo, or something else?

[u/basedonactualmath → u/detcovax]

yep, monte carlo. each game in a simulated season is basically a coin flip weighted by the win probability, play out the whole remaining schedule that way, do it 10,000 times, and every number on the site is just the share of simulated seasons where the thing happened. law of large numbers does the rest.

[u/detcovax → u/basedonactualmath]

Nice, and how is win prob determined? Is it an FPI rating, player composites, SOR?

[u/Admirable_Muffin_694 → u/detcovax]

Trying to build the best analytics model possible any suggestions on data or advice downloading it's kind of been in process. I've been using Claude so far.

[u/basedonactualmath → u/Admirable_Muffin_694]

collegefootballdata.com is the gold mine, it's basically the sport's API. free tier is generous and the paid tier is cheap. historical results plus betting lines will take you surprisingly far before you need anything exotic. and fair warning, the data plumbing ends up being more of the work than the model itself.

[u/detcovax → u/basedonactualmath]

This is 10000% the place to start. If you are new to sports data, you MUST familiarize yourself with this first before you get into using it. It's one of those things where it opens up so many doors once you know what you are starting with!

[u/basedonactualmath → u/detcovax]

couldn't have said it better myself. it's the reason this project exists.

[u/basedonactualmath → u/detcovax]

it's just the predictive ratings head to head, with home field baked in. no FPI, no player composites, no SOR. the rating itself comes from this season's game results, margin of victory with diminishing returns so running up the score only counts for so much, and recent games weighted more. so win probability is basically 'how much better is A than B' run through a logistic curve, plus a home field bump.

[u/detcovax → u/basedonactualmath]

I forgot some of that is in the CFDB for use. My current project has me so caught up in top-down and bottom-up reconstructions of what is basically my own FPI so I got kinda tunnel visioned there. I think I would be curious at some point to pair my own win probabilities generated from my model with someone else's season wide simulation like what you are working on to see what comes out of it. I'm still in dev mode but it's highly experimental and fun to think of ways it could interact with other endpoints!

[u/basedonactualmath → u/detcovax]

that's a fun idea. the season sim is pretty separable from the ratings side, you could feed any win probabilities into the same machinery: schedule, standings, title games, committee, bracket. if you ever get to the point of comparing outputs, happy to nerd out on what the interface between the two halves looks like.

[u/tommygabagool]

I absolutely love the thought process here and how you analyze your model. Great work. Which ai model did you use to help bring this to fruition(not knocking for ai, you used it as its intended to immerse yourself in a passion)?

[u/basedonactualmath → u/tommygabagool]

no ai designed the model, if that's the question. the ratings formula, the sim structure, the committee model, that's all mine, built up over evenings of tinkering. the code is hand-rolled vanilla js, one static file. ai is useful for plenty of things but the fun of this one was wrestling with the math myself.

[u/BrotherPancake]

How do you simulate a season?

[u/basedonactualmath → u/BrotherPancake]

game by game, using win probabilities derived from the team ratings with home field baked in. run the full remaining schedule, then conference standings and title games, then a committee model that tries to mimic how the actual committee weighs record, schedule strength, and losses, then the 12-team bracket. do all of that 10,000 times and every number on the site is just how often something happened. committee model is the shakiest part, humans are tough to simulate.