Does your model account for XYZ?
Exploring the most frequent question.
All model data is already updated for the next gameweek in a single page here.
I’ve been sharing my hobby of building team ratings models for several years now. Almost every week, I’m asked whether my model “accounts for XYZ”.
It is a good and fair question!
But what does it actually mean?
Considering and Accounting
One of the biggest ‘myths’ implicit in a lot of these interactions is that data-driven models are entirely objective. That there is a ‘correct’ way to do things.
Models will always, to a certain extent, reflect the beliefs of their creators. This is true mechanically. Models are shaped by countless approximations, assumptions, and decisions made during creation. These decisions are central to the core question of ‘what wins football matches?’ and venture into smaller questions such as:
How to account for red cards?
What about predicting/rewarding penalties?
How much does ‘finishing skill’ matter?
Do own goals ‘count’?
What about manager influence?
Do we count incorrect decisions (e.g. offside goal given)?
How to account for new transfers?
These are just a handful of many examples. People often ask me directly for the "answer" to questions like these. To me, at least, these are very difficult questions. There is no clear answer, and dealing with them is (currently) more of an art than a science. Over time, this art can evolve into a science as we gain insights into which assumptions are flawed and refine our approach.
The question “Does the model account for this?” is often a loaded question that jumps towards an answer to the questions above. The question is almost always implying an expectation that ‘the concept’ should be accounted for in a specific positive or negative way. In essence, the question already has the approximations, assumptions, conclusions and beliefs of the questioner baked in.
I think it’s therefore important to take a step back from these implicit assumptions and first look at the concepts in a neutral way. For this, I like to use the term consider.1
So the flow looks like this:
Concept → Consideration → Accounting
There is a concept that might affect the model (e.g., player absences, penalties, etc.). I then consider how this might affect things and determine whether it warrants attention, from which I move on to the actual doing of accounting for it in the model.
However, this does not mean that every concept automatically leads to a specific positive or negative adjustment. The jump from consideration to accounting is where the ‘values’ part steps in. Whether a concept ends up being accounted for in a particular way depends on the individual’s philosophy and personal beliefs about how football works. Simply put, the model considers everything, but it doesn’t necessarily account for everything (either at all or in the way you might expect).
Case Study I: Rodri’s Absence
Let’s take a concrete example: Rodri missing the season for Manchester City.
People asked: “Does the model account for this?”
What they really meant was: “Have you made Man City worse because he is injured?”
On the surface, it seems a straightforward and fair question: if a key player is missing, does the model reflect a drop in team strength?
However, the question itself is somewhat flawed and circular. By assuming Manchester City will be worse without Rodri, the question already has a conclusion baked in. People asking it are not really prepared to reckon with the full consequences of adjusting the team rating either way. The question I would ask back is ‘are you willing to accept a world where Man City are better without Rodri?’.
If the answer to that is no, which it often is, then any conversation about updating beliefs is dead on arrival. Logical consistency requires that we remain open to the possibility that outcomes could go either way. Teams adapt, change tactics or formations, and other players may step up, sometimes maintaining or even improving overall performance. If, in reality, Manchester City performs better without him, the logical conclusion would be that they might actually be stronger without Rodri, but that contradicts the questioner’s initial assumption.
Case Study II: Red Cards and Penalties
Another helpful example. I often receive comments on weekly team ratings updates that imply an update is incorrect because a red card or penalty was issued in a game. Penalties and red cards are difficult to predict, making them particularly challenging to incorporate into a model. It can be difficult to determine exactly how they should be considered and accounted for. Similar to the Rodri example, these events can create challenges with logical consistency (for example, what do you do if a team gets a red card and then performs better?).
A common approach is to simply dismiss red cards entirely, removing all game time after the event from the data, or to focus only on non-penalty metrics, essentially pretending that penalties never exist. Both of these approaches are the “considered but not accounted for” variety. Given the high degree of randomness in these events, you can see why this is appealing!
But ignoring actual events risks leaving the model unprepared for updating beliefs when there might be some signal, such as a team receiving red cards or penalties in consecutive games. At some point, what was considered “noise” may carry meaningful information. The key is to design a model that can sensibly consider and account for these rare events to some degree, while maintaining logical consistency and the ability to update predictions based on actual outcomes.
We want to avoid creating paradoxes.
What do you do if a team is performing better without their key player? What do you do if a team performs better after getting a red card? What do you do if a team loses a match but missed 3 penalties in the match?
If you don’t have fleshed out consideration to accounting flow response to these questions, then you’re probably creating some sort of paradox.
Conclusion
So if these are ‘bad’ questions, what do good ones look like? As a general rule, I think we should start by highlighting the consideration whilst sharing more context about our personal beliefs on accounting for that. For example, today I was asked two similar questions:
“Does this factor in Bruno’s positioning from this season or does it take data from last season.”
“does this take into account bruno's new role as a cdm?”
For the reasons outlined in this post, I don’t really know how to engage with these questions. Yes, I am considering Bruno’s position, but that still doesn’t answer the accounting part. Perhaps a better question would be something like “What are your considerations around Bruno in your model? Those Bruno projections look higher than my expectations, as I think he will play CDM and get less production from that position”.
After doing this hobby for multiple years, I think I have almost definitely considered every concept at this point.
So if the question is simply, “Have you thought about this?”, then the answer is yes.
If the question is, “Am I accounting for this concept in the exact positive/negative way you think I should?”, then the answer is: I don’t know!
How concepts are ultimately accounted for in predicted outcomes depends on personal beliefs and philosophy. I try to be as transparent as I can and share all my outputs openly. That said, I can’t spend all my time explaining every nuance of my philosophy and approach behind this little hobby. If you’re desperate to find out exactly how something should be accounted for, the best way might be to experiment and build your own model :).
For clarity within this post, I am using considering and accounting as defined terms with the specific meanings I give them. In reality, the question comes in many forms, and I cannot always know the true intentions, but “account for” is the most common phrase I perceive to be used in the manner of questioning I’m describing.