Models are unrealistic by design
Part 2 of the 6-part series, Modeling for metascientists (and other interesting people).
The simple observation that most models are oversimplified, approximate, incomplete, and in other ways false gives little reason for using them. Their widespread use suggests that there must be other reasons.
- William Wimsatt¹
A common criticism of models is that they are unrealistic. I’ve had more than a handful of conversations where a model was criticized for this reason. A typical conversation might go something like:
Hey — did you see the recent PNAS model about how rewarding replications doesn’t actually incentivize better research?
Nope, sounds interesting. What did you think of it?
It’s terrible. They assume that replications don’t take any time and that research only comes in two types: high-quality or low-quality. But obviously replications take time, and you can’t just dichotomize research into two types or think that quality is all that matters! What about impact?! The model just leaves all this out!
Ah, too bad. So it really unrealistic? Like, it just oversimplifies things?
Exactly. Modelers, amirite? Alright, gotta go — this 2 x 2 lab experiment in a windowless basement room with first-year students filling out a questionnaire isn’t gonna run itself!
The real world is more complicated than the world of the model. So, the criticism goes, the model is bad, because it isn’t realistic.
That last bit, poking fun at how the same critic is perfectly happy to run off and conduct an experiment that is equally unrealistic (and doesn’t see the irony) is there for two reasons:
- I was feeling a bit grumpy when I wrote this.
- There is a deep similarity between empirical and theoretical work.²
The world is complicated, and we must rely on simplified systems to understand it. Model organisms, lab experiments, theoretical models — all of these are “unrealistic.” But they are unrealistic by design, because this makes them easier to study, understand, poke, prod, and so on.
Some of my favorite writing on this topic is by Hanna Kokko, a theoretical evolutionary ecologist.
In Modeling for Field Biologists and Other Interesting People,³ Kokko explains this as follows (Disclaimer 1: I’m going to quote a whole page here. I feel a bit bad, but it’s better than anything I could have written, so Kokko didn’t leave me much choice. Disclaimer 2: I have modified the spacing, for readability):
…models only exist because we need them to help us: none of us are born with such supercomputer brains that we could evaluate arbitrarily complex arguments immediately and without external help.
What is the optimal complexity of a model, then? Once again, it depends on the question. Reflect for a moment that there are maps with different scales. In the context of scientific models, it is useful to be reminded of the ultimate reason we do science: it is the joy of understanding something.
If we could visualize and memorize much more detailed maps than we currently do, useful maps would include more detail than they currently do. Exactly analogously, if we could grasp much more complex processes without getting headaches than we currently do, models would look different too.
Given the way our brains are built, a good guideline is that a model should include all the relevant details for the particular question at hand, but it should be kept so simple that it can be understood (if with joy, then still better).
In other words, a model is not particularly helpful if it predicts that under conditions A the animal should do X, while under conditions B it should do X 30% of the time and Y in the rest of instances, and then there are 17 other parameters that interact with each other in producing a diversity of outcomes — but when asked why the model produces these effects of A and B, we still have no answer that can be expressed in a language that anyone’s intuition can understand. Removing some additional detail from the model can then be surprisingly helpful: the effects of A and B could still be the same, but with far simpler equations.
For example, we might have spent a lot of time modelling the distribution of body condition in a population of migratory birds, ending up with very cumbersome equations, when a far simpler way to grasp the conceptual issue is to divide up the population into two classes of individuals, ‘hungry’ and ‘satisfied’.
To show a conceptual point, this might be sufficient. Results could be far simpler to derive this way than with a more complete model, and if our understanding of the biology advances faster this way, the simplification is justified.
But how to know, then, that the division has not caused some artifacts? Perhaps an exact shape of the body condition distribution would have produced a totally different answer? The answer is . . . we don’t really know, unless we build the more complicated model too. (Which means that modellers rarely run out of models to study.)
Alternatively, it is often the case that the simple model has dealt with most of the thinking load, so that extrapolating to the last step can (fairly reliably) be achieved using imagination and verbal argumentation.
Scholars in many fields have made similar points (for a few references, see pg. 1 of Paul Smaldino’s, “How to Translate a Verbal Theory into a Formal Model”⁴). One of these, Philosopher William Wimsatt,¹ even scooped my earlier joke:
Many people who would applaud the use of appropriate controls and isolations in an experimental design inconsistently turn on mathematical or causal models of the system and criticize them for doing the same thing! This activity is desirable and necessary in either case.
I see four takeaways here:
- I am not as clever as I think I am.
- Models are useful because of their simplicity, not despite it.
- There are deep similarities between models and experiments.
- There is a large, interdisciplinary body of work that we can draw on to hone our intuitions about models. This could prove useful as we continue to have productive discussions about the role of theoretical models in the metascientific enterprise.
- Wimsatt, William C. “False models as means to truer theories.” Neutral models in biology (1987): 23–55. http://mechanism.ucsd.edu/teaching/models/Wimsatt.falsemodels.pdf
- Mäki, Uskali. “Models are experiments, experiments are models.” Journal of Economic Methodology 12.2 (2005): 303–315. https://doi.org/10.1080/13501780500086255
- Kokko, Hanna. Modelling for field biologists and other interesting people. Cambridge University Press, 2007. p. 8–9. https://doi.org/10.1017/CBO9780511811388
- Smaldino, Paul E. “How to translate a verbal theory into a formal model.” Social Psychology 51.4 (2020): 207. p. 1. https://doi.org/10.1027/1864-9335/a000425
