Final post of the 6-part series, Modeling for metascientists (and other interesting people).

Poor old psychology: looking silly again.

It wanted to be seen as a real science, hypothetico-deductive model and all. First it looked a bit silly when most of the ‘effects’ that constituted the alleged support for its theories turned out not to replicate. And now it’s looking silly for a second time, as people point out that those theories themselves are remarkably weak (actually this problem has been known from much longer, since before the beginning of the current replication crisis).¹

It’s often not clear what, precisely…


Part 5 of the 6-part series, Modeling for metascientists (and other interesting people).

Hopefully you are now convinced that formal models are useful tools to help scientists understand the world. As an anthropologist with training in evolutionary biology who started her career as a modeler, I chose to do mathematical modeling because models have the following three advantages:

  1. Compared to the real world, models are simple and easy to understand.
  2. Compared to verbal models (which all researchers use), models are clear and transparent and are more likely to generate precise and valid predictions.
  3. Compared to experiments, models can test hypotheses…


Part 4 of the 6-part series, Modeling for metascientists (and other interesting people).

Many social scientists do not consider formal models a useful tool in theory construction. They might defend this position by arguing that “formal models make assumptions that are too simplistic, too unrealistic, and too arbitrary.” Instead, they rely on verbal models.

In turn, formal modelers complain about the ambiguities associated with verbal models, including imprecise constructs, implicit assumptions, and predictions based on intuitive reasoning rather than deduction.

In disciplines where verbal models dominate, theoretical ambiguity is widespread. This raises some uncomfortable questions: Don’t scientists notice these ambiguities…


Part 3 of 6, Modeling for metascientists (and other interesting people).

http://dx.doi.org/10.13140/RG.2.2.19688.39688

In Part 1, I covered the misconception that models are mainly useful if you want to generate precise predictions. In fact, models have many uses, a key one being that models promote transparency.

In Part 2, I covered the misconception that models must be realistic to be useful. In fact, models are unrealistic by design: just like maps, models leave out details to provide a clearer picture of the things that really matter, and different types of maps (models) are useful for different purposes.¹ ²

In part 3…


Part 2 of the 6-part series, Modeling for metascientists (and other interesting people).

http://dx.doi.org/10.13140/RG.2.2.31851.87841

The simple observation that most models are oversimplified, approximate, incomplete, and in other ways false gives little reason for using them. Their widespread use suggests that there must be other reasons.

- William Wimsatt¹

A common criticism of models is that they are unrealistic. I’ve had more than a handful of conversations where a model was criticized for this reason. A typical conversation might go something like:

Hey — did you see the recent PNAS model about how rewarding replications doesn’t actually incentivize better research?

Nope…


Part 1 of 6, Modeling for metascientists (and other interesting people).

http://dx.doi.org/10.13140/RG.2.2.11024.53767

I often hear the argument that theories in psychology are immature because they don’t make “risky” predictions (that is, predictions that are likely, given the theory, but unlikely, absent the theory). So, the argument goes, we need formal models to help generate precise predictions, such as a range of values or a smallest effect size of interest.¹

Fair enough.

But formal models have many uses besides prediction.² Off the top of his head, Joshua Epstein lists 16 reasons (other than prediction) to build models, including developing causal explanations…


http://dx.doi.org/10.13140/RG.2.2.35351.50086

Brief Introduction

This is a series about modeling. It is mainly meant for metascientists in the social sciences. With a bit of luck, it will also be useful to other interesting people.

Part 1: Models are for transparency. (Leo Tiokhin)

Part 2: Models are unrealistic by design. (Leo Tiokhin)

Part 3: Five modeling misconceptions. (Leo Tiokhin)

Part 4: Strategic ambiguity or unknown unknowns? (Willem Frankenhuis, Karthik Panchanathan & Paul Smaldino)

Part 5: Modeling for metascientists: some first steps. (Minhua Yan)

Part 6: Theories and models are not the only fruit. (Daniel Nettle)

Less-Brief Introduction

Metascientists are becoming increasingly interested in developing theory, particularly…


Leo Tiokhin, Karthik Panchanathan, Paul Smaldino, Daniel Lakens

The beauty and the tragedy of the modern world is that it eliminates many situations that require people to demonstrate a commitment to the collective good.

- Sebastian Junger¹

Imagine two scientists, Kotrina and Amber, who have just obtained their PhDs and are entering the job market.

Kotrina has four empirical papers. She is first author on two, including a publication in a prominent psychology journal, Journal of Experimental Psychology: General. Kotrina has 75 citations, with two papers cited 25 times each—not bad for a newly-minted PhD. …


New blog post:

What institutional reforms will make science more efficient and reliable?

Provides some reflections on our recent paper:

Tiokhin, L., Yan, M. & Morgan, T.J.H. Competition for priority harms the reliability of science, but reforms can help. Nat Hum Behav (2021). https://doi.org/10.1038/s41562-020-01040-1

Update 2021: This post now has a DOI: http://dx.doi.org/10.13140/RG.2.2.22893.51682


Co-written with Noah van Dongen.

Our field is meta-research: the study of the research process itself. When we first got into this field, we both remember reading Munafo et al.’s “A manifesto for reproducible science” and Chambers’ “The seven deadly sins of psychology: a manifesto for reforming the culture of scientific practice” and feeling inspired. “Yes, goddamnit — these people get it!”

A few years later, while browsing Twitter, one of us saw the preprint, “A Manifesto for Team Science”. This time though, the feeling was different. “Really? Another manifesto?” …

Leo Tiokhin

Postdoc at Eindhoven University of Technology, researching how to make science more efficient and reliable. Metascience | Incentives | Evolutionary Theory.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store