Holey Models: A Final Response to Matheus Grasselli


Matheus Grasselli has responded to some of my previous posts (in the comments on this post) and some comments I made on the Facebook Young INET page. His points were as I thought they would be — indeed, I have already dealt with them in this post. For the sake of tying up loose ends, however, I will repeat my points here and make a few general comments about the use of mathematical modelling and numerical probabilistic reasoning in economics.

Grasselli had previously given me an example of how Bayesian probability can be used in the case of HIV testing. I fully agree with his approach because HIV testing and the results derived therefrom are based on experimental data that can be interpreted statistically because it is ergodic. I pointed out to Grasselli, however, that the material economics deals with is non-ergodic. For example, if I want to try to figure out how the Fed’s upcoming “tapering program” (i.e. the ceasing of the QE programs) will affect the US economy I cannot meaningfully use numerical probability estimates.

This is because the Fed taper is a unique event similar to the example I gave before of the probability of a woman calling me tomorrow morning. Grasselli takes up this example in his comment and concedes what I have been saying all along: there is no immediate way we can assign this a numerical probability and so it must be conceived of in a different manner. He writes:

The woman-calling example can be entirely laid out without numbers, exclusively with “degrees of belief” that get weaker or stronger as evidence comes forward. Think of a colour gradation, or the position of a dial, or don’t think about any metaphor at all, just stick with the difference between weak and strong belief and sense of it getting weaker or stronger. You might choose to assign numbers and they’ll be arbitrary as long as all the evidence you are gathering is also qualitative (e.g yes or no answers) and does not have any statistical regularity.

Fantastic. I’ve been saying this all along and I agree. Now, here’s the kicker: the vast majority of what we deal with as economists is data that must be interpreted in this way — i.e. that is not subject to numerical estimation. This means that we cannot use models that rely on numerical inputs. “But,” the smart reader will say, “this whole debate between you and Grasselli began because you criticised him for using such numerical estimates in his models. You said that economic data is non-ergodic and cannot use such estimates and thus his models were doomed to fail.”

So, what is Grasselli’s response to this? It is what I thought it might be: he intends on assigning these non-quantitative aspects numerical variables regardless — or, at least, so I gather from his comment. He writes:

The purpose of assigning numbers is to deal with evidence that is quantitative in nature and comes from phenomena with some degree of regularity. It is only when you have to combine both that you need to use numbers, and in this case the numbers will be anchored by the bits of data that are quantitative.

Take Nate Silver’s election-prediction model. It incorporates both quantitative data with a lot of regularity (polls, etc) and a tons of qualitative stuff, for which he has to come up with numbers just so that they can be incorporated in the model. Even though the priors that he assigns to these bits of the model are not based on statistical regularities, they get meaningfully integrated with others bits of data that do, and in the end produce a probabilistic estimate for the outcome of the election that is far from being pretentious, arbitrary, or meaningless. (My Emphasis)

But this is precisely my problem. You cannot assign this qualitative data numbers. As I said in my last post regarding the woman on the phone example (I quote this at length because I see no reason to retype an argument that I have already made elsewhere; note that we can avoid the whole “betting” thing here and just stick with my criticisms of assigning numerical probabilities):

Let’s take a real example that I used in comments of my last post: what are the chances that a woman will call me tomorrow morning between 9am and 11am and can I assign a numerical probability to this? I would say that I cannot. “Ah,” the Bayesian will say, “but you can. We will just offer you a series of bets and eventually you will take one and from there we will be able to figure out your numerical degree of belief in probabilistic terms!”

I think this is a silly load of old nonsense. The assumption here is that locked inside my head somewhere — in my unconscious mind, presumably — is a numerical degree of belief that I assign to the probability of an event happening. Now, I am not consciously aware of it, but the process of considering possible wagers brings it out into the open.

Why do I think that this is nonsense? Because I do not believe there is such a fixed degree of belief with a numerical value sealed into my skull. Rather I think that the wager I eventually accept will be largely arbitrary and subject to any number of different variables; from my mood, to the manner in which the wagers are posed, to the way the person looks proposing the wager (casinos don’t hire attractive women for nothing…).

Back to my example: what are the chances that a woman will call me tomorrow morning between 9am and 11am? Well, not insignificant because I am supposed to be meeting a woman tomorrow morning at 11.30am. Can I give this a numerical estimate? Well, it certainly would not be 0.95. Nor would it be 0.0095. But to ask me to be any more accurate would be, in my opinion, an absurd undertaking. And if you convinced me to gamble on it the wager I would be willing to accept would be extraordinarily arbitrary.

“But Phil,” some economists will say, “while it may be true that this is a questionable enterprise surely we can forgive it if it makes the model work. After all, surely we can get some idea of the problem by, say, assigning your woman on the phone example a numerical probability of, say, 0.08. I mean, if we are doing this when we are modelling surely these proxies for qualitative evaluation will work. They can’t be THAT far off the mark and besides they will likely only be small parts of a bigger model.”

For the economists who think this I will likely not convince them that they are wrong. After all, they likely have a stake in this game — possibly a financial stake in the form of funding — and it is in their interest to produce a model that “works” (note that I do not mean here that a model “works” as in it “produce relevant results” but rather that it functions on its own enclosed terms).

For those on the fence, however, I would only say this: when you start getting relaxed about undertaking even minor dubious actions in your thinking you will soon find that your entire intellectual edifice has been completely shot through and compromised by such actions. By the time you realise this it will be too late and your work will be so full of tiny holes and microfractures that it will be unable to produce either relevant results or relevant insights.

Since Tinbergen and Keynes’ critique in 1938 the statisticians, mathematicians and econometricians have been knocking on our door every few years saying that they have solved all the problems with the use of mathematical models that integrate statistical data. But its never true. And when you dig down its always the same problems that arise. This is because such problems are epistemological or even ontological; they have to do with the nature of the data we approach as economists. They are not simply due to errors or lack of sophistication; and they will plague anyone who takes such an approach to the point of making their work redundant with regards the real world.

But those advertising this approach will always find followers and funding. Nothing I or anyone else can say will prevent this from occurring. The best I can do is make those who go down what I consider (and Keynes considered) a dark path full of hocus pocus and black magic at least somewhat aware of the problems of their approach. With that, I think the debate is over, because both mine and Grasselli’s positions have been articulated. And so it is up to you, dear reader, to decide.

About pilkingtonphil

Philip Pilkington is a macroeconomist and investment professional. Writing about all things macro and investment. Views my own.You can follow him on Twitter at @philippilk.
This entry was posted in Economic Theory, Philosophy, Statistics and Probability. Bookmark the permalink.

3 Responses to Holey Models: A Final Response to Matheus Grasselli

  1. Matheus Grasselli says:

    Yup. That pretty much summarizes it well.

    My position is: use numbers from data when you can, use qualitative degrees of belief when you can’t, and mix them up in a coherent, meaningful, and predictive way using Bayes theorem when your problem is hybrid, which is to say all the time.

    Your position is: don’t use math in economics because… epistemology!

    • My position is: when you think through the details properly what you consider to be coherent and meaningful will probably end up being incoherent and meaningless and that this is because of the structural problems with applying mathematics to the data in economics.

      But as I keep saying, maybe all those that came before were false prophets and you’re about to nail it Matheus Grasselli. If you manage I look forward to watching your seminars where you present new and interesting facts and approaches that I have never before encountered. If that happens you have my word that I will concede this whole debate and even apologise for my insolence and arrogance. I won’t hold my breath though.

  2. Good posts on this topic. I am a little late to them, but enjoyed. I have written on related topics here:
    “Why inferential statistics are inappropriate for development studies and how the same data can be better used” http://mpra.ub.uni-muenchen.de/29780/1/MPRA_paper_29780.pdf
    May be on interest to some.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s