Lars Syll just ran a rather amusing quote from a handbook on mathematical statistics in which the author — a mathematical statistician — lays out all the cop-out arguments used by those who apply these methods in a dubious manner in the social sciences. I looked up the original book and I think that it might be worthwhile outlining another quote which is, in a sense, even more damning to those who try to use these methods in socials sciences and, most especially, in economics.
Such things [i.e. techniques which can help get over certain fundamental problems with statistical modelling] work only if there is some relatively localized breakdown in the modelling assumptions — a technical problem which has a technical fix. There is no way to infer the “right” model from the data unless there is strong prior theory to limit the universe of possible models. (More technically, diagnostic and specification tests usually have good power only against restricted classes of alternatives.) That kind of theory is rarely available in the social sciences.
Ouch! That’s something a lot of folks in the economics and financial community don’t want to hear! Not only is the author of a major statistical textbook saying that these methods are probably not so suitable for social sciences as some might have you believe, but he is also saying that ultimately the model is needed prior to the statistical investigation. Someone by the name of John Maynard Keynes made this very point a long, long time ago in his On a Method of Statistical Business-Cycle Research: A Comment. There he wrote:
It will be remembered that the seventy translators of the Septuagint were shut up in seventy separate rooms with the Hebrew text and brought out with them, when they emerged, seventy identical translations. Would the same miracle be vouchsafed if seventy multiple correlators were shut up with the same statistical material? And anyhow, I suppose, if each had a different economist perched on his a priori, that would make a difference to the outcome.
Keynes was not, I think, just saying that you needed a theory prior to an econometric test — many econometricians would admit that much (although some would not). He was also saying that an economic theory could not be proved or disproved by such a test. This is because, among other reasons, an economic theory may be valid in one historical period and invalid in another. Thus if an econometric test is run on a twenty year time series and a given causal theory only really applies to the first year of that time series, the test will show the theory to be wrong. Or, take another example: an econometric test is run on a twenty year time series when such a causal theory applies but then in the twenty-first year this causal theory breaks down.
Such is the nature of the non-ergodic material that economists must deal with, as Keynes knew well.
Fixing the Economists 