Is measurement in social science a fractal?

Do we know in social science what it is that we are measuring or does any bit of data we look at on closer inspection reveal more complexity, no matter how close we look, just like a fractal? Another way to put this is to ask whether anything we measure is solid, concrete and absolutely trustworthy, a true ‘fundamental’ on which one can build further theories?

Let us think about this by unpacking particularly important statistics at deeper and deeper levels so as to see whether we get anything concrete the closer we look. I will start with GDP and end up with brainwaves.

Let us start with Gross Domestic Product. It is now measured for nearly all countries in the world and GDP series for some countries go back for centuries. Angus Maddison even constructed series going back to the Roman Empire. GDP levels and changes in it are used millions of times per day in official documents, research reports, cross-country analyses, newspapers, blogs, etc.

You would then think that something as fundamental and widely used as GDP would be like a solid rock of data: something you can count on to be ‘correct’ and to measure something important. Let us see.

How is GDP constructed and what is it? In Australia, one can make the case no-one really understands where the whole number comes from, even though the Australian Bureau of Statistics gives a quarterly number. You see, it is built up from statistics from many different sources involving thousands of data gatherers, as you would expect from an aggregate number about the whole economy: wage information from all the sectors of the economy go into it, import and export information go into it coming from the trading institutions, tax information goes into it regarding investments and capital write-offs, etc. Essentially all the sources of production and income that involve money get sent to a GDP data creation group via several detours involving separate units in other parts of government making up such things as trade figures and capital accounts. The underlying statistics are thus from many groups and I am confident in saying there is probably no single individual in Australia truly on top of how all that underlying data came about.

Still, the fact that there is probably no-one who really knows where every aspect of the number comes from does not necessarily invalidate it, though it does mean one has to trust a whole lot of people to have measured ‘the right thing’ in terms of data they sent on.

Once sent to a central place, a kind of magic is let loose on the bundled production and income information. All kinds of statistics ‘needed’ to get a GDP number are imputed for which there was no information, ie they are made up. This is partially for good reasons: many statistics feeding into GDP are not in fact collected quarterly, such as income tax information. Many sources of information have missing data, such on historical capital levels or levels of imports and exports. The underlying reason for such things often have little to do with supposed laziness of the central statisticians: it is simply not so easy to know what the current value of container freights in the harbour of Sydney is and delays in data processing are partially due to honest mistakes by companies or ships that run late.

The magic does not stop at guessing a few numbers. The GDP people ‘correct’ for how many Sundays there are in their data, what the timing of the holiday was, and many other seasonal influences. After all, the general feeling is that the accident of having, say, 2 more public holidays in a quarter in one year than the next year should not be seen to be the 2% drop in production that this in reality entails. Hence, a kind of ‘holiday smoothing’ takes place. So even conceptually the GDP figures you are fed in the media are not really the ‘level of production in a quarter’. Also, because a lot of information arrives late, is imputed, or is downright distrusted, the GDP people use information on GDP in both the previous quarter and in the subsequent quarter to inform them of what GDP in one particular quarter ‘should have been’.

Think of what this kind of forward-and-backward looking does: it builds in an automatic way in which GDP starts to look like a cycle: it starts to look like a well-behaved wave simply because it is constructed as a complicated weighted average of forward and backward looking information, even if in the ‘raw data’ GDP is much more erratic. Moreover, it means that there are ‘vintages’ of GDP for any moment in time as more information becomes available. There is not just one number for GDP in the first quarter of 2013, but there will be many ‘updates’. So our guess as to what GDP is in 2013 is different in 2013 from what the guess of 2013 GDP will be in 2014, 2015, etc. And these vintages can sometimes vary an awful lot (see for instance here), meaning that a year which seemed to be a boom year can later on be said to be a recession year, and then later again a boom year. 1948 is for instance a year that only became a recession year decades later due to imputations of capital series.

Background documents telling you how GDP and its components are constructed run into the thousands of pages (glance here for instance). And even they do not have all the information you would need to get on top of it for these handbooks tells you about data manipulations and definitions, not the construction of the ‘raw data on the ground’. Hence, if you look closely, the whole construct of GDP starts to look more and more shaky, quite apart from whether it measures ‘aggregate production’ (which is another discussion, usually answered with ‘no’ as GDP fails to measure all the goods and services that are outside of the tax system, such as the environment and home production): it is not a solid and unchanging number if we zoom in, but rather a moving target. We thus don’t really know what the number means. We just know we want it to be high.

You should thus realise that almost any seminar you go to where people make a big deal about changes in GDP from one quarter to the other is dependent on these statistical conventions and tricks: a lot of STAR, VAR, and other models ending in ‘AR’ (which are estimated on a daily basis at the RBA) that are use to analyse quarterly variation in GDP run the risk that they might merely re-discover how the data was constructed rather than uncovering something deep and meaningful about our economy.

Let us then look a step deeper and pick out a very small particular aspect of what ‘should be’ in GDP, say education production. Now, intuitively you might think that production in education should be measured by how much is learned and thus some measure of the increase in knowledge held by those educated. Alas, no. That is far too hard to measure. Why is it too hard to measure? Think about it: a kid does not just learn from teachers, but also from peers, parents, own discovery, tv, etc. How would you then assign any measured increase in knowledge to the teachers? Basically impossible, so we don’t even try.

So how do we then measure educational production? Simple: we count the costs of education. So we add up all the salaries of the teachers and administrators and all the costs of the building and all the costs we see of the libraries and books. And we then call it production.

What does this mean? Think about it: whether or not there is any learning, the salaries of the teachers count even if they would not show up at school and have no pupils. It is not production in any sense except in the sense that it counts for GDP. Worse, changes in the measured costs turn up as increases in production. Hence a general increase in property prices will show up as increased value of school grounds and increased costs of building schools, and thus as an increase in education production. Note also how fragile and hard to measure this ‘property price’ aspect is: it is not easy to know what the current property prices are because not all buildings get sold every day and the price of any building can vary simply because different people showed up to buy based on fairly accidental circumstances. So not only does one have to guess the prices in reality but one has to be wary of measured prices too. Note also that of course it is not easy to say who is a teaching administrator and who administrates something else, such as, say, the ministry of education building, so there is a large amount of fudge in terms of what counts towards education versus other things.

Once again therefore, educational production disintegrates as a solid measured concept if you look close at how it is measured. Not that that prevents ‘net human capital stock’ to be compared over time and across countries for a century, but by now you should realise that lack of certainty in what data means does not stop it being used. You can guess what the underlying uncertainties means for research into such things as the ‘cost-effectiveness of education’: if you don’t actually measure anything looking like real production then good luck with being very precise about cost-effectiveness of that production. Indeed, I hope from now on you look with a bit more scepticism any time you see an analysis of ‘GDP and education’, of which there are thousands.

Let us go the next measurement layer and consider something as seemingly clear and fundamental as being female, which the vast majority of workers in education in Australia are. Surely here we have something solid: barring a few mistakes we can’t get gender wrong, can we?

Think again. If you zoom in to the concept of gender, it is suddenly not clear at all what one is measuring: is being a female about not having overt male genitalia? Is being a female about wearing a dress? Is being female about being less aggressive and looking after the kids?

Put this way, it should become clear to you that measuring gender on an all-or-nothing basis, which is what statisticians do, is a complete misnomer. Not all females are equally small, nor do they have equal levels of testosterone, nor do all have caring roles inside their families, nor do they perceive themselves in the same way as every other female, etc.. Indeed, even if you just zoom in on just the supposed genetic basis of ‘gender’ you do not get clarity: not all women are XX and not all men are XY. You do not just have all kinds of ‘inbetweens’ (XXY’s and the like), but even within the XX and the XY groups there is actually a huge amount of variation as to how many ‘gender relevant genes’ are ‘turned on or off’ depending on things as trivial as a good night’s sleep, and of course there are actual genetic differences within the genders: genes, turned off or on, differ between people.

Even the things that all ‘females’ truly have in common disintegrates when one looks closely: there are laws particular to ‘females’ (ie they until recently could not fight as frontline soldiers), there are toilets just for them, and as a quick label assumptions are made about their roles in life. These are solid things, no? Well, these things too differ in time and across space: the assumptions made about gender differ from year to year and street by street. Army duties and possibilities change over time and by army unit. Even toilets varies, as do the amenities in toilets. So on closer inspection even the things that seem truly the same for all females are not in fact the same over time and across space.

So what the statistician conveniently lumps into the all-or-nothing variable ‘female’ actually disintegrates as a solid concept if you zoom in. Note that this does not prevent nearly all empirical social scientists from happily putting ‘gender dummies’ in their empirical equations and tables as if it is a solid concept meaning the same thing for all entities labelled as ‘females’ in all years in all countries. Strictly speaking, as with all the higher-up concepts like GDP and educational production, this means that social scientists use variables with a high degree of non-random measurement error in their analyses. To a purist, this invalidates all of them, not that that has ever stopped us running these analyses. And yes, I run regressions with ‘female dummies’ all the time so I cannot claim to be holier than any others in this regard. But if we are truly anally scientific about this, using gender dummies instead of explicitly recognising that it is a variable that only with a great degree of non-random measurement error might measure some underlying fixed construct means we cannot be assured of the robustness of any estimation results involving a gender variable. The only good news to this damning reality is that the same problem occurs with any other variable we put into our analyses.

Zooming another layer deeper, let us now think of the brain activity in a particular part of females, say the cerebellum.  The uninitiated social scientist might think we have finally arrived at a level where we get precision in measurement and interpretation, and I regularly meet social scientists exited by the certainty they soon expect brain scans to give us that was not found at any other level.

Alas, not so. Not just does the cerebellum alone have more than 50,000,000,000 neurons, but each of these neurons has about 10,000 connections (dentrites) to other neurons, including neurons in the rest of the brain. Do we measure the individual electrical and chemical currents between all these individual neurons and connections? Of course not. All we manage to measure, and even then with great difficulty, is how much total activity there is in whole areas of the cerebellum. In these large areas, we are effectively summing activity over billions of neurons in thousands of functional groups, meaning that all we measure is the aggregate of thousands more specialised groups.

Even the much smaller functional groups (whose individual activity we rarely measure) have all kinds of functional roles, including motor memory but also involved in emotions and spatial awareness. A single group might have as small a basic role as calculating how to flex one of the twenty-odd muscles in the hand, yet the brain does not really work on the basis of a single clump of cells calculating something in isolation of other groups. This is basically because a functional group includes the various roles of all its individual neurons, which are connected all over the brain making the functional role connected to the whole of the brain. You should thus not be surprised to know that activity in the cerebellum has some link to human emotion (movement IS sensual!). Hence, the cerebellum has hundreds of functional roles and is involved in literally millions of different pattern-recognition activities. Good luck truly unpicking something as highly aggregated as the activity in a large area of the cerebellum! Indeed, as the pattern of connections differs for any two humans based on their life experiences no two brains are the same.

Reflect further on how integrated any individual mental activity is, such as a particular emotion (which the cerebellum is also involved in): emotions are connected to evaluations of circumstances against a historical database and all kinds of mental habits. In a sense, the displayed emotion thus depends on not just things like self-image and previous experience but also on all the immediate inputs of the whole body and sensory experiences. Hence, what the cerebellum does depends on the whole of the rest of the brain, what someone had to eat, the circumstances in the womb of the mother, and the intensity of light in the room.

I hope you can see that measuring all that to a degree that we would truly understand the cerebellum is as hopeless as correctly measuring and truly understanding GDP. Indeed, if that female is a teacher of economics her understanding of GDP will be involved in her cerebellum activities!

So not only do we not measure what ‘truly’ goes on in the cerebellum, but even conceptually it would require a being far smarter than us to interpret and use what really happens in the cerebellum.

So for the whole spectrum of social science, from what we measure at the grand aggregate level (such as GDP) down to the smallest measure of activity we have (brain patterns), we end up not truly knowing what we are measuring when we zoom in. Every time we look closer, the uncertainty is just as big as when we zoomed in at any higher level. Human life and human behaviour seems like a fractal: it does not get any clearer the more you zoom in. It is really very frustrating.

What does all this mean for how to do ‘proper’ social science and how to interpret all our claims of certainty? It means that all our stories based on ‘fundamentals’, ‘first principles’, ‘solid data’, ‘undoubted measurement’, ‘micro-foundations’, etc. are just that: stories. They help us limited beings muddle through a reality we have no hope of fully understanding in terms of some undoubted underlying truth that we will someday measure properly. Stories of certainty are then just particularly simple stories. Potentially useful, but never 100% true.

Whilst this fundamental lack of certainty in anything we do or measure in social science does not mean that we should give up on abstractions or measurement, it does mean a bit of humility is in order.

47 thoughts on “Is measurement in social science a fractal?

  1. Isn’t that why we have error terms? You find an atomistic level and accept that it is right enough, like your example of gender. Given that a lot of social science data is fairly linear and hence you are not getting terrible interactions from small effects you didn’t care about, this sort stuff is not going to hurt your story too much. It’s worthwhile noting that this sort of thing exists everywhere, not just social science. Atoms, for example, are obviously atomistic enough that you can really learn a lot without worrying what is below them.

    To me the problem isn’t that there are lots of levels of things (there is for everything), it’s that lots of problems have no decent atomistic level from which to start at and the data in many social science problems is so messy that the error coming from a level below is the least of your worries.

    • Nope, error term do not allow for measurement error in ALL of the variables in an equation, just the one on the left hand side. No easy escape, I am afraid! If you accept the argument in the post, every estimation you have ever done is most probably biased….


      I agree with that! I clearly failed to explain in the post that the measurement issue means we lack any stable and replicable atomistic level.

  2. I think that’s why context is so important. It’s important to understand the assumptions and conditions that are required for a particular measurement or model to have the meaning it is imputed to have. An important part of using any data or model is having the discipline to consider whether it is appropriate for a given purpose or remains relevant in a given circumstance. The quality of the data or model is at least as important as any particular values. Understanding the quality will often require knowledge and experience well beyond the immediate context of the quantitative task.

    With social science this quality is going to be messy and descriptive, not something as simple as measuring or calculating error values. It might be tempting to assume that measurements or models in related areas have significant relevance to each other (e.g. fMRI scans and behavioural psychology), but that’s about as naive as assuming a PowerBook will functionally connect to an alien spacecraft. There might be some crude way to connect with a model of an underlying phenomenon, but that may only constrain the new ‘comprehensive’ model with the assumptions and limitations of the underlying model in a way that makes it useless. The chaotic nature of the systems involved will often mean the direct observations of a phenomenon will reveal more ‘truth’ than attempting to model it based on underlying phenomenon.

    The challenge in dealing with complex systems is juggling the need to minimise the context in order to ensure practicability of modelling or measurement, the need for one’s insight or understanding to fit well within that context, and the need to ensuring the applicable context is sufficient to make the effort useful. This means that many quantitative goals may be within our theoretical reach, but remain outside our practical grasp. For some things, particularly explaining when and why quantitative measures are useful, we’ll just have to learn accept frantic hand waving.

    (p.s. ‘females’ can get uppity if you refer to them as such)

    • “we’ll just have to learn to accept frantic hand waving.”

      absolutely, but in order not to lose our souls as social scientist we have to be open about this when teaching new members of the tribe, even if they yearn for certainty. Some examples of the limits to measurement and the imperfect applicability of any model should be part of the standard curriculum of higher-level academic teaching.

      • We teach the idea of validity and reliability to undergraduates in psychology (I imagine most places do), although curiously many students find the idea of validity hard to understand. The idea of models and how one could go about testing them, alternatively, is not done very much (there is no philosophy of science requirement at most places), so most students have no real idea how to test something well (indeed many don’t understand they are testing theories versus the actual thing).

        • “there is no philosophy of science requirement at most places” .. is that true?

  3. Good stuff! Web of Indra comes to mind.

    And even if it was possible to ‘know’ exactly the exact state of every neuron, and every interlinking web of neurons, in a particular brain at a particular moment (and to somehow ‘understand’ this data) , it still would not tell anything much about the emotional/mind state, as felt reality, that that set/pattern of neurons isomorph to- wrong level of grain- different meaning.

  4. If you zoom in you can’t say what female is. Yes, but what’s special? I agree with Conrad: surely all knowledge is like this. If you try to zoom in on mass or time or gravity you get the same problem. They spent 13 billion Euros on the LHC in order to zoom in on mass. If we can’t say what mass is (and physics is cheerful and productive in that condition) why worry about saying what a female is?

    Whatever the problem may be, it cannot be that there is doubt about the meaning of concepts. Cannot be. The natural sciences measure things without concern for their meaning and if the social sciences want to emulate them, they too must work out how to be comfortable measuring concepts that go blurry the moment you zoom in. Evidently, to worry about the meaning of concepts is to misunderstand the meaning of understanding.

    Perhaps the problem is that the word “measure” is used carelessly. In social science the only measuring is via a Likert scale or similar—i.e., people’s opinions. Otherwise what social science calls measuring is actually counting.

    In natural science you seldom count things. You measure a property. For example you measure the dip and strike of a fault. The number of faults is less interesting and if for some reason you do want to count them, you will get into much more difficulty of identification than when counting male and female human beings. There may be exceptions (e.g., an evolutionarily stable state is worked out by counting incidences of the various types) but generally science measures properties such as weight, temperature, etc. I’m not saying it is bad to count things. It’s a great idea—but counting is bureaucracy, not science.

    So perhaps the real problem is that before social science can be really effective, i.e., hope to understand as hard science does, it will have to work out how to measure properties of things, not just count them. Then it won’t worry about the nature of the things.

  5. Mike,

    you say that all knowledge is like the uncertainty of what a female is, including the issue of mass in physics colliders. No it is not. Here is the difference: In the large physics collider you are banking on the assumption that the underlying elements of, say, single protons that you shoot at each other are always the same: any two protons derived from hydrogen are identical in terms of constituent elements. You might not know the constituent elements, have a great difficulty measuring aspects of a proton, but the assumption that any two are the same ‘works’ in the experiments of the colliders. It is that assumption of comparability that allows you to use statistical inference on the debri of a million collisions between pairs of protons.

    Now, which two women are the same in their underlying characteristics? No pair of women is identical! So treating any two as if they are the same (even conditioning on a few other ‘measured traits’), which is implicitly done when one throws them into a single estimation equation, is going to get you into biased inferences.
    The issue of how measurement error on the right-hand side of an estimation equation gets you into trouble is a very old statistical point. You would have to know the nature of the measurement error if you wanted to overcome it. And how are are you going to learn that?

  6. “it’s that lots of problems have no decent atomistic level from which to start at…”

    Can’t really be right. Galileo and Newton knew nothing of atoms but the knowledge they generated helped change the world and is still in daily use.

    • Mike, my position is that for some things there is so little error you can happily use an atomistic level, and not worry too much about the error that perculates up (there is in fact a fair bit of maths done looking at constraints on this — none of which I know much about!). Alternatively, for some problems, it’s very hard to define this level (hence “lots of problems”), and then it is a problem. I think many social science problems of general interest that we’d like to know the answer to fall into this category. It also depends on what you are striving to achieve. Newtonian physics, for example, is all fine and dandy for any number of problems. But if you want to send a rocket to the moon, you need to worry about more.

      This means I really think you do need to worry about the context — or what the problem is (unlike you). For example, sunshine makes me happy, but if I was interested in happiness (not a bad thing to think about), sunshine wouldn’t be a good proxy, so I’d still need to call it sunshine and not happiness. So if I have a thing called happiness, I need to think about how I can measure it, and indeed, what type of error the best measurement I can come up with would have.

  7. Paul,

    The physicists bank on the assumption that two protons are the same. The statistician banks on the assumption that all women are the same. What’s difference?

    If the purpose is to decide where and how big a new female toilet should be, the assumption that all females are the same will be valid. It “works” just as the physicist’s assumption that protons are the same works in the experiment.

    That’s my point—there is no point in worrying about what something IS. Physicists don’t and social scientists shouldn’t.

      the difference is that we know that the statistician wrong at the outset in that none of the underlying characteristics between any two women are the same when we measure those (however imperfect!). The physicists have had it easy in that their assumptions of comparability do seem to hold up much better when they unpack things: two metal balls weighing the same and of the same shape will roll off a slope at the same rate, give or take very little difference. Near-perfect measurement. On the other hand, two economies with the same underlying level of production can be easily 50% apart when their GDP is measured by current ‘best practice’, for instance because in one country all the home production is unmonetised whilst it is monetised in the other.

      I am, truth be told, quite happy you read this post Mike. You see, it goes to the very heart of your often stated advise to social scientists to be more like physicists. I hope this post convinces you that that is an illusion: what on earth does ‘falsification’ really mean when everything you measure has, say, a 50% margin of error? One is looking at a whole different ball game when one accepts the impossibility of good measurement in social science. Conrads take on it is essentially to then use more sources of information (context). What you have said so far sounds like you only want to deal with those aspects of social science that involve near-perfect measurement. An empty set, I am afraid.

  8. But Paul, you are talking of a difference of degree, not of kind. Of course, measurement of GDP is more approximate than measurement of rolling metal balls. “More”—a difference of degree. The metal balls you say are near-perfect. “Near.”

    My advice would indeed be that social science should adopt the methods of natural science. It has been so successful that deviating from its approach should require justification. Yet the opposite is the case—as this discussion indicates. You call for humility—well, social science should stop going off on its own frolic. As your post made clear it isn’t even working. You say physicist’s assumptions hold up “better” but is this a reason to deviate from the tried and true? If deviation is justified by the measurement errors being higher in social science, at what point is the standard abandoned? 50%? It is not humble to make an assumption like that. If the error gets down to 25% should we revert to natural science methods?

    Also, perhaps the difference is not really as great as the example indicates. Those metal balls are not real-world phenomena; they are artificial, made to be as near perfectly spherical as possible. If they were natural things, say rolling rocks, measurement of them would start looking like the measurement of GDP. You can’t manufacture an artificial, near-perfect, GDP. What you can do is make a mathematical simulation and you can measure in it with a perfection exceeding even that of the rolling metal balls. Economists do this sort of thing; the rest of social science doesn’t. That’s why economics is relatively successful—it works from theoretical models.

    Physics is often the most straightforward touchstone but above I used geology. One thing natural scientists don’t do is count things. They measure them. Just imagine trying to count geological faults: Is this a fault or is it just some local fracture? Do we count this as another fault or is it an extension of that one? Counting women would be a doddle. I don’t know if geologists ever do count faults but if so (a) I bet they can never agree and, (b) I bet nothing much depends on the number, i.e., they don’t do regressions and draw conclusions from them.

    Geologists are just not interested in counting the number of faults. They measure their known properties (strike, dip, direction) and they draw plenty of conclusions from that. I suppose if the properties of females matter for some purpose you need to measure them. Actually I cannot think of any instance where those doubts about the meaning of woman would play a role in social science. Doctors measure things—hormone levels or whatever. They don’t worry about the definition of female or whether they are all the same. They are doing real science and they don’t count them.

    Natural science basically doesn’t count things, ergo social science shouldn’t either (if it wants to be science, not bureaucracy). When you measure GDP, it seems to me you are measuring, not counting, and your problem is the complexity. That itself is not unscientific: you measure various things and combine them to “GDP.” 50% is a big error but no measurement is ever perfect in any field of human endeavour. It’s just a matter of degree.

    Yes, Newton was improved upon and falsification is another characteristic of natural science where social science deviates: nothing in social science is ever improved on. If I were king I would decree that social science papers must state falsification criteria or it’s off to the dungeons. Economics, by exception, has improved because some notions have been shown to be false. Surely the formulae for GDP has been sharpened by precisely this process. This effectiveness is made possible because economics works from theoretical models and measures, rather than counts, things.

  9. The impediments to making progress in the social sciences start well before we even get to measurements.

    Think about theories of power in the political science for instance. Social scientists can’t even agree on a definition of power, for example some include and emphasize “influence” whilst others focus on “coercion”. Then you have at least three main groups of theoretical approaches- pluralist, elite and conflict – and with each passing year these approaches metastasize like malignant cancers, get sideswiped by new academic fads then backflip and do somersaults without making any measurable progress. The only things that remain constant are (a) nothing like a consensus on anything ever emerges and (b) the thicket of increasingly abstract and incomprehensible theory becomes less penetrable with each passing day.

    Another main problem is that everyone has skin in the game, and as Hume noted long ago, Reason is a Slave to the Passions. That is to say, theorists marry a theoretical approach early on because of their pre-existing beliefs and do not leave it easily because it is so tied up with one’s fundamental values and feelings about right and wrong, good and bad. You can’t say the same about the dry subject matter of physics (although even the development of physics is subject to human foibles).

    A third intractable problem for the social sciences is the fundamental nature of man and how we should treat it in theory. By this I mean the free will versus determinism argument. Obviously if the fundamental unit of the social sciences- man- has free will some serious limitation is placed on what the social sciences can achieve.

  10. Mel -

    “Social scientists can’t even agree on a definition of power…”

    Physicists can’t “even” agree on the definition of mass. Or a definition of time. Or gravity.

    Definitions are useful for teaching children established knowledge but science does not depend on definitions. It can’t define anything. Hence definitions are not “before” measurement since science measures everything it can.

    Hard to accept? It’s the way it is. Definitions are nowhere in science. Could geologists agree on the definition of “fault’? I bet they don’t even try. I doubt you could get a serious number of people to agree on the definition of “chair.” So what chance “power”? There is no point in trying, since looking to definitions is evidently not the scientific approach. I think psychology publishes whole books of definitions. I don’t know if they are rubbish but I do know they are not science.

    I am not persuaded about skin in the game since some natural scientists get very attached to their favourite theories—and some social scientists are quite stand-offish.

    You have lost me with the free-will bit.

  11. Mike Pepperday:

    “I am not persuaded about skin in the game since some natural scientists get very attached to their favourite theories … ”

    Maybe so, but wars are not fought over such theories; to my knowledge string theorists didn’t kill their opponents when they were in the ascendancy and now that they are in retreat, they haven’t resorted to a counter insurgency.

    On the other hand, millions died last century in wars that went to the very heart of the questions political scientists seek to answer, for example do we have a capitalist ruling class that exploits the working class as per the labour theory of value and will the latter class eventually smash the former and establish a society based on equality?

    Sendero Luminoso, a notorious Marxist guerilla group, began in the Universities of Peru; its leader was a philospohy professor. Again, I can’t think of any physics equivalent, not even among the reputedly headstrong string theorists. Also, what is the string theory equivalent of Antonio Gramsci’s Prison Notes that still resonates in critical theory circles? And how many physicists cf. social scientists have been tortured, imprisoned or killed by oppressive regimes based on theoretical disputes over the last few decades?

    Lame stuff, Mike. Very lame.

  12. Mike:



    “Definitions are nowhere in science.”

    Definitions can’t define anything? Definitions are nowhere in science? Umm, gee, Mike, how could anyone measure something that isn’t defined? I seem to recall using plenty of established and accepted definitions in high physics physics and chem to solve problems. I can’t say the same for ungrad social science about anything, ever.

    I also seem to recall that many hard science concepts, like continental drift were conceptualised and defined decades before technology made measurement possible.

    “Hence definitions are not “before” measurement since science measures everything it can.”

    Say what? If that was the case a physicist would be as gainfully employeed counting his own nasal hairs as he would be using the applicable technologies and procedures in order to pin down the gravitational constant (something which you imply is an unfructuous endeavour, btw, since it can’t be defined).

  13. There is a bit of false opposition going on. All systems of meaning that derive from ‘isomorphism’ are grounded in ratio/ relative relationships not the ‘atomistic’. All measuring involves counting in a meta sense- if it fits exactly in a ‘meter’ then ‘its’ a meter. Measuring something that is sentient- involving a lot of self-reference self-replication in a complex web of some times ironic meaning/representation- is a different level of project to that of studying the behavior of sodium atoms or beams in a steel structure.

    Doesn’t mean it cannot be done, just that it needs some different or extra tools, metaphors, to put things in.

  14. Definitions are nowhere in science.

    Definitions are important in science. The difference is that in science definitions are founded in reliably repeatable experiments. For example, an abstract concept such as mass will have its definition capable of being be demonstrated by comparing weights on a set of scales, or observing the impact of snooker balls. In social science definitions are far more subjective and are rarely founded in anything tangible. In hard science you need evidence to get people to accept your theories, in social science you just need a good argument.

    You still have to define a concept and how it will related to the real world before you can construct an experiment to measure it though. That’s not to say scientists can’t debate about the boundaries of words, just as anyone using language can, yet this isn’t usually a debate about the underlying concept. During the debate about whether to call Pluto a planet or not, there was no disagreement about what Pluto actually was (i.e. a chunk of rock going around the sun its own unique way).

    I am not persuaded about skin in the game

    Scientist are not immune from having skin in the game. There’s the Plank Quote about how “Science advances one funeral at a time. However, a key part of studying science is learning how to measure things in the real world for yourself. This allows each new generation to form their own understanding of phenomenon, as their understanding will be founded in observation. Studying social science however, typically requires learning how to frame your ‘novel’ ideas in a way that binds them to widely accepted theories of others. Once someone has developed their own ideas within such a framework it becomes much more difficult to break with the past.

  15. Kees van Deemter
    Forgive the oxymoron, but how do you define vagueness?

    “A vague concept allows borderline cases. The potential confusion is that people think vagueness is when they don’t quite get what someone means. For people in my area of logic, it’s actually a much narrower phenomenon, such as the word “grey”. Some birds are clearly grey, some are clearly not, while others are somewhere in between. The fact that such birds exist makes “grey” a vague concept. The vagueness does not arise from insufficient information: some concepts are fundamentally vague.

    On the other hand, if I say that I have fewer than three children, that’s not vague. In fact, it is the opposite, it is “crisp”. It is true if I have zero, one or two children, and it is false if I have three or more.”

  16. You have lost me with the free-will bit.

    Assuming free-will when attempting to define or model social phenomenon can blind you to what’s really going on. The problem is the assumption explicitly rules out the significance of causative links between environmental factors (physical, social, cultural, etc) and the way people think. As many people have a cultural, emotional or theological bonds to the notion of free-will, it is a trap that they often fall into. The end result is that people make naive assumptions convenient for their purposes such as that people behave rationally.

  17. Desipis –


    By definition I mean some words. As such there is no agreed definition of mass. There is no dependence on word definitions anywhere in science. Therefore, in social science, if you define power—say what it is in words—you are not doing science. That is point here: they can’t “even” define power. A scientist does not try.


    No. There is no definition of mass. There just isn’t. There is no definition of time but it is readily measured.

    How it is related, yes. That is crucial. To tackle power we must see how it relates to the real world. Is not every science theory an expression of the relationship between two or more objects (or concepts)?

    “there was no disagreement about what Pluto actually was…”

    “No disagreement.” That’s another matter. Not having a disagreement is a different thing from agreeing on a definition. Indeed, not having a disagreement holds for mass and time—and everything else, I suppose. No one has a disagreement over them except those who argue over definitions. So stay away from definitions.

    Mel –

    “how could anyone measure something that isn’t defined?” Well, they do. Get used to it. As a genuine question (not a statement of opinion) this should be thought-provoking.

    “I seem to recall using plenty of established and accepted definitions in high physics and chem to solve problems.”

    Quite. I did say that definitions were useful for teaching children established knowledge. Pick up a different text book and find a different definition but no matter; nothing depends on definitions.

    “many hard science concepts, like continental drift were conceptualised and defined…”

    Conceptualised yes, defined no. People conceptualise mass and time. Then they measure. I expect that at conferences the geophysicists have hefty arguments about just what constitutes continental drift—i.e., argue over the definition. But nothing scientific depends on the definition.

    Something bureaucratic might depend on it but nothing concerning scientific understanding . Bureaucrats need to count things so they need definitions in order to know what is in and what is out. As I discussed above, the geologist who tries to count faults has a constant definition problem.

    It seems, then, that for the purposes of measurement scientists don’t disagree over concepts. But they will disagree if they try to express their concepts as word definitions—except where the definition is a short-cut for learning established knowledge.

    When it comes to power, that is not established knowledge. Thus it is not science to begin by defining it and then making subsequent discussion dependent upon that definition. We have a century of failure using this approach.

  18. A clade is everything below any binary fork in the evolutionary path. That would be my definition of “clade” and though it’s not my game I shouldn’t think anyone would dispute it. I don’t know what you are getting at by saying a clade is a definition. Clade is a single word, not a definition.

    • Kees van Deemter puts it better than I can.

      Put a magnifying glass to many scientific concepts and you find vagueness. Take the idea of “species”. For centuries, biologists searched for crisp distinctions between species. A common definition today is to say that two animals only belong to the same species if they can interbreed. But if A can interbreed with B, and B with C, it doesn’t always follow that A can interbreed with C.


      Should we give up on the concept?

      The notion is incoherent, but biologists continue using it – with a pinch of salt. Richard Dawkins calls this tendency to think in discrete categories “the tyranny of the discontinuous mind”.

      Sorry but meaning/definitions are inescapable .

  19. Mike,

    like John, I think you overstate your case when you say other sciences do not have definitions. There is a long tradition of definitions in many sciences, including the French around Napoleon’s time saying that a particular piece of platinum defines a kilo, to the modern day definition of a second as ‘The time needed for a cesium-133 atom to perform 9,192,631,770 complete oscillations.’.

    Now, what is true is that it is the measurement apparatus that ultimately defines things: kilos, time, gender, and GDP are all in fact defined by the procedure to measure them, even though in all cases there is an underlying notion of meaning that is itself dependent on the supposed relation of the measured concept with other concepts.
    The rub of the post is that in the case of social science the underlying characteristics of any two cases that give the same ‘measurement’ of a concept can differ by as much as 50%: the underlying relations are not the same for two identical measurements. For instance, the ‘income-gender’ relation is not the same for any two women because the underlying characteristics important for wages are not the same. Identical measured level of gender, different set of underlying characteristics of importance in terms of relation with other concepts.

    This aint true for the physicists: any two cesium-133 atoms oscillate just as fast, are just as heavy, have the same number of protons and neutrons, react in the same way to other substances, etc. Try making that argument for any two women, any two countries with the same GDP, or any two people with the same level of ‘measured education’ and the relations with other concepts.

    Hence the physicists have it easy. They are lucky enough to have found forms of measurement that lead to comparable and replicable results. The task of the social scientist is far harder. Telling the social scientist to build theories and then drop them if they are demonstrably falsified is then basically pretty useless: no theory i know of in economics has been able to withstand all data. We are more in the world of ‘this theory seems supported by 80 careful papers, refuted by 10 other careful papers and 15000 badly done papers’. What do you do then?

    Time to drop the facade hence that social science can be or should be like the physical sciences. Its a different ball game. Doesnt mean we should not have theories and attempt as best we can to think of ‘robust theories’ that make sense of a lot of vague data (I am a big fan of good theories), but forget about the whole certainty business.

  20. Paul
    There is some truth in what mike is saying, the social sciences have been moving into areas that are really not a fit for a scientific approach for example Visual Anthropology (and even worse -visual cultural studies) has been invading and taking over art history for some time , the result is ‘science’ about art that is science without semantics – nonsense (without the fun).

  21. Paul
    Yes, measurement in the social sciences can be seen as a fractal and we all – physicists and social(scient)ists, should start playing the new ball game and modelling the world as a complex dynamical system.

    I was a total convert when I came across the whole dynamical systems theory stuff back when I was doing the PhD. The possibilities for applying it to human behaviour were exciting, but challenging for me because of the math required to do anything except speculate wildly and at that stage (10 years ago) it was very difficult to reduce the ideas to a decent hypothesis that could be tested.

    But I just came across a review of a new book by John Holland, “Signals and Boundaries: Building Blocks for Complex Adaptive Systems,” that seems to provide a bit more substance to the theory and some more foundation for attempts to use the insights to model human behaviour.

    The reviewer, Chris Adami has some interesting research happening also.

    • ah yes, complex systems theory. One of the many many ‘new approaches’ to have promised so much and so far delivered so little. Its failings in economics are many: complex systems theory neither improves the measurement nor does it take as its basic unit of observations (signals and networks) something that has obvious meaning and interpretation. Its impact on economics has thus so far been approximately zero I am afraid. This is not to say it has no potential uses if done well (I think it has potential for analysing health systems and failed states once you combine it with some equilibrium thinking), but the marketing sell that its proponents put on it is over the top. If only things were that easy.

  22. john r walker –

    “Sorry but meaning/definitions are inescapable.”

    Sorry, but this conclusion is precisely contrary to the text you quote.

    First let me note that where Desipis wanted to elide the distinction between definition and demonstration and relation to the real world, you want to slide it together with “meaning.” I cannot imagine we could ever “escape” meaning.

    Van Deemter says about species what I said about geological faults. Exactly the same. The definition doesn’t count for anything. Scientists know the MEANING of “species” and science merrily proceeds. When we try to DEFINE it we get into strife but science doesn’t care about that. Said Newton: “I do not define time, space, place, and motion, as being well known to all.” He knew.

    Definitions are required for counting, which is bureaucracy not science. When van Deemter talks of defining species A, B, C, he’s trying to count them: Are there two or are there three? As he says, the biologists continue using the concept. For them the definition is irrelevant.

    A definition is a bunch of words which purport to tell mother nature what she is. She is utterly disdainful. She doesn’t reveal her secrets in response to attempts to dominate—as when people make investigations of her dependent upon some human definition. Nature is about relationships. To investigate “power” (for example) you have to investigate power relationships, not define the word.

    Definitions are nice. They convey meaning. We’d be lost without them. But language is one thing and science is another. It’s just a fact and your van Deemter quote confirms it perfectly. We have no reason to think we will ever get anywhere in social science as long as we insist that we should agree on definitions. The only way we agree on a definition is when it is stated in an act of parliament. That’s bureaucracy.

    To make definition prior to measurement is to ignore the lessons of two centuries of successful science; it must surely rule out scientific progress before even starting. Economists, note, don’t do it. Economics concentrates on relationships.

    I will respond shortly to Paul’s comment (about Napoleon’s kilogram) and perhaps others.

  23. I, too, reckon system theory, complexity, and also network theory, are hollow. At least I have never been able to get anything from them. I once regurgitated some systems theory in an exam and the lecturer complimented me on it later. I thanked him and kept my mouth shut.

    Paul –

    I did not say that “sciences do not have definitions.” That would indeed be an overstatement. I am talking about the social science mistake of defining the object it is trying to understand—which science never does. Is it a natural object? If it is, then a definition counts for nothing. Caesium is caesium is caesium and no definition has the faintest effect on that.

    A kilogram is defined. It is not a natural thing. It does not exist outside of human beings. The kilogram is defined as “That-there lump of noble metal in that-there Paris vault.” Everyone agrees on the definition and there is no problem. When a problem arises it is resolved by a new agreement at the next international conference.

    That’s not science. It’s pure bureaucracy. Bureaucracy is as essential to science as it is to everything else (doesn’t get half the credit it deserves) but it isn’t science. It isn’t the understanding of nature. It’s a different thing.


    Well, a kilo is not defined by any measuring apparatus. It is that hunk of metal in that vault. The hunk of metal is a “kilo” only because we SAY it is. Verbally. So leave kilos and all units of measure out because they are not natural things; they are necessarily defined in words.

    Consider your next item, time. Now I agree with you (mirabile dictu) except that the word “defined” is a bad one. Most people think that a “define” means some words, some verbal statement. That is the sense when we want to define “power.” You are saying that things are “defined” by the measuring procedure. Well, I have been harping on measurement. Measure time, do not attempt to verbally define it. In order to be science, social science has to measure power without defining it.

    But I quibble, too, with “measure” here. More generally, I would say we KNOW (rather than define) what something is when we know its relationship with something else. We know what mass is because we know its relationship to force and acceleration. Presumably, knowing the relationship would always require some kind of measurement, but more important: the relation to something else is actually how we know the object, or concept, exists. I think that the only way we know thing A exists in a scientific sense is if we know its relation to thing B.

    In other words we do not KNOW what power is—in a scientific sense—because we do not understand its relationship to anything else. We do not know if it exists in a scientific sense. Consider gravity: before Newton most objects possessed a downward tendency and some, like flame, had an upward tendency. These imputed properties, so obvious to everyone, do not exist scientifically. Perhaps this applies to many social science concepts. The only way to find out is to work out how they relate to other things.


    Yes indeed. As I see it, scientific knowledge takes the form of an interrelationship of two or more objects. (This applies to economics, too.) The objects cannot be defined in words to everyone’s agreement and nothing depends on any verbal definitions (same in economics). That is the vital thing. Units of measure can’t exist without a definition but natural things exist independently and are not defined by human words.

    Thus science can never be about one lone thing (species, clade, time, whatever) because if it were, that thing would have to be defined in order for us to communicate. The moment you do that, you are not doing science.

    • I agree with this for, say, 70%. Yes, concepts acquire meaning via relations and yes economics is all about causal storylines. Yet it would go too far to say that we ‘know’ concepts through any particular theory. It is more a process and it is more vague than that: we have hundreds of stories and internal images as to what ‘time’, ‘weight’, ‘income’, and ‘exchange’ is. We ‘know’ none of them from just one theory even if there is a dominant one (which often there is not). Also, there is the issue of how the theories get formed; there are prior theories and images surrounding concepts that inform the latest batch of theories.
      Also, at the end of the day scientific concepts are words and we ‘know’ many of them much like we ‘know’ other words: by associations including sounds, visual cues, and abstract connotations. We thus do have a set of internal images that tell us what a tree is and do not ‘just’ know what a tree is from fairytales about the roles of trees. Defining what a tree is aint easy, but 99% of trees would still be recognised as a tree even if there were nothing else like soil or an horizon to give it ‘meaning’.

      Still, this harping on small differences in our use of language to describe a desired process of scientific discovery should not obscure the main point of the post and the thread: we consistently fail to measure well the aspects of any concept about which we have causal storylines. We have not found a level of measurement whereby what we measure captures just the concept we were after and nothing else, such that any two ‘things’ that yield the same measurement can be reasonably said to indeed relate to all the other concepts in the same way. So ones standards of proof and rejection should adjust to this reality.

  24. Mike , how about conditions, shall we call them.
    Anaemia can be defined as a deficiency of red blood cells.
    Degrees of anemia exist but the definition is still correct once a threshold is passed.
    Red blood cells can be counted and compared to total blood volume.It is possible to understand that anemia is a man made concept nonetheless it exists while it isn’t an object.
    All the different types of blood cells are known and defined by their characteristics as are their precursor cells.

  25. Mike Pepperday:

    “Scientists know the MEANING of “species” and science merrily proceeds. ”

    As someone with a longstanding interest in Australian botany, I can tell you this is hilariously misguided nonsense. There is actually a great deal of dispute regarding the meaning of the word species and what one thinks its means is no more separable from what one thinks is a good definition than wetness is from water.

    Google species lumper splitterfor examples of the problem and for details on its practical implications.

    • Mel, completely off topic, however I have been intrigued for years by the way that mistletoe species that infest, for example, casurarinas, have leaves that are roughly needle-shaped, whereas mistletoe species that infest eucalypts have leaves that are roughly similar to a eucalypt leaf. I haven’t found any explanations of what sort of selection pressure might explain this phenomenon. Can you advise?

      • David Watson lists 3 theories in Mistletoes of South Australia:

        *Concealment from mammalian herbivores- mistletoes are generally more nutritious and moister than the host
        *Parallel evolution with the host due exposure to the same environmental factors
        *To assist in seed dispersal by making it less easy for seed eaters to detect mistletoe without first landing on a tree.

        Theory 3 sounds most realistic to me.

        Apols to Paul for the OT comment

        • Apols to Paul for OT question!
          Thanks for the reference.

          Sounds like both 1 and three could play a part.

    • Mel—
      Odd. I think you are actually agreeing with me. That the word is disputed is my point. Start talking about a definition—of anything—and there is dispute. But scientists do use the word “species” and it does convey meaning for them. And that meaning will be relational.

  26. Murph the surf

    The anaemia exists independent of any human agency (assuming the famous “tree in the quad” exists). The exact definition is a bureaucratic decision but at some cell count the line is crossed: slight anaemia, severe anaemia… doesn’t matter what you call it, the anaemia is there. By “object” I include concepts. Gravity, heat, heart rate, are (scientific) “objects.”

    I looked into blood groups once, along with fingerprint and cloud classifications. Initially I wanted to know if there were any rules on how to classify (seems not) and this turned into the question: What makes a concept “scientific”? “Blood,” I concluded, is not a useful scientific concept. It’s not as useless as “downward tendency” but we’ve known about blood for a million years and it was only when we could see what it does (circulates) and see its parts and how they fit together that we were doing science. We know the parts exist in an objective, human-free sense because we have discovered how they interrelate.

    (So, Paul, “tree” is not a scientific object—until you find it is made of cellulose whereas grass is made of whatever, and so on. Colloquially we all know what a tree is. I am saying that scientifically it only has meaning through its interrelationships.)

    Blood count (Wikipedia tells me) is about the proportions of red and cells and platelets. Different proportions indicate diseases. Relationships again. That’s science. So “power” is scientifically useless. Relate power to other things and you’d have science. Or identify its components—which can only be recognised by working out their interrelationships—and that would be science.

    The above examples of blood count, geo fault, species, mass, force, time, gravity, indicate that we can only relate things when we can measure them.

    And we can only measure when we have agreed measurement units. This would be why economics shines relative to the other social sciences: it has an agreed unit of measure. Sort-of. Some economists want to relate their unit to a noble metal in a vault.

    That would be why the social sciences except economics thrash around getting nowhere. If we don’t measure, we don’t identify relationships so we can’t identify components. We can’t say whether power (or freedom, cooperation, justice, religion, shame, trust…) exist in a scientific sense.

    Instead of measuring, social scientists try to define social science concepts in words. Imagine defining blood in words. Where would it get you?

    And then, having defined its alleged concepts, social science proceeds to count them. When blood cells are “counted” it is to measure the level of oxygen (or something). So it is more of a measure than a count. Maybe it sometimes scientific to count. It was once pointed out to me that an ESS (evolutionarily stable state) relies on the counts of the component types. Bit theoretical. I don’t know any really persuasive example.

Leave a Reply


