Consider the following drawing:

This is a drawing of a suspected serial killer, based on the memories of a three eye witnesses. What kind of a guy does he look like? Kind of scary and intense, but could pass for a normal-ish guy?
No, wait – this is an artist’s idea of what a man, a truck driver and father of 2, would look like today, 18 months after he vanished after wandered out of a hospital where he was being treated for a brain injury. So, now what kind of a guy does he look like? Man, what a sad story!
OK, no really – this is a reconstruction based on analysis of the DNA found in some 7.000 year old bones recently found in a Spanish cave. The scientists concluded he had the gene for dark skin as well as the gene for blue eyes – the rest of the face is pretty much fantasy – in fact, given how genes work, ALL of the face is petty much fantasy.
The point of this little exercise, something I thought was pretty well understood: don’t prime the pump. People are at least as likely to opine based on what you’ve told them as on what they see or remember.
This comes up because of this story, about a brain researcher James Fallon, that I first heard about through Ye Olde Statistician’s blog. Seems Dr. Fallon has written a book now. He was interviewed by the Atlantic. In that interview, he discusses the method he used to check if he really is a psychopath: he asked friends and family – am I a psychopath?
After discovering that he had the brain of a psychopath, Fallon delved into his family tree and spoke with experts, colleagues, relatives, and friends to see if his behavior matched up with the imaging in front of him. He not only learned that few people were surprised at the outcome, but that the boundary separating him from dangerous criminals was less determinate than he presumed. Fallon wrote about his research and findings in the book The Psychopath Inside: A Neuroscientist’s Personal Journey Into the Dark Side of the Brain, and we spoke about the idea of nature versus nurture, and what—if anything—can be done for people whose biology might betray their behavior.
This is, how you say? WRONG. That people were not surprised at the outcome means he told them what the “outcome” was. No, no and no. He primed the pump. He lead the witnesses. This is not science, it’s astrology, which works by telling people who they are and inviting them to agree.
So, here’s how a scientist would investigate:
1. In your base data – the brain scans of the known criminal psychopaths – is the correlation, run through a standard double-blind process, 100%? If not, do not pass go, do not collect $100 – you’re done. Because if you run a real double-blind analysis, AND include many scans of people who are not criminal psychopaths AND have some sort of defensible and objective definition of psychopath the does not in any way rely on the brain scans, THEN you could tentatively conclude that the pattern under discussion proves someone is a psychopath ONLY if 100% of the psychopaths display the patterns AND 100% of the non-psychopaths do not. If there are any psychopaths who do not exhibit the brain scan pattern – any at all – then – follow carefully – the brain scan pattern is insufficient to show someone is a psychopath. Similarly, if there are any non-psychopaths that exhibit the pattern. then the pattern is not an indication of psychopathy.
What one is after logically is ‘necessary and sufficient’ evidence: if someone can exhibit the brain pattern without being a psychopath, the brain scan pattern is not sufficient to prove one is a psychopath; If someone can be a psychopath without exhibiting the brain pattern, then the brain pattern is not necessary to prove psychopathy.
Was this done? None of the stuff I’ve seen suggests it was. If not, the science is over before it began.
The above should be clear. No one to whom it is not clear has any business being a scientist. I suggest they get into retain sales.
2. Clear evidence that step 1 was never done lies in the mere fact that Fallon has written a book on his own experiences. If the science of step 1 was done, and the answer was: yes, the psychopath brain scan pattern is necessary and sufficient proof that one is a psychopath, THEN there would be no place for Dr. Fallon’s surprise. The surprise comes from the discrepancy: clearly, Fallon is not a psychopath, at least not in any meaningful sense. So, what we do as scientists at this point is: throw out the theory. It has been destroyed by an ugly fact.
But let’s play along, and imagine what the investigation might look like if it were to be conducted scientifically:
I have identified an outlier, a data point that does not fit the theory: an individual who, despite showing necessary and sufficient brain scan proof that he is a psychopath, does not exhibit psychopathic behaviors.* We have two propositions to test: 1 – our outlier has been misclassified, and is indeed a psychopath**; or 2 – our theory is wrong.
To be science at this point, we need an objective standard of what a psychopath is, as well as a double-blind mechanism so that a: the people gathering the inputs to make the classification are not looking for anything in particular; and b: we have a large enough set of people subjected to the classification to show that it works in general.
What this might look like, if we were to use something like the ‘ask friends and colleagues’ approach Fallon used:
– 1,000 people are recruited for this test;
– Everyone is classified using the objective non-scan based criteria as either a psychopath/not a psychopath;
– Everyone is classified using the brain scan pattern method.
– Those for whom there is a mismatch – classified as a psychopath under one method and as a non-psychopath under the other – are the persons of interest. Of course, there shouldn’t be any of these if the theory was properly tested as described in the base science above, but we are here testing the curious data point of Dr. Fallon, and can’t rule out the existence of other Dr. Fallons out in the wild.
– Each identifies friends and colleagues who will be interviewed;
– Fallon is just one person in the test as far as any of the testers know;
– The friends and colleagues are asked a set of general, non-leading questions about the kind of person the volunteer is, as well as the same questions about other people they know, to be used as a control of sorts. These questions should be designed so the answers will permit the classification of the volunteers into psychopath/non-psychopath groups independently of the original classifications performed prior to the data collection;
– Another set of researchers, completely separate from the ones who did the interviews above, does the classification of the volunteers from the interview results without knowing the identities of the people being classified;
– The results are reviewed: do the initial classifications done in second step match the classification done immediately above? If yes, or if yes for everybody except Dr. Fallon and the other (very few) outliers, then maybe we have something here. If mis-matches are more common, it calls our assessment protocols into question – it does not let us conclude anything except that we need to refine or change our experimental method.
There’s only one outcome to the above that would give us a clear answer: all the initial assessments match all the final assessments, including Dr. Fallon’s. That would mean that the theory that brain scan pattern reveals psychopathy is FALSE, as Fallon has been revealed not to be a psychopath. Any other results could have a variety of causes, and would require reassessment of the experimental protocols. All this assumes, of course, that the 100% correlation between the brain pattern and psychopathy has been established before hand – if it hasn’t, this study would be pure nonsense.
And so on. I may have missed some steps – a sufficiently large grant would enable me to fix that (hint, hint!) Does this sound hard? It should – real science is hard. Does it sound like it is going to be difficult to get a definitive answer? Except for *disproof* of the theory, which such a test could provide, yes, it’s going to be difficult to get a clean answer. Did I mention science is hard?
But what you can’t do – it’s worse than even just anecdotal evidence – is go aroind asking your friends and family: “Am a a psychopath? My brain scans say I am.” That is worse than bad science. It’s out and out anti-science.
BTW: No fair playing the ‘brain science is too complex to be understood with such a simple experiment’ card – if so, it’s also too complex to make sensational claims for, like how you can’t tell a salmon is dead – oops, I mean: how you can tell who is a psychopath based on a brain scan.
* Note that this is where having a good, solid, objective definition of what makes a person a psychopath comes in. Ideally, Dr. Fallon, BEFORE submitting his brain scan as a control, would have undergone exactly the same battery of testing used to determine who is and is not a psychopath in the original study done in step one – so we would already know the answer! That would good science, or at least non-stupid science.
** a result which throws the entire real science in theoretical step 1 into question – how did he slip through?