George Rebane

Reason and logic have always had a low survival rate in courtrooms, especially where poorly understood evidence of a technical nature is presented by self-serving lawyers to innumerate juries.  In recent memory, the OJ Simpson murder trial may serve as a nationally prominent Exhibit A for this proposition.

Today we have the much examined and touted (for various agendas) Zimmerman/Martin shooting case that will soon be going to trial.   Much effort and focus is put on a piece of recorded audio that appears to have become the evidence by which Zimmerman’s claim of justifiable homicide will stand or fall.  This audio clip is now undergoing forensic analysis by state agencies and private parties having court-acknowledged expertise in matters pertaining to analyzing speech.  Some of these experts have apparently weighed in with some “preliminary findings” which now serve as grist for the media mill and a large gaggle of very opinionated bloggers, commenters, and self-appointed jurists in the blogosphere.

RR has chosen not to join this cohort opining on Zimmerman’s guilt or innocence, instead we wish to use the case to examine the larger questions of ‘stand your ground’ victims’ rights, and in this  piece how to properly look at reported evidence in such cases.  In the following we take a look at a report on the blog of “legal scholar” Professor Jonathan Turley by guest poster Mark Esposito.

I have no idea of Mr Esposito’s qualifications, but simply post his opening paragraph that also happens to be his summa.

The  chair emeritus for the American Board of Recorded Evidence, Tom Owen, and Ed Primeau, a Michigan-based audio engineer and forensics expert, have independently concluded that the furtive pleas for help clearly heard on the 911 tapes are not George Zimmerman’s. Both acknowledged experts used voice enhancing software, but different techniques, to rate the probability of the voice being Zimmerman’s at no more than 48%.  A 90% match is considered scientifically reliable.

So there you have it, the case is seemingly closed.  But not quite.

Taking a look at what output such a forensic examination is able to yield immediately tells us that either Mr Esposito doesn’t understand what was told him by Messrs Owen and Primeau, or the latter two are blowing smoke.  Being careful to note that the working hypethesis to disprove Mr Z’s voice is notZ, the only output that is available from such an examination consists of two probability measures – 1) P(TR|notZ), the probability of the test result (TR) given that the voice on the audio was not that of Mr Z (notZ); and 2) P(TR|Z), the probability of TR given that the voice on the audio was that of Mr Z.  That’s all.

Here it is important to distinguish between the desired P(notZ|TR), the probability that it was NOT Mr Z’s voice given the TR, and P(TR|notZ) as defined above.  They are totally different measures.  Nevertheless courts, journalists, and non-professionals habitually confuse the two.  In this case we want to use the test results to find support for notZ, the hypothesis that the audio evidence did not contain Mr Z’s voice.  To compute P(notZ|TR), the desired probability, we appeal to the Bayes Theorem (readers rusty on Bayes are encouraged to review ‘Making Medical Decisions’) which tells us

From Esposito’s report, it is clear that the best attribution we can give the cited 0.48 number is that it is P(TR|notZ), a measure of test performance also called the test’s sensitivity.  (A similar analysis could be done assuming that P(TR|Z) = 0.48 giving very similar results.) To claim that 0.48 is the desired P(notZ|TR) is simply ludicrous, and an admission of test weakness as we’ll see in the following.

But first, please note that no data was given on P(TR|Z), the probability that the TR could have been obtained given that it was Mr Z’s voice – i.e. the test gives a false positive.  That is not only an important metric, but also an absolutely necessary measure, also called the test’s Type1 error rate.  The other absolutely necessary piece of information required here is P(notZ), the prior or pre-test probability that the audio was not the voice of Mr Z.

Since we discount Mr Z’s claim that the audio records his cries for help, and are ignorant of the identity of the voice – it could be Mr Z or it could not be Mr Z – the probabilistic expression of such ignorance is the 50-50 condition between two equally probable contingencies.  This requires us to use P(notZ) = 0.5 (also making P(Z) = 0.5) in the Bayes formula above as the measure of our state of prior knowledge.

Given this, we see that Esposito’s quoting P(notZ|TR) = 0.48 is ludicrous, because professionals such as Owen and Primeau would never publish such a result of their analysis.  What such a result says is that their testing is totally ineffective to resolve the question of the voice on the audio, since 0.48 is essentially equal to 0.5, and therefore such an admission means that they resolved nothing.

Now I’m not saying that the experts or the reporter could not have issued such a lame result, but only that such a conclusion would and should not support their standing in the American Board of Recorded Evidence or of the standing of the ABRE in the scientific community.

So knowing this, what we should ask for from Owen and Primeau is the value of P(TR|Z) described above so that we can compute the indicated likelihood ratio of the test, and go on to compute the actual value of the much sought after P(notZ|TR).  To demonstrate the value of such proper information about the test, let’s assume that the false positive rate of the test is a low P(TR|Z) = 0.1 – only one in ten times would the test falsely identify the audio as belonging to Mr Z.

This lets us calculate the likelihood ratio L(TR|notZ) = P(TR|notZ)/P(TR|Z) = 0.48/0.1 = 4.8.  Substituting these values into the Bayes formula would then give us the correct update of the probability that the audio was not Mr Z as P(notZ|TR) = 0.83.  According to Mr Esposito, a “90% match is considered scientifically reliable”.  I presume that this match is really the probability threshold of supportive tests to be considered as reliable evidence by the court.  This itself is astounding since it says that a ten percent error rate in the conclusion is “beyond a reasonable doubt” of the evidence being valid.

But as far as attributing scientific reliability to it is again ludicrous.  No scientific theory that has such a low reliability would be accepted by peer scientists as progress in pushing back the frontiers of knowledge.  But as I stated at the beginning, courts live in a mysterious and often pernicious world of their own, and this kind of analysis can be confusing.

So there you have it, a detailed look at how evidence should be considered and interpreted.  To put a bow on it, we finally compute the test’s P(TR|Z) required to bring the Owen/Primeau process up to the court’s ‘scientific reliability’.  That yields a required value of 0.05 or only about one out twenty chance of obtaining the test result that would erroneously report that the audio was of Mr Z.  Please recall that the probability of the voice being of someone other than Mr Z given the test results is not the same, nor does it equal the probability of obtaining the test results given that the voice did not belong to Mr Z.

But the bottom line is that Esposito’s report per se provides no information for the technically versed to support any reliable conclusion about the identity of the voice on the recorded audio.  Nevertheless, it sure gets the lay public hyperventilated.

[28apr12 update] To examine the pre-trial extra-judicial influences, PJ Media has done a little research on the broader context in which the Z/M case is being viewed primarily in the progressive community with a history that ties it to the current administration.  PJ Media reports about Afrolantica Legacies, “a disturbing book by one of Barack Obama’s intellectual mentors. In Professor Derrick Bell’s “Afrolantica Legacies,” the founder of Critical Race Theory weaves together a mystical, science fiction fantasy with radical utopian politics. We’ve put these blog posts together into one guide that reveals the shocking and strange ideas of one of the progressive movement’s most influential legal thinkers — a man who imagined himself as a prophet spreading secret teachings delivered to him by an alien goddess.”

In this narrative Derrick Bell appears clearly to support two alternative futures for America – the Great Divide (q.v.) or a “post-meritocracy” communist state.