Are innocent people being jailed due to blind faith in unreliable forensic techniques?
This is the question that has recently been sparked by Justice Chris Maxwell, President of the Victorian Court of Appeal, who states that forensic techniques such as gunshot analysis, footprint analysis, hair and bite mark comparison are unreliable in accurately identifying criminals.
Justice Maxwell refers to a 2016 report (“the Report”) to the US President titled Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods. The Report, written by the US President’s Science and Technology advisors (PCAST), concludes that DNA analysis is the only forensic technique that is absolutely reliable.
These finding have helped to rectify many wrongful convictions around the world and have led to major changes to the US and British legal systems, but not so in Australia.
This begs the question, as posited by Justice Maxwell: “Why are we not having appeals based on questions of admissibility of forensic evidence?”
Closing the gaps to make forensic evidence more reliable
The President’s Council of Advisors on Science and Technology (PCAST) is an advisory group of the United States’ leading scientists and engineers, appointed by the President to provide advice and make policy recommendations on science and technology-related matters.
The Report was prepared in response to a question posed to PCAST from former US President Barack Obama following a 2009 National Research Council report on the state of the forensic sciences, which identified systemic problems such as inadequate training, resources and capacities of laboratories, and insufficient research to establish the scientific basis and validity of many routinely used forensic methods.
The question concerned what scientific steps could be taken to help ensure the validity of forensic evidence used in the US legal system.
PCAST concluded that there were two important gaps:
(1) the need for clarity about the scientific standards for the validity and reliability of forensic methods
(2) the need to evaluate specific forensic methods to determine whether they have been scientifically established to be valid and reliable.
The PCAST study and subsequent Report aims to help close these gaps for the case of forensic “feature-comparison” methods such as the analysis of DNA, hair, fingerprints and firearms. These are methods that attempt to determine “whether an evidentiary sample (e.g. from a crime scene) is or is not associated with a potential ‘source’ sample (e.g. from a suspect), based on the presence of similar patterns, impressions, or other features in the sample and the source.”
We explore the key findings and recommendations contained in the Report.
The impact of “scientifically invalid” forensic methods
The use of unreliable forensic methods has been shown to have contributed to the criminal conviction of hundreds of innocent people in the United States.
Various studies have been conducted raising serious concerns about the reliability of certain forensic methods, such as 2009-10 studies on bitemark evidence which found that current procedures for comparing bitemarks are unable to reliably exclude or include a suspect as a potential biter.
These problems are exacerbated by expert witnesses claiming that their conclusions are “100% certain” – statements which are not scientifically defensible. In an unprecedented 2012 review by the Department of Justice (DOJ) and FBI of testimony in more than 3,000 criminal cases involving microscopic hair analysis, it was shown that FBI examiners had provided scientifically invalid testimony in more than 95% of cases where that testimony was used to inculpate a defendant at trial.
Satisfying Scientific Validity
In order to satisfy scientific validity, there needs to exist:
- Foundational validity – based on empirical studies, the method is repeatable, reproducible, and accurate. In other words, foundational validity means that a method can, in principle, be reliable.
- Validity as applied – the method has been reliably applied in practice, i.e. an expert “has reliably applied the principles and methods to the facts of the case.”
A further complicating factor arises with “subjective methods” which rely heavily on human judgment, as they are especially vulnerable to human error, inconsistency across examiners, and cognitive bias. In establishing foundational validity, it is therefore essential to perform studies that measure the overall error rate across many examiners.
Evaluation of Scientific Validity for common forensic methods
In its Report, PCAST examines the scientific validity of seven common feature-comparison forensic methods:
- DNA analysis of single-source and simple-mixture samples
- DNA analysis of complex-mixture samples
- Bitemark analysis
- Fingerprint analysis
- Firearms identification
- Footwear analysis
- Hair analysis
DNA Analysis of Single-Source and Simple-Mixture Samples
This refers to samples from a single individual or from a simple mixture of two individuals (e.g. in a rape case), which the Report describes as “an objective method in which the laboratory protocols are precisely defined and the interpretation involves little or no human judgment.”
Whilst this is a foundationally valid method that is “repeatable, reproducible, and accurate”, like all forensic analyses, errors can and do occur. Such errors may stem from sample mix-ups, contamination, incorrect interpretation, and errors in reporting.
DNA Analysis of Complex-Mixture Samples
Some investigations involve highly complex DNA analysis, such as mixed blood stains from multiple unknown individuals. This type of analysis involves the subjective interpretation of the DNA profile, introducing significant risk of both analytical error and confirmation bias.
PCAST therefore finds that “subjective analysis of complex DNA mixtures has not been established to be foundationally valid and is not a reliable methodology.”
Bitemark Analysis
Bitemark analysis typically involves examining marks left on a victim or an object at the crime scene and comparing those marks with dental impressions taken from a suspect. The report describes this as a highly subjective method with ill-defined standards for feature identification and matching. Scientific evidence indicates that “examiners not only cannot identify the source of bitemark with reasonable accuracy, they cannot even consistently agree on whether an injury is a human bitemark.”
PCAST therefore finds that bitemark analysis falls very short of meeting the scientific standards for foundational validity.
Fingerprint Analysis
PCAST finds that fingerprint analysis is a foundationally valid subjective methodology, but that there are a number of issues with respect to validity as applied, including:
- Confirmation bias – Work by FBI scientists has shown that examiners often alter the features that they initially mark in a latent print based on comparison with an apparently matching exemplar
- Contextual bias – e.g. examiners’ judgments being influenced by irrelevant information about the facts of a case
- Proficiency testing – insufficient testing to assess an examiner’s capability and performance in making accurate judgments
Firearms Analysis
In firearms analysis, examiners attempt to determine whether or not ammunition is associated with a specific firearm based on marks produced by guns on the ammunition.
PCAST finds this to be a foundationally invalid method, due to insufficient studies to understand the reliability and reproducibility of the methods.
Footwear Analysis
Footwear analysis typically involves comparing a shoe to an impression found at a crime scene, to assess whether it is likely to be the source of the impression.
PCAST finds that there are no appropriate studies to support the foundational validity of this method.
Hair Analysis
Forensic hair analysis is a process by which examiners compare microscopic features of hair to determine whether a particular person may be the source of a questioned hair.
PCAST finds that there are no appropriate studies to support the foundational validity of this method.
Recommendations
The Report makes various recommendations concerning the application, use and development of feature-comparison forensic methods. Whilst there is an inherent US focus with recommendations made for specific organisations such as the FBI, we present a summary of these recommendations in a broader context:
Scientific recommendations
- It is important that scientific evaluations be conducted on an ongoing basis, to assess the foundational validity of current and newly developed forensic feature-comparison technologies. To ensure the scientific judgments are unbiased and independent, such evaluations should be conducted by an agency which has no stake in the outcome.
- Three currently subjective forensic methods – fingerprint analysis, firearms analysis, and DNA analysis of complex mixtures, should be transformed into objective methods through coordination amongst the relevant agencies.
- There should be increased organisation in developing and promoting standards and guidelines to improve best practices in the forensic science community.
- A research and development strategy should be developed to conduct required studies, improve current forensic methods and develop innovative new methods.
Judicial recommendations
- It should be ensured that expert testimony in court about forensic feature-comparison methods meets the standards for scientific validity.
- When deciding the admissibility of expert testimony, judges should take into account the appropriate criteria for assessing scientific validity, including foundational validity and validity as applied.
- Judges should ensure that when experts are testifying about a foundationally valid method, the stated accuracy of the method is in line with what empirical evidence supports.
- A best practices manual should be developed to provide guidance to judges concerning the admissibility of expert testimony based on forensic methods.
