Roadblocks: Cultural and Structural Impediments to Forensic Science Reform

I. Research-Based Sciences vs. Outcome-Based (Forensic) Sciences

A. General Differences in Operating Environment

Forensic scientists do not have the luxury of unlimited time and open-ended inquiries.^[1] Instead, they are often faced with temporal and resource constraints that arise from the legal burdens and expectations imposed on forensic evidence.^[2] These constraints tend to have a greater presence in forensic sciences than other scientific fields because the criminal justice system has increasingly utilized forensic evidence to support investigations and prosecutions.^[3] Moreover, unlike many research fields, forensic science was born out of a public safety need, and thus was never entrenched in a traditional scientific model approach that flows from observation to hypothesis to experimentation to data analysis, and, only then, to conclusions. As the role of forensic evidence collection and analysis has increased, so have the demands placed on forensic scientists.^[4] Consequently, scientists at forensic labs often find themselves pressured to produce rapid results at a low cost.^[5] This systemic pressure, in turn, creates a culture based on results and getting the “bad guy” instead of testing the evidence against scientific principles.

Indeed, forensic science focuses on specific and particular truths.^[6] In fact, “[m]any of the [forensic] disciplines evolved solely for the purpose of solving crimes . . . .”^[7] This is a departure from the generalizable understandings and rules produced by research-based sciences. “Many forensic techniques, such as hair and fiber analysis . . . and fingerprint analysis, rely upon little more than a matching of patterns where a forensic analyst compares a known sample to a questioned sample and makes the highly subjective determination that the two samples originated from the same source.”^[8]

The specificity of findings produced by outcome-based sciences equate to a low possibility of replication.^[9] This clashes with the popular belief of many philosophers who consider reproducibility to be the “hallmark of ‘science.’”^[10] Research-based science promotes the idea that valid scientific knowledge claims “should hold in places and times other than where and when the claim was initially proposed.”^[11] Although it is possible for forensic scientists to verify their findings through repetition, this process is distinct from replicability.^[12] The reason for this is twofold. First, forensic knowledge claims are highly specific.^[13] For example, the assertion that a firearm found in the suspect’s possession fired the bullet recovered from the body of the victim. The data that then supports this claim is based solely on the evidence that was collected for the criminal event under investigation.^[14] It is highly unlikely that similar weapons are tested to see if the toolmarks on the bullet could have come from another firearm. This quasi-experiment conducted in the crime lab would be acceptable if it could piggyback on larger quantitative experiments that have produced troves of data that inform individual casework, but those larger scientific studies are lacking.

Second, external factors influence the analytic outcome and resulting testimony. The absence of a research-oriented culture in forensic science is dangerous because it can result in errors in the way that forensic evidence is used in prosecutions.^[15] A recent study showed that forensic errors are most often found in the interpretation or analysis of the evidence and the resulting testimony as opposed to the “actual scientific testing.”^[16] Prior knowledge is one factor that can handicap the interpretation of evidence.^[17] Forensic scientists receive some information about the evidence or the case which is then juxtaposed with the examiner’s own knowledge, beliefs, and attitudes.^[18] Add in the overarching concern of needing to solve a crime, and these factors influence how the examiner filters and interprets the evidence.^[19] If not set aside, these contextual biases have the potential to create dangerous miscarriages of justice.^[20]

B. Measures of Success

Contemporary science often operates in a “prestige economy” where innovation, professional recognition, and the like are the goal.^[21] In this economy, monetary incentives take a backseat to social capital, which is earned through publication and other academic accolades such as tenure, grants, and research leaves.^[22] The value of this “currency” is reinforced by its implicit ability to guard the production of scientific knowledge against false results, plagiarism, and other forms of scientific fraud.^[23] In other words, the threat of losing professional recognition and the respect of peers is considered an effective deterrent against scientific misbehavior.^[24]

The prestige economy reward structure does not exist for forensic scientists.^[25] Instead, forensic scientists often find themselves entrenched in bureaucracy and its accompanying hierarchies and reward systems.^[26] Forensic scientists are “on the clock,” to produce results at what the scientific field might view as ludicrous speed, and lack the comfort of multi-year research studies and funding. Thus, forensic science (and crime labs) proceed more like a results-oriented business rather than a scientific exploration. The nature of the field makes it less autonomous free-agent work with a greater focus on productivity.^[27] Consequently, the amount of evidence churned through the lab in the shortest amount of time is significantly more important than the contribution to a general body of scientific knowledge.^[28] That is not to say that this is a wholly incorrect approach given the larger concerns of public safety and criminal justice, but it is important to appreciate that forensic science is a business with consumers (police and prosecutors) who can (and will) interfere with scientific research and development.

C. The Role of Adversarialism

Most contemporary research sciences encourage “organized skepticism” in the form of peer review.^[29] Although this constant critique and doubt can serve as more of an ideal than a reality, it is an important part of the research science culture.^[30] Further, peer review can act as a guard against fraud and other errors.^[31] Scientific peer review can be classified as adversarial because the reviewer is required to have a critical mindset.^[32]

In contrast, the research culture has not found a foothold in forensic science.^[33] Forensic science disciplines lack significant peer-reviewed research of the scientific bases and validity studies that should support the forensic methods.^[34] Forensic scientists often operate in the “shadow of adversarialism,” knowing that they can be subjected to the defense’s attacks at any point.^[35] Attacks by the defense are often more intensive than the standard of peer review, but they are significantly less frequent. The practical results of this system include increased audits, proficiency tests, and lab accreditations.^[36] Further, forensic scientists fear any admission of error lest it be brought up again to impugn them, their colleagues, or their discipline in the future.^[37]

Forensic scientists, thus, often take a defensive stance towards outsiders and are hesitant to allow them into their “house.”^[38] This is evidenced by a reluctance to release data or methodological information.^[39] In addition, some data is never gathered by forensic scientists in an overt effort to prevent dissemination.^[40] A majority of forensic science labs do not record their errors or the number of internal disagreements (or conflicts) on a finding.^[41] Oftentimes, even if these error reports are compiled, they are not made available to the public and can only be reached through discovery requests.^[42] Further, many defendants are ignorant to the types of records kept by these labs.^[43] This can lead to overbroad discovery requests which may be severely limited or stopped entirely by prosecutors.^[44] Because forensic scientists cannot separate legitimate discovery requests from those targeted at impeachment, all outside requests are viewed with suspicion.^[45]

“[F]orensic science is a vital component of the criminal justice system.”^[46] The belief that forensic science helps to solve crimes likely contributes to the field’s skepticism towards outsiders and the belief that obstructionist tactics (or defense shenanigans) are at play.^[47] This combination of attitude, opacity, and obscurity differs greatly from scientific norms of transparency and research sharing.^[48]

Adversarialism has a different meaning among forensic scientists as opposed to research-based scientists.^[49] Although many mainstream scientists dislike peer review, they have an understanding of its utility and necessity. The harsher nature and one-sided experience of forensic scientists, however, can instill a negative outlook on the adversarial process, especially when just the thought of being subjected to cross-examination is a negative experience.^[50] To forensic scientists, adversarialism may be viewed as a devious effort to keep the guilty out of prison rather than an earnest search for truth.^[51] Further, internal disagreements between forensic scientists (particularly related to interpretation) lead to greater uncertainty—not greater knowledge.^[52] Thus, they are robbed of the ideal benefit touted by adversarialism in other sciences.

D. Audiences and Reporting

Mainstream scientists generally present their work product to an audience largely comprised of fellow scientists.^[53] Here, scientists are primarily motivated by impressing their peers and they believe that only other scientists can adequately understand and judge their work.^[54] This leads to modest claims which are driven by a strong desire not to be refuted or disproven because research-based scientists understand that their audience is highly capable of critique and feedback.^[55] The evaluation of the research is more stringent, making the work itself more rigorous. This is not the case in forensic sciences.

Forensic scientists often present their findings to audiences comprised of prosecutors, judges, police, and jurors who have minimal training in the subject area.^[56] Consequently, the audience usually lacks the capacity to fully interpret the findings without assistance from the scientist.^[57] In addition, with the exception of defense attorneys, no one in the legal realm considers contesting the forensic results a part of their job.^[58] This troubling acceptance is furthered by courts who “are [often] extremely welcoming of forensic science” and jurors who have also proven to be largely accepting.^[59] In fact, forensic analysts are often considered the “rock stars” of the criminal justice system because their objective scientific skill can “move the jury from maybe to guilty. . . .”^[60] Further, “everyone can sleep better at night because ‘science’ solidified the conviction.”^[61]

Television, popular culture, and news outlets promote the belief that forensic evidence is well-founded and near-certain proof of an individual’s guilt.^[62] Ultimately, this growing acceptance has permitted forensic evidence to proceed unhindered in criminal trials.^[63] In most cases, forensic scientists do not have to work to “win over” the audience, and they lack a strong push back from their peers.^[64] This is because their peers are usually constrained to applying the same or similar methodologies to the same set of data rather than expanding the universe of the problem (and resulting hypotheses) to contest or affirm findings.^[65] Further, forensic scientists tend to couch answers in vague qualitative phrasing, but their results are still often proclaimed with a level of quantitative certainty that is highly unusual in mainstream sciences.^[66] In fact, “forensic evidence has the essential hallmarks of certainty that juries need and society craves” when it comes to criminal prosecutions and pushing a case beyond a reasonable doubt.^[67]

E. Research Agendas and Feedback

In almost any scientific field, different audiences promote different research agendas.^[68] While a research science agenda might be set by paradigms, corporate capitalism, or intellectual curiosity, the research agenda for forensic scientists is most often pushed by the single-largest stakeholder: the state.^[69] The result is a robust research agenda with a focus on newer and faster methodologies.^[70] Meanwhile, validation studies are neglected by scientists and by courts.^[71] This is evidenced by the many forensic techniques that have been applied to cases without validation studies.^[72] Once established, the ongoing use grandfathers these applications in and courts cement their longevity.^[73]

Feedback on both research-based science and forensic science is less common than one would anticipate.^[74] Most research-based claims are eventually tested against independent data sources and new experiments.^[75] These tests generally begin after a fellow scientist senses that a particular new theory is off base.^[76]

The structure of forensic science makes it uncommon for researchers to receive valid feedback.^[77] Occasionally common-sense feedback is provided on false results.^[78] For example, an unbreakable alibi might be presented at the time the sample was deposited.^[79] More often, no feedback is offered at all and forensic results are simply accepted as facts.^[80]

This side-by-side comparison evidences the lack of science in forensic evidence. In addition, it illustrates the conflicting demands that forensic scientists face.^[81] They are expected to perform rapid, accurate, and cheap tests that oftentimes have life-altering consequences.^[82] Implementing research-based principles into the forensic sciences would greatly assist in finding a balance between these competing demands.

II. The Role of Scientific Rigor in Promoting a Culture of Transparency and Collegiality

A. Introduction

Forensic science is best understood as the provision of information to help answer questions of importance to investigators and to courts of law. A scientist becomes a forensic scientist when their scientific knowledge is used to assist courts in understanding their test results.^[83] Forensic analysis involves both deductive and inductive research techniques.^[84] Deductive reasoning begins with a conceivable explanation that is tested and verified.^[85] On the other hand, inductive investigations combine confirmed activities with observed traces to create case-specific knowledge.^[86]

Scientific expert testimony is required to meet certain standards of reliability in order to be admissible in court.^[87] The Daubert test determines whether scientific expert testimony is admissible in both state and federal courts.^[88] The test requires judges to act as gatekeepers who may only admit evidence that proves to be both “relevant” and “reliable.”^[89] Notwithstanding the evident “roadblocks” imposed by the Daubert test, judges often accept forensic evidence that is backed by questionable scientific methods.^[90] In fact, American courts have welcomed factual and opinion-based forensic reports with open arms.^[91] As a result, many forensic disciplines have been improperly legitimized without subjection to the type of scrutiny that is required of novel scientific or technical evidence.^[92]

The comparison between research-based science and forensic sciences highlights the many flaws in crime lab culture. For instance, forensic laboratory processes are often inconsistent and undependable because of the “vague scientific protocols” that underlie many fields of forensic science.^[93] In addition, setting aside DNA, “no single forensic technique has proven to have the ability to accurately . . . verify a match between evidence samples.”^[94] The “[s]hortcomings in forensic science have harrowing implications” for both the accused and the scientists.^[95] For instance, flawed forensic science has contributed to the conviction of numerous innocent people.^[96] Further, if a scientist is unable to reach the expected result, it might cost them their job because they may be required to withhold exculpatory evidence.^[97]

The apparent strengths of forensic evidence and crime lab culture, in addition to some of their flaws, are well demonstrated by recent advances in DNA evidence.^[98] DNA evidence has often been considered the “gold standard” of forensic evidence.^[99] In fact, the NAS report considers it the one method that “has been rigorously shown to have the capacity to consistently, and with a high degree of certainty, demonstrate a connection between evidence and a specific individual or source.”^[100]

Today, DNA evidence is becoming even more sophisticated. Scientists are now able to fill in blanks while assigning probabilities to potential outcomes. In addition, more labs are engaging in ground truth explanation, and numerous software systems now exist to evaluate DNA. GeNie, BNlearn, R libraries, gRain, HUGIN, and AgenaRisk are some of the evaluating programs that are now available.^[101] Still, other technologies implementing advanced algorithms and the arguably more subjective interpretation of “touch DNA” samples have been called into question.^[102] Determining the reliability of a DNA expert will inevitably become more and more difficult as new methods of evaluating DNA samples continue to emerge with varying levels of procedural subjectivity.

B. Implementing Research-Based Principles in Forensic Sciences

Forensic science should seek a greater implementation of research-based values and practices. Many characteristics and practices of research-based sciences are missing from results-oriented forensic sciences. This section will examine how forensic science can benefit from articulating probabilities, providing baseline questions for witnesses, identifying categories of investigative opinions, and implementing other research-based principles.

1. Changing the Approach to Articulated Probabilities.

First, forensic scientists can reframe the way they approach probability. Typically, an expert witness in forensic sciences will testify simply that there is a specific likelihood or level of certainty that something happened. Unfortunately, this can often lead to oversimplifications of the facts and, in some scenarios, it may mislead or confuse a jury.^[103]

Instead, forensic scientists can be encouraged to elaborate on this probability. They can answer “what is the probability that you got your given results if the prosecution’s proposition is true?” as well as “what is the probability if the defense’s proposition is true?” This type of questioning should help prevent assigning probability as a subjective measure of belief rather than a mathematical calculation. There is no way of validating an expert’s single probability assessment beyond agreement with the rules of probability.

2. Baseline Questions for Each Expert Witness to Answer.

There are several questions that should always be asked when applying forensic sciences in a criminal court. The questions also have an implicit hierarchy. From most important to least important, they are: First, what is the source of the recovered material? Second, what activity happened? Finally, what offense, if any, has been committed? If a scientist cannot provide an answer to the underlying source-level question, it follows that they cannot provide an answer to the latter activity and offense questions.

Presently, there is no hardline rule requiring expert witnesses in forensic sciences to answer all of these questions and, in doing so, lay the appropriate foundation for their assertions. It is thus incumbent upon the questioners, the court, and attorneys to hold experts to this standard. Doing so would bring these witnesses more in line with research-based scientific principles.

3. Identifying Categories of Opinions.

Forensic science opinions can be classified into three categories: investigative opinions, preliminary evaluative opinions, and fully evaluative opinions. Investigative opinions are explanations, or conjectures, for observations. These can be associated with posterior probabilities for the explanations. Preliminary evaluative opinions are expressions of the likelihoods for the findings given the truth of individual propositions. Finally, fully evaluative opinions are expressions of the magnitude of the likelihood ratio.

In litigation, it is paramount that forensic science witnesses identify and explain which type of opinion they are giving in testimony. This provides the finder of fact with the ability to evaluate the evidence on the merits instead of getting sidetracked by the expert’s credentials and convincing opinions.

4. Other Characteristics of Research-Oriented Science That Need to Be More Prevalent in Forensic Sciences.

Posterior probabilities are the revised probabilities of an event occurring after new evidence is taken into consideration.^[104] In other words, a posterior probability determines the probability of a scientist’s original hypothesis as it relates to the given evidence.^[105] The potential bias influencing these probabilities leads to a difficult question: should forensic scientists consider the prior probability of the hypotheses that they are asked to examine?^[106] Ultimately, the answer is dependent on the forensic scientist’s role in the legal system.^[107] If the ultimate legal determination is left to the scientist, they must examine the posterior probabilities surrounding their research.^[108] However, if the truth of the hypotheses will be decided by a court, the forensic scientist should limit herself to assigning the conditional probability of the findings and leave the posterior probability assessment to the finder of fact.^[109]

While DNA analysis is often held as the gold standard, it is not immune from criticism nor lacking in need for improvement. For example, activity level reporting (and studies that examine this) should have a greater presence in DNA analysis. Activity level reporting uses data and subjective opinion to consider the issues that surround DNA transfer, prevalence, and recovery.^[110] Probabilities are then assigned to the activities that putatively took place as part of the defense or prosecution’s version of the event of interest. While activity levels are currently examined in DNA cases, most studies examine only one activity.^[111] Forensic scientists should instead compare the probability of the outcomes related to the alleged activity against at least one alternative.^[112] This should enable them to decide whether the variation impacts their conclusions.^[113]

Bayesian inferences should also play a greater part in forensic science. These inferences assist with the inductive reasoning process by evaluating which hypothesis can be justified considering the evidence brought to light during examinations.^[114] Here, construction is affected by the way that the propositions refer to the actors of the alleged crime. Returning to DNA analysis as an example, Bayesian inferences consider the presence or absence of DNA where the defense proposition may state that: (1) the activity never took place; (2) the activity was carried out by someone else; or (3) the accused carried out some alternate activity that led to the findings.

The likelihood ratio is a formula based on the laws of probability.^[115] It takes into account factors that are important in the evaluation, considering also the relationships between these factors.^[116] In DNA science, the likelihood ratio examines evidentiary support for the suspect identification hypothesis.^[117] It is a standard information measure that can clearly state the support for a hypothesis. The likelihood ratio is omnipresent in research sciences but forensic scientists and crime labs often set it aside in favor of DNA inclusion statistics.^[118] This is because likelihood ratios can be difficult to explain in court.^[119] Forensic sciences should work towards a greater implementation of likelihood ratios because they are more reliable than DNA inclusion methods which use less DNA data and often lead to inconclusive results.^[120]

5. How Research-Oriented Principles May Benefit Litigation.

Research-based principles have the potential to benefit forensic evidence in litigation in many ways. This was clearly evidenced by the 2009 NAS report which concluded that “much forensic evidence—including, for example, bitemarks and firearm and tool mark identifications—is introduced in criminal trials without any meaningful scientific validation, determination of error rates, or reliability testing to explain the limits of the discipline.”^[121]

Those in favor of forensic science reform, however, should accept the fact that courts are often resistant to change.^[122] Because judges are charged as gatekeepers under Daubert, forensic legislation should encourage forensic scientists to explain their research to the judiciary in layman’s terms.^[123] In addition, any limitations to the scientist’s evaluations should be clearly stated in their reports given at trial. Eventually, this should serve to prohibit decisions based solely on an expert’s experience and their opinion instead of the underlying evidence and data. Judges will likely continue to allow faulty evidence to be heard at trial unless the questionable validity of many forensic sciences is brought to light.^[124]

Probability is the measure of belief in the occurrence of an event. A normative perspective of probability makes explicit a coherent thought process about real-world events. It is inherently a measurement of uncertainty because knowledge about the event is almost always incomplete. Therefore, it is the forensic scientist’s responsibility to present their probability, as the probability does not exist per se as an external concept that can be investigated and known independently of an individual thinking mind. Here, an emphasis should be placed on the subjectivist view of probability.

Three primary questions concerning probability should be asked of scientists reporting at trial. First, what is the forensic scientist’s probability for the forensic result if the first proposition is true and given the relevant conditioning information? Next, what is the probability for the same forensic result if the second proposition is true and given the relevant conditioning information? And finally, is the forensic result more probable if the first proposition is true or if the second proposition is true?

C. Remaining Hurdles to Implementation

Implementing research-based processes will require a “buy-in” from law enforcement officers, the legal system, scientists, universities, and other participants in the forensic evidence process.^[125] For scientists, preparation for these advances will require adequate training and a thorough understanding of more fundamental principles. Further, experts should be able to clearly explain their reasoning and the limitations of their opinion in a courtroom setting. This will require training on the logical principles of evidence evaluation and the role of the expert in the legal process. For the legal system, it will require courts to accept changes in the way that forensic evidence is produced and is presented in court.

Budgetary constraints are an additional hurdle.^[126] The growing amount of forensic research requests have made it increasingly more expensive to complete the necessary tasks.^[127] Aside from DNA technology, there is little funding for research in forensic science.^[128] This ultimately results in a vicious cycle where the low priority given to advancing research methods in forensic science equates to limited rewards, which in turn limits opportunities for scientists who seek to develop a true research culture.^[129]

Some of these hurdles can be overcome by forensic research partnerships between crime labs and universities or nonprofits.^[130] Forensic research partnerships have the ability to simultaneously combine the “underlying theories of forensic science with [improved standards for] application and practice.”^[131] By making forensic laboratories a part of the reform process, they become part of the solution instead of being held out solely as problematic.^[132]

III. Conclusion

Research must become an integral part of the forensic science culture. Forensic evidence should prevent wrongful convictions and currently it is causing them.^[133] Although it will be difficult to implement research-based practices in a culture that has been described as the “wild west,” it is not impossible.^[134] The difficulty of instilling heightened research standards is far outweighed by the decrease in wrongful convictions that we will see as a result.