D. Krane, S. Ford, J. Gilder, K. Inman, A. Jamieson, R. Koppl, I. Kornfield, D. Risinger, N. Rudin, M. Taylor, W.C. Thompson. "Sequential unmasking: A means of minimizing observer effects in forensic DNA interpretation." Journal of Forensic Sciences. 2008;53(4):1006-7.

SEQUENTIAL UNMASKING: A MEANS OF MINIMIZING OBSERVER EFFECTS IN FORENSIC DNA INTERPRETATION

Sir:
Observer effects are rooted in the universal human tendency to interpret data in a manner consistent with one's expectations (1). This tendency is particularly likely to distort the results of a scientific test when the underlying data are ambiguous and the scientist is exposed to domain-irrelevant information that engages emotions or desires (2). Despite impressions to the contrary, forensic DNA analysts often must resolve ambiguities, particularly when interpreting difficult evidence samples such as those that contain mixtures of DNA from two or more individuals, degraded or inhibited DNA, or limited quantities of DNA template. The full potential of forensic DNA testing can only be realized if observer effects are minimized. We met n December 1 and 2, 2007 in Washington, D.C. to discuss the implications of observer effects in forensic DNA testing and ways to minimize them.

The interpretation of an evidentiary DNA profile should not be influenced by information about a suspect's DNA profile (3-6). Each item of evidence must be interpreted independent of other items of evidence or reference samples. Yet forensic analysts are commonly aware of submitted reference profiles when interpreting DNA test results, creating the opportunity for a confirmatory bias, despite the best intentions of the analyst. Furthermore, analysts are sometimes exposed to information about the suspects, such as their history or motives, eyewitness identifications, presence or absence of a confession, and the like. Such information should have no bearing on how the results of a DNA test are interpreted, yet may compound an unintentional confirmatory bias. This bias can result in false inclusions under not uncommon conditions of ambiguity encountered in actual casework. It can also render currently used frequency statistics or likelihood ratios misleading.

These problems can be minimized by preventing analysts from knowing the profile of submitted references (i.e., known samples) when interpreting testing results from evidentiary (i.e., unknown or questioned) samples. The necessary filtering or masking of submitted reference profiles can be accomplished in several ways, perhaps most easily by sequencing the laboratory workflow such that evidentiary samples are interpreted, and the interpretation is fully documented, before reference samples are compared. A simple protocol would dictate a separation of tasks between a qualified individual familiar with case information (a case manager) and an analyst from whom domain-irrelevant information is masked.

Such a protocol would have the following steps. First, the analyst interprets the results of testing on the evidentiary samples. In this initial interpretation, the analyst would:

1. determine the alleles associated with each sample;
2. assess the number of contributors; and
3. assess the likelihood that the test procedure failed to detect some of the alleles of contributors (e.g., allelic dropout).

After the results of the initial interpretation are documented, information about reference samples should be unmasked in a sequential manner. In cases where an individual is expected to be a contributor to a sample (e.g., the victim's DNA in a sexual assault sample), the analyst should next compare this reference sample to the evidence profile and evaluate the foreign donor profile in light of this unmasked information (and document again the alleles that would cause any other person to be included or excluded as a possible contributor). At this stage (before knowing the profiles of any suspects) the laboratory should also compute the frequency in appropriate populations of individuals who would be included as possible additional contributors. Only when these computations are recorded should the laboratory undertake the final step of determining whether the other submitted reference samples have the documented genotypes of potential contributors. Cold hits illustrate that it is feasible to interpret evidence samples without knowledge of the reference profile(s). In cases in which a suspect has been identified, a masked interpretation of the evidentiary profiles should have the same utility.

We are not suggesting that forensic scientists be blind to information that might afford them the greatest opportunity to generate reliable information from evidentiary samples. For instance, the nature of the substrate associated with a sample may dictate that certain extraction procedures be used. The case manager should decide what to test and how to test it and could supervise testing through to the development of a DNA profile. However, a sequential unmasking procedure must be used to shield the analyst from task-irrelevant information when interpreting results in order to minimize observer effects. Such procedures can and should be adopted immediately by all forensic DNA testing laboratories.

Sequential unmasking is the most efficacious means of reducing the compromising influence of observer effects on the utility of forensic DNA evidence. We hope this letter will also initiate a dialogue about other safeguards that might be employed to combat observer effects in DNA testing and other areas of forensic science. In the long run, organizational changes may be required to ensure the integrity of the masking process and a reliable separation between DNA analysts and domain-irrelevant information. A properly designed information firewall, for example, could reduce the danger that case managers will inadvertently leak information to analysts, thereby undermining the masking procedure.

With advances in technology, DNA testing has increasingly been used to analyze marginal samples that are likely to produce ambiguous results, such as older samples, samples exposed to environmental insult, and limited samples resulting from incidental contact. Consequently, the need for measures to minimize the consequences of observer effects in forensic DNA testing is growing.

Dan E. Krane, Ph.D., Professor of Biological Sciences, Wright State University, Dayton, OH;
Simon Ford, Ph.D., President, Lexigen Science and Law Consultants, Inc., San Francisco, CA;
Jason R. Gilder, Ph.D., Senior Systems Engineer, Forensic Bioinformatics, Inc., Fairborn, OH;
Keith Inman, M. Crim., Senior Forensic Scientist, Forensic Analytical Sciences, Inc., Hayward, CA;
Allan Jamieson, Ph.D., Director, The Forensic Institute, Glasgow, UK;
Roger Koppl, Ph.D., Director, Institute for Forensic Science Administration, Fairleigh Dickinson University, Madison, NJ;
Irving L. Kornfield, Ph.D., Professor of Biology and Molecular Forensics, University of Maine, Orono, ME;
D. Michael Risinger, J.D., Professor of Law, Seton Hall University School of Law, South Orange, NJ;
Norah Rudin, Ph.D., Forensic DNA Consultant, Mountain View, CA;
Marc Scott Taylor, Laboratory Director, Technical Associates, Inc., Ventura, CA;
William C. Thompson, J.D., Ph.D., Professor and Chair, Department of Criminology, Law and Society, University of California, Irvine, CA;

References
1. R. Rosenthal, Experimenter Effects in Behavioral Research (NY: Appleton-Century-Crofts 1966).
2. D. M. Risinger, M. J. Saks, W. C. Thompson, R. Rosenthal, Calif. L. Rev. (January, 2002).
3. E. S. Lander, Nature 339, 501-505 (1989).
4. National Research Council (U.S.), DNA Technology in Forensic Science (National Academy Press, Washington, D.C. 1992).
5. National Research Council (U.S.), The Evaluation of Forensic DNA Evidence (National Academy Press, Washington, D.C. 1996).
6. P.C. Gianelli, Confirmation Bias Crim. Just. 60 (Fall, 2007).

Last modified: 03/12/10