 |
 |
 |
The Psychometrics and Science of the Standardized Field Sobriety Tests
Published in The Champion (publication of the National Criminal Defense Lawyers Association), May and June 2003 issues and Voice for the Defense. (Publication of the Texas Criminal Defense Lawyers Association), April 2003 issue. The National Highway Safety Transportation Administration (NHTSA) standardized field sobriety tests (SFSTs) came under intense scrutiny by the defense community when they went into widespread use in the 1980's. At that time, the scientific literature to support their use was limited to two NHTSA-sponsored laboratory studies1 and two very modest field studies.2 Both the NHTSA researchers and critics pointed out that the tests had not proven themselves in the field and that studies done under roadside conditions were badly needed. Several groups of critics trenchantly derided the SFSTs and their supporting empirical base and detailed other significant problems.3 In the past seven years, three large-scale field studies have been conducted that potentially address some of the problems noted earlier. Indeed, Marcelline Burns, a primary researcher in the development of the SFSTs, has stated the initial laboratory studies have limited relevance to understanding the use and accuracy of the SFSTs twenty-five years later in field settings.4 Have the subsequent Colorado, Florida, and San Diego SFST field studies rectified the earlier problems? What about research by other researchers or agencies? This paper will review the NHTSA SFST field studies and related works, appraise their impact on the research base for the SFSTs, and review the SFSTs' standing as psychological tests in light of current standards.
The NHTSA SFST Field Studies Presumably because of this initial negative experience, subsequent field testing locations were chosen largely based on the cooperation and support of the administration and officers that would carry out the testing ("…only agencies that could assume an extremely high level of cooperation and commitment would be recommended for participation."6 ). The officers that would perform SFSTs in the new generation of studies were not reluctant draftees, but volunteers7, SFST instructors8, or exhibited "genuine interest in the study and eagerness to be selected."9 The three major NHTSA field studies consist of investigations carried out in Colorado, Florida and San Diego in 1995, 1997 and 1998, respectively.10 The designs of the studies were highly similar, so they will be discussed together. In each, actual traffic stops using the SFSTs were investigated. Police officers were recruited to participate in the study from agencies that supported the research efforts. Officers had previous training and experience in the SFSTs (in the Florida study, all 16 were SFST instructors) and received "refresher" training before beginning data collection. In the Colorado and Florida studies, observers from the study (either researchers or participating police officers) monitored about half of the stops to ensure they observed the study protocols (no use of portable breath tests [PBTs] until after the SFSTs were given and scored) and that the SFSTs were administered correctly. In the Colorado and Florida studies, researchers obtained PBTs on the majority of drivers who were tested but released. This allowed an estimate of false negatives-failures to make an arrest when warranted. The different studies investigated the SFSTs performance at BAC levels of .05% and .08%. All three studies reported that correct arrest decisions based on the SFSTs exceeded 90%, with two of the three reporting higher levels of false negatives (erroneous releases).
Table 1
In all three studies, the proportion of drivers arrested to those tested was quite high-well over 50%. Mean BAC level of those arrested were .138% (San Diego), .150% (Florida), and .152% (Colorado). In the Colorado study, HGN was scored differently than in all other studies, as scores for left and right eyes were not distinguished, and the scores ranged only from 0-3. There is no indication of what instructions were given for the WAT and the OLS in the Colorado and San Diego studies, while the instructions used in the Florida study differ substantially from the 2000 NHTSA student manual. The Colorado study reported that only 13 errors of administration and 6 errors in instructions were observed in 305 SFST administrations (only 41% were observed). No errors were observed in the 313 SFST batteries given in the Florida study, although only two-thirds of the administrations were monitored. The NHTSA Student Manual,11 the official SFST training guide for police officers, provides cutoff scores for each test to optimally classify a person as above or below .10%. It appears that the NHTSA-suggested decision rules for the SFSTs were not used in the Colorado and Florida studies-officers had access to test scores but used their own best judgment as the final criterion for arrest. Officers' failures to follow the recommended SFST-decision rules were cited as a significant problem in the San Diego study. In the Colorado study, incorrect arrest decisions were attributed to officers focusing on poor Walk and Turn (WAT) and One Legged Stand (OLS) performance when the suspects' HGN performance was normal.
Other Studies The only NHTSA-sponsored sobriety test studies that have been published in peer reviewed journals detail the development of a standardized boating sobriety test14 and an investigation of various sobriety tests at detecting BAC at .04%15 . The marine environment is unique because the motion of a watercraft makes the WAT and OLS unsuitable for on-the-spot testing. Like the 1981 SFST study, both laboratory and field observations were made. HGN and three other tests were identified as most promising based on their correlation with BAC. In the field portion of the boating study, the Maryland Department of Natural Resources Police administered the four SBST candidate measures. Officers all had been certified on the SFSTs, were described as "highly experienced" regarding DUI/BUI, and were given an additional day and a half of training before beginning the study. Officers were instructed not to obtain PBT readings until after recording the SBST results, but no observers monitored this or administration and scoring procedures. HGN was found the best individual test, correlating .77 with BAC in the field stops. Using HGN scores alone resulted in 100% classification of BAC >.10% and 90% correct classification below .10%. Two tests used in the field battery, "saying the alphabet" and "hand-pat," showed respectable correlations with BAC but did not improve upon decisions based on HGN alone. The authors nonetheless recommended the full battery because the latter tests provide some measure of performance impairment (vs. BAC level), whereas HGN does not. 16 A very recent investigation17 found that only HGN was effective at distinguishing persons above or below a BAC of .04%, a standard sometimes applied to drivers of commercial vehicles and, in some states, to drivers younger than 21. Both laboratory and simulated field conditions were investigated, and several variations of HGN and the OLS were tried. The variations did not matter much, but the optimum cut-score for HGN was two clues rather than four. Even so, the observed accuracy level obtained was lower than for higher BAC levels: 79% of those above .04% were correctly identified, while 38% of those below .04% were wrongly classified.
Critique of the SFST Field Studies Ideally, subjects in an experiment are randomly assigned to a treatment or experimental group. In this way, differences between the groups are minimized. The original NHTSA laboratory studies assigned subjects to a target BAC group based on their drinking history. In the field studies, there were no experimentally created groups-just drivers stopped for one reason or another. Therefore, the NHTSA field studies are quasi-experiments, not experiments. 19 All the officers employed the SFSTs and no control group was used. A control group is considered a near-essential feature of a rigorous study because it duplicates all the relevant factors that might account for the results in the experimental group except for the variable under study. In the case of the SFSTs, adjacent jurisdictions might be compared-one department using the SFSTs and another not. Or some members of the department might be trained in the SFSTs, others given other DWI-detection training. Without such a control group, the results observed are ambiguous. Is 90-95% a better accuracy rate than without the SFSTs?20 Was the high accuracy rate due to the quality of the officers? Their sensitization to DWI detection because of their recent training? The fact that they were observed by researchers and supervisors? Significant defects of the SFST field studies as rigorous scientific studies can be summarized in the following five points: 1. The field studies validated the arrest decisions of the officers in the studies, not the SFSTs. Because officers had access to driver behavior and demeanor, the field studies did not specifically test the accuracy of the SFSTs as stand-alone tests. They were not conducted "blind," much less double blind. As stated in the Colorado study, "Some of the information underlying an officer's decision is not documented and cannot be examined." 21 In the San Diego and the boating studies, officers may have also had use of PBTs, which would contaminate the test with the criterion-a fatal flaw. Even in the other two studies, large proportions of the stops were unobserved, so officers could have used PBTs before scoring the SFSTs. In sum, the officers' judgments of intoxication and arrest decisions were not solely due to the SFSTs, and cannot provide solid evidence for SFST validity. 2. The police officers and the degree of supervision in the field studies were not typical of typical DWI stops. In each study, participating officers were highly motivated, highly experienced volunteers. In two studies, they were monitored by either civilian research observers or their colleagues. It is well known that people who are watched tend to perform better-in social psychology this is known as the Hawthorne Effect.22 Supervision likely made officers more attuned to accurate administration and recording than an officer working on his own would be. The very low rate of administration errors reported for the Colorado and Florida studies attest to this, and contrasts greatly with the experience of many DUI attorneys.22 3. The studies are insufficiently documented for scientific papers, a point made in U.S. v. Horn23. For example, two of the SFST studies do not specify the instructions used to administer the tests (the instructions have changed considerably since the initial 1977 study). None of the studies examined the combination of HGN and WAT that is referenced in the NHTSA manuals, or examined interrater reliability (how well different observers agreed on scoring or arrest decisions) or internal reliability (how well the different scoring clues agreed). There is no discussion of the weaknesses or limitations of the studies, as is customary in the discussion section of a published paper. Instead, the Florida study ends with an astonishingly strong conclusion: "There appears to be little basis for continuing legal challenge (to the SFSTs)."24 4. The authors did not report the accuracy of arrest decisions for stops that were observed vs. those that were not, or for SFSTs performed under adverse climate conditions vs. those that were not. This is surprising, since this latter issue was a one of the primary goals of the Colorado study. 5. None of the SFST field studies have been published in peer-reviewed scientific journals. The reports were submitted to state DOT agencies or simply "written up." Peer review exposes the work to the criticism of other researchers and authors who may not share the same beliefs and purposes, and who have training and experience in valid experimental design. The scrutiny that this process brings is crucial to detecting error and bias. Because of the limitations of the field studies cited above, it could be argued that the 1981 laboratory study, and a similar work by non-NHTSA authors,25 remain the primary evidence of SFST reliability and validity. Supporting this claim, NHTSA continues to cite the accuracy figures from the 1981 study in the student manual26 rather than much higher figures obtained in the field studies. Although the laboratory studies were rigorous in some respects, they have several significant limitations: 1) subjects had no reason to fear detection/arrest, 2) testing was conducted during the day rather than night, when most DWIs occur, 3) officers were able to observe, talk to, and smell the subjects, 4) for the NHTSA study, subjects were recruited from the state employment office and are not representative of the general population, and no attempt was made to justify this source as representative of DWI stoppees, and 5) the same subjects were used to create the cutoff scores for the test and to evaluate the accuracy of these cutoff scores. This procedure will lead to inflated estimates of accuracy, because the test decision rules are tailored to the subjects on which it was calibrated.27 The cutoff rules from the first group should be cross-validated on a new group of subjects. The accuracy level achieved in the second group will be an unbiased estimate of the accuracy when applied to a new group of similar subjects, such as DWI suspects, assuming the base rates (frequency) of intoxicated persons are similar in both groups.
A Comment on HGN Difficulties can arise in several other ways when interpreting HGN. Are a subject's eye movements smooth pursuit movements with nystagmus or natural saccadic movements? At least one board certified ophthalmologist wrote that NHTSA's recommended "smooth pursuit" administration (two seconds across each eye) invites saccadic movements because it requires the eye to move too fast.32 The 1981 study authors acknowledged that as many as 50% of people show some nystagmus at maximum deviation in at least one eye."33 In New Hampshire v. Dahood, the court reported "Drs. Citron (an ophthalmologist) and Rizzo (a neuro-ophthalomologist) were adamant in their opinion that the distinct nystagmus at maximum deviation clue should be eliminated from the HGN test."34 Recently, it has been reported that fatigue can induce nystagmus at maximum deviation in 50% of people, and that nystagmus persists after BAC levels have fallen to zero.35 Lastly, the Maryland court of appeals in Shultz v. State recognized thirty-five causes of nystagmus in addition to alcohol.36 Two recent court opinions have held that HGN does not meet Daubert37 standards to be admissible as direct evidence of intoxication or impairment. In U.S. v. Horn, the court held HGN is not generally accepted among psychologists.38 In New Hampshire v. Dahood39, the trial court, on remand from the Supreme Court of New Hampshire on the issue of admissibility, cited an inability to determine error rates and concluded HGN is not generally accepted among ophthalmologists. On appeal, however, the New Hampshire Supreme Court held that HGN does meet the four Daubert criteria, and reaffirmed other state court opinions that the relevant professional communities for HGN include behavioral psychology, highway safety, neurology, and criminalistics in addition to optometry and ophthalmology, where it maintained.40 However, it maintained that HGN is only circumstantial evidence of impairment and cannot be introduced at trial to estimate BAC. The SFSTs as Standardized Tests SFSTs are quite similar to the neuropsychological tests, which detect brain damage and assess sensory, motor, and cognitive impairment. To the extent that the SFSTs are standardized tests, they should meet the relevant professional standards. Standards for Educational and Psychological Testing41 is an authoritative guide that enumerates many criteria for test construction, reliability, validity, documentation, and implementation, and provides a useful introduction to these issues. Some of these are directly relevant for the SFSTs. For example, Standard 1.10 states "When interpretation of performance on specific items, or small subsets of items, is suggested, the rationale and relevant evidence in support of such interpretations should be provided." This is not addressed in the SFST literature. The following table lists the standards that are most relevant for examination of the SFSTs. The next sections address problem areas regarding standardization, reliability, and validation. |