Page 2

Table 2
Selected Standards for Psychological Tests

Standard #
1.10	When interpretation of performance on specific items, or small subsets of items, is suggested, the rationale and relevant evidence in support of such interpretations should be provided.
1.17	If test scores are used in conjunction with other quantifiable variables (i.e., driving errors, odor of alcohol) to predict some outcome or criterion (i.e., BAC), regression (or equivalent) analysis should include those additional relevant variables along with test scores.
3.5	Relevant experts external to the testing program should review the test specifications.
3.6	Test content should be chosen to ensure that intended inferences from test scores are equally valid for members of different groups.
3.9	The process by which items are selected and data used for item selection, such as item difficulty, item discrimination, and/or item information, should be documented.
3.23	Scorer reliability and potential drift over time in the scorer's rating standards should be evaluated and reported…
4.19	… the rationale and procedures used for establishing cut scores should be clearly documented.
5.2	Modifications or disruptions of standardized test administration procedures or scoring should be documented.
5.9	When test scoring involves human judgment, scoring rubrics should specify criteria for scoring. Adherence to established scoring criteria should be monitored and checked regularly. Monitoring procedures should be documented.
6.5	The test manual/documentation should include the standard error of measurement.
6.7	Test documents should specify qualifications that are required to administer a test and to interpret the test scores accurately.
7.2,7.3	If age or other demographic variables effect test performance, these issues should be studied and the test used only for those subgroups for which evidence indicates valid inferences can be drawn from test scores.
9.3	(Tests) generally should be administered in the test taker's most proficient language…
10.1	Test users should take steps to ensure that the test score inferences accurately reflect the intended construct rather than any disabilities and their associated characteristics…

Deficiencies of the SFSTs as Psychological Tests
Standardization Problems - As the name implies, the SFSTs gain their special status because they have been standardized, meaning specific rules for administering, scoring, and interpretation have been specified and researched. Standardization is crucial if research findings are used to support the validity of the tests, since a test that is modified is no longer the same test. As NHTSA states, "If any one of the standardized field sobriety test elements is changed, the validity is compromised."⁴² A number of courts have held that if not properly administered, the SFSTs are not admissible.⁴³

The following problem areas are organized in the chronological order that the SFSTs are administered and scored.

1. Screening questions for possible medical problems and conditions should be standardized and validated. The NHTSA student manual states the officer should ask about certain topics, but does not specify the form of the questions. The wording of a question, and how it is asked, are crucial to obtaining valid data. Screening questionnaires are used in a variety of medical fields. A good screening test should identify virtually everyone who has the condition being queried about-and should be demonstrated to do so. In the case of the SFSTs, the questions should uncover relevant conditions that could invalidate or affect SFST performance. No research has been conducted on this issue.

2. The SFST instructions have changed repeatedly from the initial laboratory studies to the field studies to the current NHTSA student manual used to train police officers.

3. SFST training does not emphasize rigorous adherence to the standardized instructions. Psychologists routinely administer standardized tests. Many, like the Wechsler intelligence tests, come with materials that direct the examiner to read the instructions verbatim. This was my expectation when I learned the SFSTs. Although the NHTSA instructions are given in quotation marks, suggesting they should be delivered verbatim, this level of proficiency is not specifically endorsed. Consequently, students and instructors do not seem to aspire to it. Some training films actually demonstrate inaccurate delivery.⁴⁴

4. SFST training materials do not address how instructions are to be delivered (attitude, speed, and tone). Should the officer be polite? Authoritative? Commanding? Is it OK to be impatient, surly, and condescending? How does this affect performance? What about speed of delivery? Should the officer's demeanor facilitate maximum performance? That is the usual standard for neuropsychological tests.⁴⁵ In contrast, some officers appear to make the tests harder by delivering instructions in a rapid, bored, monotone voice. It is unlikely that the officers in the laboratory studies, using volunteers and monitored by the researchers, adopted the hostile, impatient demeanor sometimes displayed by officers during SFST administrations. To the extent that arresting officers behave differently than the officers in the NHTSA studies (which was not recorded), the validation evidence is diminished.

5. For the Walk and Turn, a variety of line situations are permitted. There is no research on the effect of using an imaginary line, a crooked line, an offset line, or one that the line creates an uneven surface.

6. What constitutes "demonstrates understanding"? For the WAT and OLS, officers are directed to determine that the suspect understands the instructions. A "yes" or "no" question often suffices. If a suspect equivocates, the officer may become impatient and demand an answer. Clearly, this is not an adequate assessment. The tests are designed to test ability to follow directions and perform after the instructions are understood. (Standard 9.3)

7. Scoring rules are often inadequately specified. What constitutes an "inappropriate turn?" In HGN, the examiner must make two passes for each eye to assess each of the three signs. Does the clue have to occur on both passes, or just one? If it occurs on just one, should the examiner administered another pass and make a decision based on two out of three?

8. It is unclear, both in the studies and the student manual, what the criteria are for failing the SFST battery. The student manual provides cutoff scores for each test, plus a decision grid for the combination of the HGN and WAT. What it does not say is what criterion is primary. Thus, a suspect apparently can fail at least four ways (from each of the three tests and from the combination of the HGN and WAT). If the defendant is given multiple chances of failing, the risk of a false positive finding will accumulate with each additional test unless credit is given for those tests passed.

9. Officers are not specifically directed to record their observations immediately. Failure to do so encourages a tendency to assign scores consistent with the officer's arrest decision and, for example, to remember seeing a particular cue in both eyes rather than one. As the authors of the 1981 laboratory study stated, "…many of the advantages of standardized scoring are lost when the scoring is left to memory."⁴⁶

Reliability and Validity Problems
1) The SFSTs have not been subjected to a rigorous "blind" assessment of their validity. As discussed above, none of the studies of the SFSTs have been truly double blind, as expected in medical research. The laboratory studies came close; the field studies do not. (Standard 1.17)

2) The effects of fatigue, sleepiness, circadian rhythm, driver stiffness or roadside conditions on SFST performance have not been adequately investigated. (Standard 10.1) The angle of onset of nystagmus was found to advance five degrees in the hours after midnight, while the other laboratory studies were conducted during daytime hours.⁴⁷ In the 1981 study, the authors stated that exercise, sleep loss, elevated temperatures, and antihistamines are associated with increased body sway.⁴⁸ Strobe and emergency lights, gusts of wind from passing traffic-all have unknown effects on SFST performance and validity given the limitations of the field studies.

3) Drivers suspected of DWI and subjected to the SFSTs may be highly anxious, which alone or in combination with small amounts of alcohol, may influence their performance. In the laboratory studies, subjects were volunteers who had no reason to be anxious, aside from possible self-consciousness. There are theoretical reasons to believe that fear, anxiety, or stress may affect performance on the WAT and OLS⁴⁹, and no study has demonstrated these factors are not relevant.

4) The clues for the WAT and OLS lack documentation of their individual validity and reliability. The validation and reliability data focus solely on the total scores, not the individual clues. It is possible that all eight clues are valid-or that half of them are not. Since there is no published data on this issue, it cannot be assumed that the clues your client failed are valid ones. (Standard 1.10)

5) Reliability data are lacking or below accepted standards for psychological tests used for making decision about individuals. Reliability refers to the consistency with which a test produces results across conditions that can change, such as testing at different times or by different evaluators. Authorities recommend such tests show "a bare minimum" reliability of .90, with .95 "considered the desirable standard."⁵⁰ None of the reliability figures for the SFSTs are this high, and most are much lower. Different raters scoring the same subject at the same time show reliability coefficients between .62 and .74 on the SFSTs, and lower figures (.58-.59) for their decisions about whether the person is impaired and should be arrested. Other NHTSA researchers assessed the SFSTs to be quite low on "Ease of Scoring," providing ratings on a 1-100 scale of "5" for HGN, "25" for WAT, and "30" for OLS.⁵¹ No figures have been reported to assess the internal reliability (coherence) of the SFST items. This is a standard, expected piece of information for a psychological test. The following table displays the only figures that have been reported. ⁵² Reflecting on these figures, the authors candidly admitted, "… the interrater reliability for the nystagmus score is not as high as expected…" ⁵³

Table 3
Reliability Coefficients for the SFSTs

	Types of Reliability and Associated Coefficients
	Test-retest	Interrater
HGN	.66	.62
WAT	.72	.74
OLS	.61	.70

The reliability coefficients are estimates of how much of the test score is reliable-a reliability coefficient of .70 indicates 70% of the score is reliable and 30% is error. However, each reliability coefficient reflects only some of the potential sources of error: The observed score is a function of the quality that is being measured (intoxication) plus numerous sources of error, including who administered the test, the particular occasion and conditions it was administered under, and the quality of the items composing the test. Unfortunately, you cannot simply add up the errors from the different reliability estimates. However, one dramatic illustration of the role of multiple sources of error comes from the 1981 study: The test-retest coefficient for the WAT scored by a different rater is .34, as opposed to .61 when scored by the same rater. The moderate reliability figures cast doubt on the high accuracy rates reported in the field studies, since high reliability is a prerequisite for high validity.⁵⁴

6) Standard errors of measurement (SEM) are not provided. (Standard 6.5) The standard error of measurement is the average amount of error in the typical measurement for that test. The SEM is used to create confidence intervals around an observed score to show how precise the estimate (observed score) is. For example, a 95% confidence interval around a score of 4 on the HGN might be 2 to 6. But NHSTA studies do not include basic descriptive statistics of the data (means and standard deviations) that would allow calculation of these values.

7) SFSTs have not been normed on sober people. As acknowledged in the 1981 study, "Balance tests of various sorts show large individual differences in the performance of sober individuals…"⁵⁵ When most psychological tests are developed, they are tested on a large sample to determine what is "normal." The Personality Assessment Inventory is a self-report test designed to assess psychopathology. Before it was published, the author administered it to some twelve hundred psychiatric patients-the intended population for the test. But he also administered it to over twelve hundred volunteers from around the country. Then, volunteers were dropped in order to obtain a census-projected nationally representative sample in terms of age, race, and education.⁵⁶ The SFSTs have never been administered to a large, representative group of sober people. We don't know what is a normal score.

8) There is very limited data on the SFSTs for people under 21 or over 50-55. (Standard 3.6) Only 3.1% of the NHTSA 1981 study sample used to standardize, calibrate, and validate the SFSTs were older than 55. Reporting of age groups is inconsistent across the field studies, but in all three, people above 50-60 made up a very small portion of the sample. There have been no comparisons made of the validity of the SFSTs for younger vs. older groups. (Standards 7.2, 7.3, 10.1)

9) SFSTs have questionable validity for those who are elderly, in poor physical condition, or overweight. If the SFSTs are of questionable validity for people more than 50 pounds overweight, what about short people who are 45 or 40 pounds over the ideal? Proportionately, a person who is 4'8" and 40 pounds overweight is likely to be more physically impaired than someone 6'3" and 51 pounds overweight. Why does the test suddenly become invalid when one goes from 50 to 51 pounds over the ideal? Obviously, the impediment due to weight is likely to be gradual. The same issue applies to people in their late 50's vs. the arbitrary cutoff of 60⁵⁸ or 65.⁵⁹ Physical health and condition are likely to be more important than age. (Standards 7.2, 7.3, 10.1)

10) Even NHTSA claims the SFSTs, when optimally used, are only 80% accurate.⁶⁰ This is perhaps the most direct and compelling evidence of the SFST validity problems. Although a 20% error rate may be acceptable in a test used for evidence of probable cause of a BAC of .08% or more, it seems insufficient when the SFSTs are used as to establish, beyond a reasonable doubt, intoxication or impairment. Further, consider that the SFSTs were 1) evaluated by the tests' developers, 2) under laboratory conditions, 3) only a fraction of subjects were in the critical .05-.15% BAC range, and 4) the same subjects used to calibrate the tests were used to assess their accuracy. Given all of these potential biases in their favor, a hit rate of 80% is unimpressive.

Another perspective on SFST accuracy is provided by using a bathroom scale as an analogy. Even a cheap scale might be expected accurate within a few pounds. Yet, the NHTSA authors state " …it is unrealistic to attempt to use behavioral tests to discriminate BACs in a +.02% margin around a given level."⁶¹ This is equivalent to a one hundred pound woman stepping on a scale, seeing a reading of 120, and being told the scale is functioning within its design limits. And this is under ideal conditions. But how well can police officers actually estimate individuals' BACs? In the 1981 laboratory study, police officers' estimates of BAC (measured by Intoximeters) were incorrect by an average of .03%⁶² -meaning approximately half the errors were larger than this.

Psychologists often calculate confidence intervals to communicate that a given score, like an IQ, is an imprecise measurement. For example, an IQ of 100 may have a confidence interval of 94-106. If someone obtained an IQ of 100 on one occasion, it is likely that he or she would obtain a score within the confidence interval if tested again. Confidence intervals are not absolute, but based on probability. The most common probability used is 95%, meaning that on 95 of 100 retests, the new score would fall within the confidence interval created from the first score.

Let's return to the analogy of a one hundred-pound woman stepping on a bathroom scale using the SFST BAC estimation errors. Using the most conservative average error reported (.03%), and using standard tools to create a confidence interval,⁶³ we find that a 100-pound woman would observe a scale reading of between 25 and 175 pounds on 95 of 100 trials. The other five percent of readings would be more inaccurate. In the 1981 field study, officers' average BAC estimates were off by an incredible .077% before training and a whopping .0537% after training.⁶⁴ Creating a 95% confidence interval from the "before training" figure (.077%) means our 100-pound woman will weigh anywhere from -93 to 293 pounds on our SFST bathroom scale-95% of the time.

Miscellaneous Issues
1. The SFSTs have been evaluated primarily by NHTSA supported researchers, with no rigorous evaluation by disinterested researchers in a field settings. Replication by impartial researchers is the sine qua non of reliable scientific knowledge.

2. SFSTs have usually been evaluated in high base rate settings where up to 92% of the persons tested were legally intoxicated. Base-rates have a major effect on the confidence that can be given to a test result. In both the laboratory and field studies, the majority of subjects or drivers tested were intoxicated. Aside from the other problems with the studies, generalization to settings (sobriety checkpoints or daytime stops) where the incidence of DUI is much lower is not warranted. An earlier NHTSA study⁶⁶ showed high rates of false positives when the frequency of intoxicated (BAC > .10%) drivers was experimentally set to 48%. The following table from that paper illustrates that HGN, either alone or in combination with observations of driver behavior and appearance, showed false positive rates of up to 75% for those with a BAC between .05% and .09%. Officers who received only three hours of training in administration of HGN assessed 24% of those in the .00-.04% BAC range as impaired-and the great majority of these were probably completely sober.

Page 1 | Page 3