In Lewis (2008), the primary issue was the first prong of the MR determination---level of general intellectual functioning. Lewis had been tested three times (of course, with the traditional adult WAIS-III battery), but his most recent IQ score was based on the Stanford-Binet Fifth Edition (SB5; SB-V), a battery based on the contemporary CHC theory of cognitive abilities. Lewis had full scale scores of 59 (WAIS-III),70 and 79. The later score (SB5 score of 79) was the major point of contention in this decision. Although it is not atypical for attorneys representing either side in Atkins cases to raise challenges about proper administration and scoring of an IQ test, what is unique (in this case) is the filing of an affidavit (by the defense) by the author of the SB V (Dr. Gale Roid). Asking authors of the IQ tests to provide expert input is not the norm in Atkins cases, and does take the issue of proper test administration and scoring to a higher level.
Before proceeding, I should reveal that I am a colleague of Dr. Roids and was initially on the revision team for the SB V (check out acknowledgments section). I did not continue on the project as the revision of the WJ III was ramping up (I'm a coauthor) and I did not have sufficient time to work on both. There is a small fraternity/sorority of applied psychometricians who specialize in developing or revising individualized intelligence tests. Dr. Roid is one. I have tremendous respect for his psychometric skills and professional integrity. Also, by way of background, my interest in starting the ICDP blog originated when I was asked to do the same, but for the battery I am a coauthor (WJ III/BAT III). I did not review taped test administration but was asked to check the scoring and correct use of the computer scoring and proper norms. I will eventually make a post about this experience in the context of the issues of test administration and scoring errors.
It is my professional belief that some (but not all) test authors can provide an important external validity check re: questions raised about proper test administration and scoring of the tests they develop. This is my professional opinion as a coauthor of an individually administered cognitive and achievement test battery. However, in many cases it is not possible to identify a single author that can serve as an expert for a test that may be used in court proceedings. Some IQ tests have multiple authors on the team, each responsible for different (and sometimes overlapping) parts of the tests construction, norming, and statistical analysis. In other cases a test author may be an professional with a strong publication background and solid theoretical knowledge, but it is the internal staff at the test's publisher that do the major test construction, norming, and psychometric analysis....the publishers internal psychometric staff may know more about the technical aspects of a test (that would benefit the court) than a particular author.
There is only a handuful (I can count them on one hand) of intelligence test authors who are involved in all aspects of the development of the test (inclusive of the psychometric analysis and development of the test score norms) and who can provide valuable, objective, external expert information to the courts. Also, some authors may have contractual limitations with a publisher that prohibit them from testifying about the test in legal proceedings. All this being said, it is my professional belief that authors that are appropriately and uniquely qualified (e.g., Dr. Gale Roid) can provide valuable input to courts regading testing issues related to their test. Professionally I believe I have these qualifications related to the WJ III system of instruments and intelligence test development and psychometrics in general. But let’s get back to the Lewis decision.
The SB5 score of 79 was the primary point of argument in the Lewis decision. As often happens in these cases, when IQ testing is conducted, the testing is often taped. This allows for state and/or defense experts to review the IQ testing for the appropriate use of standardized testing procedures (as spelled out by the tests manuals) and possible scoring errors. A defense expert (not Dr. Roid) had “re-scored” the SB5 and came up with an adjusted score of 75, based on the judgment that errors were made in administration in scoring.
Dr Roid concluded that the original administration and scoring of the SB5 was improper as it included "nine procedural errors"--errors that significantly invalidated the resultant score. According to the decision:
Dr. Roid is the author of the Stanford-Binet IQ test administered to Lewis by Dr. Rosin. Dr. Roid reviewed Dr. Rosin's testimony, notes from, and video recordings of, Dr. Rosin's administration of the IQ test to Lewis, and the test results. The conclusions detailed in Dr. Roid’s affidavit were: administration of the test to Lewis was invalid because standardized procedures were not followed, making the use of the published norms questionable; and, the effect of assistance given by Dr. Rosin to Lewis during the IQ test administration would result in an inflated IQ estimate.
Of particular interest for this blog post, is the fact that the appellate courts oral arguments re: the SB5 test administration and scoring are available for all to hear. I found the discussion by the very involved panel and lawyers insightful as they struggled to understand psychometric concepts in their own language and vocabulary. The oral argument can be accessed at the court's argument page [follow the link and type in the docket number - 07-70024 - in the space provided]. Or, a copy of the MP3 file can be downloaded by clicking here [warning...it is a large file]
Some quotes from the taped oral argument that struck me where:
- "Outlier" IQ of 79
- “variances from protocol” --- which meant failure to follow the SB5’s standardized test administration procedures
- "infected the score" --- which was a panel member’s way of communicating that a score had been invalidated
- "dumbed it down" --- a quote referencing the fact that the examiner did not follow the standard directions for a working memory test via not requiring the essential task demand of the working memory test (active manipulation of stimuli while being held in working memory)---thus changing the construct being measured. The test was now measuring an easier aspect of short-term memory.
In my opinion, the decision of the appeals court was correct. They got it right. They vacated and remanded the original district court decision---telling the district court judge to "go back to the drawing board."
I would urge psychologists who engage in intelligence testing to listen to the oral arguments...to be the mouse in the corner and hear how legal professionals grapple with IQ testing and psychometric concepts. It is very interesting. Also, this decision, as well as others I've read, suggest the need for future blog posts that more precisely define the psychometric concepts of the standard error of measurement (SEM) and standardized testing procedures. I will work on those in the near future.
Technorati Tags: psychology, forensic psychology, criminal psychology, criminal justic, psychology and law, educational psychology, school psychology, neuropsychology, IQ, IQ tests, IQ scores, SEM, standard error of measurement, psychometrics, Atkins cases, death penalty, capital punishment, Stanford Binet Intelligence Scale, SB5, SB V, WJ III, BAT III
