Tuesday, November 20, 2012

AP 101 Brief #18: Misunderstanding and misuse of achievement test scores in Atkins MR/ID death penalty cases: Part 2--Range of expected grade equivalents



AP 101 Brief #18:  Misunderstanding and misuse of achievement test scores in Atkins MR/ID death penalty cases:  Part 2 -- Range of expected grade equivalents

Kevin S. McGrew, PhD.
Institute for Applied Psychometrics (IAP)





            In Part 1 of this AP 101 Brief Report (AP 101 Brief #17:  Misunderstanding and misuse of achievement test scores in Atkins MR/ID death penalty cases:  Part 1 -- Range of expected standard scores), the 95% confidence band around the expected/predicted achievement standard score for an individual with an IQ of 70 was calculated to be 36 points (+18), assuming a correlation between IQ and achievement tests of .75.[1]  The point-specific expected/predicted achievement standard score (that accounted for regression-to-the-mean) was 78 (+18; 60 to 96 95% confidence band for expected/predicted scores).  If the reader has not read the first installment in this series, I strongly recommend you stop reading the current brief and read the first brief.  Part 1 provides considerable background information upon which this second part in the series is based.  Below is the visual-graphic summary of Part 1 of this series. [Click on all images to enlarge them for better viewing]

            In Part 2 of this series, the expected/achievement score of 78, as well as the expected range (@ 95% confidence) of standard scores of 60 to 90 are converted to grade equivalents (GE) for Broad Reading, Math and Written Language at ages 25, 35, and 45 years of age in the WJ III NU norm data.  The following general procedure was followed by accessing the WJ III norm tables.  The WJ III NU norm tables were used as they provide data-based values associated with expected standard scores (and GEs) and not values based on prediction equations not based on real data or statistical simulations.
·         For each of the three WJ III achievement clusters, the specific WJ III  W-score[2] associated with a standard score (based on age norms) of 78 was identified.  This W-score was then entered in the WJ III NU grade norm tables to identify the specific GE associated with the W-score.  This step was repeated for the lower (60) and upper (96) standard scores of the 95% confidence band scores—resulting in GE values for both standard scores for each of the three achievement clusters.  This resulted in three GE values at each of the three selected age groups (GE for achievement SS = 78; GE for achievement SS = 60; GE for achievement SS = 96).  These three sets of values were then plotted on graphs and lines connecting each corresponding GE/SS value connect.
The three resulting figures are presented below.  The Broad Reading figure is discussed with the general interpretation being the same for Broad Math and Written Language, although the specific GE values in each figure should be substituted for those discussed with regard to Broad Reading. [Click on images to enlarge for better viewing].



            The Broad Reading GE values associated with the expected/predicted SS of 78 is 5.5 (25 years of age), 4.1 (35 years of age), and 4.4 (45 years of age)—ranging from the beginning of 4th grade to the middle of 5th grade.  This is the bold middle line.  The Broad Reading GE values associated with the expected/predicted SS of 60 is 3.0 (25 years of age), 2.0 (35 years of age), and 2.7 (45 years of age).  This is the bottom line in each figure.  The Broad Reading GE values associated with the expected/predicted SS of 96 is 10.9 (25 years of age), 11.2 (35 years of age), and 11.4 (45 years of age).  This is the top line in each figure.
             Thus, for a person with an IQ score of 70, the expected WJ II Broad Reading achievement GE’s range between 4.1 and 5.5, depending on age.  However, given the large range of standard scores associated with the 95% prediction confidence band (range of 36 points), it is not surprising that this range, when converted to GE’s, can vary from between 2.0 /3.0 to the end of 10th grade and the beginning 11th grade (10.9 to 11.4). 
            A quick review of the figures most likely raises many questions.  For example, why is the distance between the bottom line (GE’s associated with SS=60) and the middle line (GE associated with expected/predicted score of 78) much narrower than the distance between the same middle line and the top line (GE associated with expected/predicted score of 96).  Also, why are the three lines not consistently linear?  The answers to these questions would require excessive detail, statistical explanations, more graphs, etc., that would likely confuse readers.  The answer lies in the fact that (a) standard scores are equal interval metrics and GE’s are not, (b) standard scores are partially derived from the standard deviation (SD) of the W-scores at each age within each achievement domain, and these values are not the same across achievement domains nor across ages, and (c) W-score growth score curves show differential rates of rapid growth during the early ages/grades, then a plateau, and then a much slower rate of decline.  Enough said.
Summary
            Similar to the conclusion from Part 1 of this report, which dealt with expected standard scores, the expected range of GE’s for adults (ages 25 to 45) with an IQ of 70 can, for some individuals, vary tremendously.  The presence of some achievement scores significantly above expectations for an IQ associated with mild MR/ID (70), possibly into the junior and senior high grade levels, are possible when the less-than-perfect correlation between IQ and achievement scores is acknowledged.  One must recognize that although correlations in the .70’s are high and statistically significant, they indicate that IQ scores can only account for up to approximately ½ (50% of tested achievement scores).[3] 
Too many lay persons and, unfortunately many educators and psychologists, have fallen prey to the IQ-Ach fallacy, which is the non-science based assumption or belief that individuals can only achieve at or below their measured achievement.  The appropriate scientific fact is that for any IQ score there is a symmetrical range of possible expected achievement scores which, whether reported in terms of standard scores or GE’s, can be large.  Achievement scores that are above predicted levels based on measured IQ scores will occur with some degree of regularity for individuals with mild MR/ID and should not be incorrectly interpreted as a knee-jerk indication that a person may not considered for diagnoses as MR/ID, assuming they meet all relevant criteria or prongs.
Finally, the all the calculations in Part 1 and 2 of this series are based on the WJ III NU norm data.  The extent to which the results, especially the GE results, generalize to other achievement tests is unknown.  However, I am reasonably confident that although the specific GE's that would be obtained by completing the same methods with the norm data from different achievement tests (e.g., WIAT series) might vary slightly, the overarching conclusion that  "achievement scores that are above predicted levels based on measured IQ scores will occur with some degree of regularity for individuals with mild MR/ID and should not be incorrectly interpreted as a knee-jerk indication that a person may not considered for diagnoses as MR/ID, assuming they meet all relevant criteria or prongs" would generalize. 

[1] If a lower level of IQ/ACH correlation is assumed then the range of expected standard scores (Part 1 report) or grade equivalents (GE) will be larger.
[2] The WJ III scales are based on Rasch Item Response Theory (IRT) scaling methods that results in raw scores being converted to the equal interval W-score growth metric, which is then used to calculate all derived scores (AE, GE, SS, etc.)
[3] This percent figure represents the coefficient of determination which is calculated by squaring a correlation (e.g., r = .70 squared is .49) and then multiplying the value by 100% (thus, 49%).