The autumn 2010 release by the National Research Council of A Data-based Assessment of Research-doctorate Programs in the United States was anticlimactic for many, and highly frustrating for others (see ACA Newsletter – Education Europe, September 2010). And rather than dissipating over the last seven months, the controversy surrounding this project has recently been rekindled with the announcement by the NRC on 21 April that it was revising some of the findings from last year, on the basis of having discovered “four substantive errors”.  

The NRC indicates that the adjustments were made on the basis of “communications and queries received” in the period since the results were announced last September, and relate specifically to four variables:

• average citations per publication, with the publications for one specific year  (2002) having been originally “mislabeled in all non-humanities fields”;
• awards per allocated faculty member, which had been undercounted;
• percent with academic plans, which was changed from being calculated as a percentage of all PhDs awarded to a percentage of respondents to the question, given that the response rates varied so considerably on this question; and
• percent of first-year students with full financial support, which was changed to “N/D” in programmes that had no first-year students, whereas such programmes had originally been given the value of “0”.

Most of these adjustments caused programmes to fluctuate slightly on the assessment’s performance scales. In a few cases, more dramatic changes in standing (largely improvements) resulted, which has caused some critics to point out that the approach taken by this project was ultimately too sensitive to minor changes. In a more dramatic development, Jonathan R. Cole, a former member of the NRC committee responsible for conducting this assessment effort, published a highly critical opinion piece on 24 April in the Chronicle of Higher Education, in which he described his reasons for having quit the committee before the project’s end, and why he had determined that “the report’s quality was not worthy of publication”. Ultimately, according to Cole, the project’s inability to admit failure – and to learn from that process – was its biggest failure. The assessment of this controversial assessment exercise looks likely continue.

The National Academies

Chronicle of Higher Education