February 10, 1999

Prof. James D. Unnever

Radford University

Department of Sociology

Radford, VA 24142

Dear Professor Unnever,

I am pleased to inform you that your article, "District variations..." (Ms. 97 -1 75), has been accepted for publication in the Economics of Education Review subject to the following revisions:

1. Address the concerns of the referees (see attached). While there will be a philosophical gap between you and referee 3, I would like to see Greater convergence. At the minimum, you can state that some view the traditional educational production function that includes SES variables as flawed, because SES is highly correlated with per pupil expenditures, and therefore some of your results may unduly reduce the effect of schooling variables. One way to get around this is to estimate the production functions with and without SES variables and see how the results compare. It would also be desirable to cite research on both sides of the controversy. In sum, I would like to see a more balanced paper that describes alternative models and discusses results without revealing a certain philosophical point of view of whether "money matters or not. I also agree that federal spending should be removed, because it present a downward bias on the relation between school resources and output (because federal spending is largely directed to disadvantaged pupils). You may also want to speculate on how recent changes in funding in Virginia might influence the results.

2. I enclose a form that I would like you to complete and a "reference standardizations list for the reference section. Please prepare an electronic manuscript along the lines shown in the yellow flier (enclosed), but do not yet send the file to me.

3. Please send me two copies of the revised version of the paper.

You will be notified about the final decision and date of publication as soon as we have an opportunity to examine your revision.

With best wishes

Elchanan Cohn

Editor

Reviewer #1

1. The main contribution of this paper is that if ability is not taken into account, the effects of resources on measures of achievement might be inflated. The authors also conclude that teacher qualifications have a positive effect on achievement. I would add a note of caution to the author's conclusion that teacher's qualifications matter that much. A positive correlation between qualifications and attainment does not necessarily prove that qualifications matter that much. Further, if I am reading the paper correctly the author shows that more affluent districts have less qualified teachers! Should one conclude that more affluent districts are at a disadvantage?

2. Why don't you adjust for all of your school-resource variables in one equation? The correlation coefficients between these variables are not that high.

3. The paper could be shortened by excluding Tables 1 to 4 and Figure 1. The correlation matrix is not needed as well.

Reviewer #2

February 9, 1999

Professor Elchanan Cohn, Editor

Economics of Education Review

College of Business Administration

University of South Carolina

Columbia, South Carolina 29208

Dear Professor Cohn:

Enclosed please find my referee's form pertaining to the revisions of "District Variations in Educational Resources and Student Outcomes" (MS #97-175). I am recommending that you accept the paper (with only one possible change which I will describe briefly in this letter; I defer to your judgement on whether you think it is worthwhile).

I think the authors' deserve credit for responding to what turned out to be a rather daunting list of suggestions/demands. I felt my own original comments were largely addressed (although I think the quantitative interpretations of the empirical results is still opaque at best). Referee #3 appeared quite critical but, in my view, the authors' have made a good faith effort to integrate the main points raised, and the paper is better for having done so.

I do think the empirical results are now weaker from the standpoint of whether "money matters", a point that the I sense the authors are unwilling to accept philosophically. I base this inference on the tone of the concluding section. In other words, I think the authors reveal a pretty clear professional bias on this issue that detracts from the scholarly effort. The paper might benefit if this tone could be eliminated or softened. Perhaps you got the same impression, and if so, you might want to suggest this one editorial change.

Sincerely,

Reviewer #3

District Variations in Educational Resources and Student Outcomes

Manuscript 97-175 Revised

x The article should be rejected.

This research is similar in design to the Coleman study and has many of the same problems. It does not advance the debate, uses data that are almost a decade old, and repeats past errors thus ignoring recent research.

Some of the problems with the research are discussed below.

o Variable misspecification. 1) The research includes federal assistance in the measure of per-pupil expenditures. This revenue supplements state-local spending and should not be included in the variable according to scholars. It is distributed to low-achieving students confounding assumptions--that more money relates to higher achievement. 2) Per pupil expenditures is said to include classroom related expenditures--define what was excluded and what federal funding was included. 3) Controls for student ability--the authors state that standardized scores were used for 11th grade (is this the Stanford test?) and percentiles for 4th grade (Iowa Basic Skills test). How were percentiles used to estimate achievement gains against standard scores? Explain: if two different tests were used the achievement gain measure is invalid.

o The model is flawed. It does not "take into account researchers' criticisms of the prior research", as the authors state on page 12. For example, it does not address data attenuation, or threshold effects. Importantly, it erroneously includes home background factors first in the final regression (stepwise/hierarchial) in an attempt to "correct" for a student's SES. This results in overlapping variance being credited to the background of the student rather than school resources. This problem has been well recognized and criticized in the literature but has been repeated in this research.

Almost 15 years ago Marshall Smith pointed out this same problem in assessing the Coleman report. He notes that

"Coleman et al. chose to assess the importance of the school wide factors eliminated variations shared with Home Background. But an assessment of the potential effects of school resources would have to take into account the possibility that the apparent effects of family background are actually the results of advantaged children attending better-than average schools." (p. 239) In suggesting alternative models and methodology he notes that: "First, the relationship between the school wide resources and student achievement should be shown without first allocating all of the shared variance to the Home Background [SES] of the student" (pp. 240-241). Home background not only shares some of its explanatory power with school resources but the school wide resource factors overlap. (241) (in Mosteller, F. and Moynihan, D. P. On Equality of Educational Opportunity, Random House, 1972).

According to Kern Alexander and Richard Salmon (1995):

The order of variable entries in the equation was a major factor in the results [of the Coleman report] since interaction variance was added to the first termed entered into the equation, weighing its explanatory power unjustifiably and inaccurately. Family background was entered first and school inputs last. Thus, family background consumed explanatory powers that may have belonged to schooling." (p.360, 1995)

o Data Aggregation. There should be an attempt to discuss both sides of the debate. For example, there is a unbalanced reliance throughout on Hanushek, Rivkin and Taylor who argue that aggregated data create model misspecification and an upward bias of resource effects. The author's assertion is debatable and should be stated as such (see J. S. Akin & T. J Kniesner-Journal of Human Resources; also, Rizzuto and Wachtel).

o The data are almost a decade old. Since then Virginia has had a court challenge, changed its distribution system and level of resources. Several poor districts are omitted.

o Table 5 should include resources in addition to test scores and background factors. By omitting resources from the final calculations, the interaction effect of resources and SES is not specified; and all variance is credited to social context factors. This is a major problem. Tables 3 & 4: overall R2 is needed; when a variable is entered first, it should be placed first on the table with increases in R2 noted for each subsequent variable. Perhaps this final regression should include all factors without correction for background variables, but with correction for student achievement (if it can be specified correctly).

o The title might be: the effect of background variables on student achievement and resources.

o References: The references need attention. In reviewing the first 10 pages only, several questions arose.

Page 1: Verstegen, 1994-not in references. Cited under Verstegen (1987) is an article apparently written by Walberg and Fowler. The 1994 article indicates that the author did not indicate a controversy including "student outcomes" as indicated on page 1.

Page 4: Kozol-not in references. It is questionable whether his work relies on "anecdotal evidence".

Page 5: Parrish and Fowler-there

is another author (2nd author? that should be cited).

Page 6: Walberg and Fowler-not in references.

Page 7: Verstegen, 1990-not in references-can you quote where these authors say that student outcomes should be correlated with per-pupil expenditures? At least one that are cited and that I reviewed indicated the opposite view.

Page 9: Griffin and Ganderton,1996-this research relates to earnings not student outcomes as suggested by the authors. Blackburn and Neumark, 1993, 1995-do authors refer to social context or only ability?

Page 9: According to the author(s): The critics [of production function analysis--including referenced Alexander (K) and Salmon] say that misspecification occurs when researchers fail to control....for the social context in which school systems operate. Note: Alexander and Salmon among others find that the order of entry of the variables is problematic when family background factors are entered first.