Saturday, May 14, 2016

Can VAM be Fixed? Judge Throws Out VAM Eval. in New York

This Washington Post article reports that A New York judge has ruled that an "ineffective evaluation" of a  teacher because of the use of the Value Added Model (VAM) was arbitrary and capricious and invalided the evaluation. This case could have implications for Louisiana teachers because a very similar scheme is used as part of the Louisiana VAM evaluation system.

The Louisiana legislature is now in the process of considering legislation that would modify the Louisiana teacher evaluation plan by reducing the VAM portion of a teacher's evaluation from a weight of 50% to a new weight of 35%. Senate Bill 477 (click on bill # to see text)  by Senator Boudreaux has passed the Senate and is headed to the House Education Committee for consideration next. This bill is supposed to be a compromise agreed to by the hard line proponents of VAM (The Louisiana Association of Business and Industry or LABI) and the Edwards administration and teacher organizations. But even if this change in the weight of VAM is approved by the legislature, there are still other matters not yet settled that are similar to the New York case that could still derail VAM in Louisiana.

The major problem still remaining with VAM in Louisiana is because of the arbitrary ratios of teacher performance levels in the VAM calculation. The use of a Bell Curve to arbitrarily rate a certain percentage of teachers (10%) as ineffective each year and then a certain percentage (20%) as highly effective regardless of overall student performance will not hold up in court in my opinion. (See this video by Herb Bassett) The judge in New York specifically ruled that there was no justification for using a bell curve to determine a certain percentage of teachers ineffective. That was part of the reason for his "arbitrary and capricious" ruling against John King who was state commissioner of education in New York at the time of the evaluation in question. Some critical flaws in the VAM system were obvious from the start.

Within a few months of John White being appointed as State Superintendent of Education in Louisiana, I was granted a meeting with him to discuss certain issues I wanted to bring to his attention. The plan for the new Louisiana teacher evaluation system which included VAM had just been released but had not yet been approved by BESE. I wanted to try to convince White that two parts of the plan were flawed and should be changed. By the way, the draft of the teacher evaluation plan had actually been written by a non-educator (Rayne Martin) who was serving on the State Department of Education staff for a brief period. It was partially based on an observation system developed by Charlotte Danielson, who later disavowed any support for the Louisiana system because it did not effectively adhere to her program. In the course of my meeting with White, I asked what was the rationale for the 10% ineffective rating each year for all teachers who teach state tested students? He could not give me a rationale except that he felt that this was a reasonable percentage. I posed to him the following scenario: Assuming that the purpose of VAM is to encourage overall improvement in student performance by evaluating teachers using the Value Added Model, what would happen to teacher evaluation results if Louisiana actually produced better overall performance over a period of several years? The VAM system requires that the bottom 10% of the teachers get a rating of "ineffective" each year. But if student performance goes up all over the state and suppose that 98% of the teachers have more growth in their students scores than the year before, why would the evaluation system still find 10% of teachers ineffective? But that's what it does every year without regard to overall student growth. White did not make the change in the Louisiana plan when he should have. This is part of the reason the New York judge ruled that state's evaluation system arbitrary and capricious.

The reduction in the percentage of a teacher's evaluation based on VAM reduces the impact of an "ineffective" VAM component on the final evaluation after figuring other student growth factors and the observation component, but it could still put VAM rated teachers at a disadvantage compared to other teachers rated using Student Learning Targets. Also, it should still be expected that a smaller percentage of VAM rated teachers would be able to qualify for a highly effective rating and benefit from merit pay as compared to teachers of non-tested subjects.

Important notice: I reviewed my notes on the recommendations of the special subcommittee of the Accountability Commission that was required by a law passed by Rep. Hoffmann to make recommendations to BESE for changes in the evaluation system. I found that the subcommittee recommended that the "override" provision of the evaluation system be removed from the evaluation plan. The override provision mandated that when a teacher got an "ineffective" VAM score, the qualitative portion (the principal's evaluation) of the evaluation was overridden and the teacher received an "ineffective" overall evaluation. It is my understanding that this provision has been removed. Also, an official of the LDOE has informed me that the latest revision of the evaluation plan allows the principal to raise or lower the VAM by one level of performance based upon other measures of student progress.

My point is that the so called "compromise legislation" in SB 342 still leaves important issues in the teacher evaluation system unresolved. John White, in a letter to district superintendents has praised the VAM compromise. Now John White has another opportunity to try to fix the VAM system. Will he do his part in fixing the rest of VAM? But the real question is Can VAM be Fixed?


Daniel Granfors said...

Your last question is easy, NO!. Statistical process control developed for livestock has no place in evaluating the performance of students, teachers, or schools. It is just another invocation of "junk science" in our lives. What disturbs me most is the "postulate", held by almost everyone in Louisiana and indeed the country, that the lamentable condition of education in this country is due to ineffective/lazy/incompetent teachers. Who are these teachers? In 20 years in private, public, and now adult education, I have met very, very few. I have met scores of dedicated people, striving to do their jobs amid third-world conditions, while being slandered by blowhard politicians and ass-bags in business suits, and having to implement asinine ideas and programs which serve profiteering corporations. I not only believe that VAM cannot be fixed, I believe the whole apparatus should be scrapped.

Michael Deshotels said...

Thanks Daniel. My sentiments exactly!

Anonymous said...

While I appreciate everything you are doing and always look forward to your updates, you have mentioned teachers who are evaluated using SLT's having an advantage over those evaluated using VAM several times. As a teacher at a school where every student is special ed and therefore has SLT's I can assure you that they are used to fail teachers just like VAM is. They are arbitrary, often designed by administrators instead of teachers, not tied to IEP's or actual GLE's, are not properly pre- and post-tested with any legitimate assessment and clearly rigged to be sure there are teachers who fall into the bottom 10%. Even when students show significant growth on all legitimate assessments given a special ed teacher can be scored as ineffective. To us it appears that determinations are made at the beginning of the year to be sure the numbers are what the administration wants them to be. It seems that there are people besides John White who think that creating a man-made bell curve proves something. What that is I don't know. It's particularly sad given that any administrator should be looking to have teachers who are all at the top of the heap. yet top of the heap teachers are being told they are bottom of the barrel. maybe SLT's are used appropriately elsewhere. All we know is that they are, for us, no better than VAM.