> If you want to evaluate how good a teacher is, one of the things you need to measure is how well their students performed vs. how well those same students would have performed under the hypothetical average teacher.
But you can't measure that.
> it seems obvious that a set of students doing well does not, in isolation, indicate their teacher is a good teacher.
True. But in this case, at least, we're not talking about "in isolation". We're talking about a 17-year track record of a teacher's students doing well.
> what's a better way to do it?
When I went to school, the people who made these judgments were my parents. You can't make these judgments by formula, and you can't make them if you don't know the details of each individual case. To me, the fact that so many schools are fixated on "data-driven" student evaluations means that parents are not engaged.
>"When I went to school, the people who made these judgments were my parents. You can't make these judgments by formula, and you can't make them if you don't know the details of each individual case. To me, the fact that so many schools are fixated on "data-driven" student evaluations means that parents are not engaged."
Many parents know their children are receiving a substandard (or even damaging) 'education'; the problem they often face is that they are powerless to fire the teachers, or switch schools. Being 'engaged' does basically nothing to fix the schools in these cases. Rich parents exercise school choice by moving to affluent communities with good schools, but the poor often do not have this option.
The VAMs aim to reliably give the school administrators access to the knowledge the parents (and often principals) usually already have, as well as give cause for discipline, incentive, or firing.
> Many parents know their children are receiving a substandard (or even damaging) 'education'
Many parents believe that.
Many parents have children for which that is true.
The overlap between those two sets, OTOH, may be smaller than you think.
> the problem they often face is that they are powerless to fire the teachers, or switch schools. Being 'engaged' does basically nothing to fix the schools in these cases.
IME, this isn't really true -- but the perception it is true results in parents not being engaged. When parents are active in addressing perceived problems with teachers in public schools, it is very effective in getting those teachers out of the classroom. (And I've seen it happen numerous times both to bad teachers based on parents acting reasonably based on real problems, and to good teachers based on parents acting unreasonable out of offense that their special-snowflake children weren't getting handed grades on a silver platter. Sometimes it doesn't mean the teacher gets fired, sometimes they get laterally transferred or technically promoted to a position with the school system which is out of the classroom and not dealing with students, and sometimes they just voluntarily leave teaching. But parent activism is quite effective at getting teachers out of the classroom.)
I could be interpreting your comment wrong, but it seems to put the onus heavily on the teacher, which seems a bit naive. My friend just started teaching in New York actually. And she specifically wanted to teach at a Title 1 school (i.e. poor) in order to reach students with less income and opportunity (she teaches an East Asian language in a predominantly non-Asian minority community). That was the idealized plan anyway. Yet, the school won't give her basic supplies. There is no lounge. No fridge. Ok, fine, that's tolerable (odd to me since I too was raised in an affluent community). But the teacher's need paper. There is a locked up supply room full of supplies, but the teachers are told they can't have access to it. Why? Nobody knows. Printing rights are curbed.
And on top of that all, NYC DoE requires a masters. That equals debt, if you went to a "good" school. But the DoE pays crap. Oh and for some reason at one point, the DoE lowered requirements to be a principal. 3 years of part time teaching, and you could be a principal. Her principal taught dancing for 3 years, and is now purportedly fit to analyze the effectiveness of foreign language teaching methods. But anyway, that's beside the point, that's just describing the environment.
Her program is new. But instead of asking students/parents and filtering for those who might be interested in an East Asian language, the administrators decided to randomly force students into the class, regardless of their prior language history or their year. You know what you get with that? Hostile students. The kind that scream fuck you in your face. OK, fine, just "standard" difficult students. But like any other job, if your "manager" has your back, you can usually deal. But this school doesn't believe in detention. Okay... The "dream" is that if a student is causing trouble, the deans will talk to said student. Uh oh, what wasn't accounted for was the saturation of the deans time due to trouble students. Now you got deans telling you that, sorry, they can't deal with disruptive students telling you to fuck off because they're overloaded. So now you have fire support and you're in the trenches alone. I'm not even going to get into the gay teacher's story. Once the students caught onto that...
This doesn't even include off-the-clock work that is required, which I won't get into. I'm sorry, but as someone in the tech field, or any privatized field, the shit that teachers have to put off with is insane.
I know teachers have been demonized, but the turnover rate in NYC for teachers is apparently extremely high (I have another friend working at the DoE itself). I'll have to get a source, but I seem to recall her saying it was around 70% after 2 years. And after hearing all the ridiculous war stories, I'm not surprised. Ha, another one of my teacher friends was moved to an empty classroom. Upon asking for desks, the administrators told her "we don't know where they are" and left it at that. So she had to essentially salvage desks marked for discard.
I'm not exactly sure how VAM works, but I'm skeptical that an algorithm can model something so complex.
> I'm not exactly sure how VAM works, but I'm skeptical that an algorithm can model something so complex.
And even if it could, should an algorithm like this be obscured from view? Should we rely on "black box" algorithms like this, or should we at least insist that, if not the code, the research behind the code be released openly so that it can be held to proper scrutiny?
People are not engaged. They are fixated on numbers, models, and abstractions, with the assumption that the numbers, models, and abstractions have actual meaning tied to them. They don't, aside from creating more students that create more systems that define more numbers, models, and abstractions. Some of these students get pissed off and try to create the opposite.
This is why the humanities are important. You can have the soundest logical systems, with the most elegant mathematical models, and completely miss the point. People get so caught up in the minutiae of measurements that they forget to see the big picture. Measurements don't mean anything when the measurements are used to measure themselves.
I swear, sometimes I really wonder whether society has it's head up it's ass. If these systems can model 'theoretical students' under 'better conditions', then why aren't the 'theoretical student' models running the world? Oh, that's right, because there's a gigantic difference between data and theory. How do they even know what a better student is? How can anyone in their right mind, define that? How can anyone even pretend to know what a great student is?
> When I went to school, the people who made these judgments were my parents. You can't make these judgments by formula, and you can't make them if you don't know the details of each individual case.
When I went to school, I used to think my grades meant something more than being a very complicated way of validating someone else's world view, in a way that tricks everyone into thinking we've made any progress at defining or understanding intelligence at all.
> To me, the fact that so many schools are fixated on "data-driven" student evaluations means that parents are not engaged.
The parents may be extremely engaged, but they just don't know whether to call their kid smart or not. If the kid complains about the book they read because it was boring, is that appropriate? If the kid programs a calculator to do 4 years of standardized test math homework, is that appropriate? Education is doing a fantastic job at driving the personality out of people by forcing them all to sit in the same box. The parents don't know, the teachers don't know, the government doesn't know, society doesn't know, yet we all pretend we know.
score_at_end_of_year = score_at_start_of_year + a * parents_income + b * parents_education_level + c * teacher_skill + random_noise
You can figure out the values of a and b by comparing the performance of different students in the same class. If you have a large enough sample size to average out the random noise, you're left with (c * teacher_skill).
Not a perfect method, obviously - but if the best teachers at schools full of poor kids could get more incentive pay by switching to schools full of rich kids, would that be incentivising the right thing?
Then you can obviously calculate what the model outputs for any set of inputs. But that's not measuring; that's modeling. They're not the same thing. You can't measure the students' performance with a hypothetical "average" teacher because that teacher does not exist. You can only model what you think are the relevant factors involved and then input actual data into the model to see what it says.
Can such models be informative? Certainly. Are they a substitute for human judgment? No way. Yet that is what these schools appear to be doing.
> if the best teachers at schools full of poor kids could get more incentive pay by switching to schools full of rich kids, would that be incentivising the right thing?
Does that actually happen? And if it does, would this model prevent it? Once again, models are no substitute for human judgment.
The proposed model makes no assumptions about the "average" teacher -- it proposes a simple metric applied to student performance. Under this metric, some teachers would get a positive score because their students' scores improved more than the average. Others would get a negative score because their students' scores improved less than the average.
> would this model prevent it?
I don't know, but the goal of the model is to prevent this, or even reverse it -- that is, the best teachers would want to work with the least advantaged kids, because arbitrage -- it should be easier to make a difference for kids starting at a low performance level, and harder to make a similar difference with kids starting at a higher level.
I don't know whether this works even a little bit in practice, but the thinking sounds promising.
> The proposed model makes no assumptions about the "average" teacher
Sure it does. How else is the teacher_skill rating calibrated?
> the best teachers would want to work with the least advantaged kids
Why is this necessarily a good thing? Shouldn't there also be an incentive to have the best teachers work with the brightest students, to ensure that those students are actually challenged instead of just skating through school?
Put another way: such modeling is probably informative over the aggregate. That is, when comparing school districts to each other. But such modeling is likely not valid for an individual teacher.
Yes, and then you can figure out different values of a and b for a different teacher, different class, different year, different town, etc.
And you're really left with c*teacher_skill + noise, where the noise in the teacher_skill measure may be what results in some good teachers getting bad ratings.
In my view, empirical modeling assumes that there is some underlying regularity that can be modeled, but this assumption seems questionable. And part of the process has to be to decide an acceptable level of good teachers getting bad ratings.
But you can't measure that.
> it seems obvious that a set of students doing well does not, in isolation, indicate their teacher is a good teacher.
True. But in this case, at least, we're not talking about "in isolation". We're talking about a 17-year track record of a teacher's students doing well.
> what's a better way to do it?
When I went to school, the people who made these judgments were my parents. You can't make these judgments by formula, and you can't make them if you don't know the details of each individual case. To me, the fact that so many schools are fixated on "data-driven" student evaluations means that parents are not engaged.