Then you can obviously calculate what the model outputs for any set of inputs. But that's not measuring; that's modeling. They're not the same thing. You can't measure the students' performance with a hypothetical "average" teacher because that teacher does not exist. You can only model what you think are the relevant factors involved and then input actual data into the model to see what it says.
Can such models be informative? Certainly. Are they a substitute for human judgment? No way. Yet that is what these schools appear to be doing.
> if the best teachers at schools full of poor kids could get more incentive pay by switching to schools full of rich kids, would that be incentivising the right thing?
Does that actually happen? And if it does, would this model prevent it? Once again, models are no substitute for human judgment.
The proposed model makes no assumptions about the "average" teacher -- it proposes a simple metric applied to student performance. Under this metric, some teachers would get a positive score because their students' scores improved more than the average. Others would get a negative score because their students' scores improved less than the average.
> would this model prevent it?
I don't know, but the goal of the model is to prevent this, or even reverse it -- that is, the best teachers would want to work with the least advantaged kids, because arbitrage -- it should be easier to make a difference for kids starting at a low performance level, and harder to make a similar difference with kids starting at a higher level.
I don't know whether this works even a little bit in practice, but the thinking sounds promising.
> The proposed model makes no assumptions about the "average" teacher
Sure it does. How else is the teacher_skill rating calibrated?
> the best teachers would want to work with the least advantaged kids
Why is this necessarily a good thing? Shouldn't there also be an incentive to have the best teachers work with the brightest students, to ensure that those students are actually challenged instead of just skating through school?
Put another way: such modeling is probably informative over the aggregate. That is, when comparing school districts to each other. But such modeling is likely not valid for an individual teacher.
Then you can obviously calculate what the model outputs for any set of inputs. But that's not measuring; that's modeling. They're not the same thing. You can't measure the students' performance with a hypothetical "average" teacher because that teacher does not exist. You can only model what you think are the relevant factors involved and then input actual data into the model to see what it says.
Can such models be informative? Certainly. Are they a substitute for human judgment? No way. Yet that is what these schools appear to be doing.
> if the best teachers at schools full of poor kids could get more incentive pay by switching to schools full of rich kids, would that be incentivising the right thing?
Does that actually happen? And if it does, would this model prevent it? Once again, models are no substitute for human judgment.