Is AI the right tool for evaluation ?
I have a few observations to share. Let’s examine them one by one.
Unlike the human evaluator, AI evaluator depends on “your” prompts to evaluate your answer.
If you simply say, “Evaluate this answer,” it might focus on content coverage, answer structure, or the application of thinkers, all of which depend on the details of the prompts you provide.. The same answer can receive 5/10 in one chat and 8/10 in another with just a “different prompt”. Human evaluator, on the other hand, doesn’t evaluate on the basis of the prompt you give to him.
Secondly, you are in a power relation with AI evaluator in which you are the master and AI is the Slave of your command. Most of the times, it will give you “encouraging feedback” because the slave cannot afford to annoy the master. Sometimes the master may have to prompt it to give critical feedback. The Slave would now, like a loyal subject, fulfil the wish of the master. Howsoever be the nature of command, the Slave would not be “allowed”and may not even become the “master” in this relation.
Thirdly, many times, AI evaluators tend to ignore grammatical errors, sentence construction errors, and mismatches between statements and examples. Instead, they often “assume” what the writer intended to convey. A human evaluator, however, does not ignore these aspects and is not expected to infer your intended meaning. If a human evaluator overlooks them, a significant portion of the answer remains unevaluated. Therefore, they cannot simply ignore them.
Lastly, tthe limitations of your prompt often become evident when you upload a topper’s copy that appears deceptively simple.. The AI evaluator may even award it a poor score, not because the answer lacks quality, but because your prompt has constrained its judgement. In this strange trilogy of you, the topper, and the AI, everyone is left wondering who actually failed. Was it the topper, the AI, or the prompt ?
The slave can only answer your questions. The human evaluator questions your answers.
