How should we measure hallucinations?

[findmyway]

By manually reviewing generated responses, I find the most serious hallucinations come from misuse of built-in functions.

• In some cases, LLM uses islower to determine lower case letters. The actual function should be islowercase.
• The digits function is used in many problems. However, it doesn't understand the expected result with negative numbers.
• Also, the mean function is not included in Base

I do not have any good idea of how to capture these cases quantitatively.