What does it mean to understand something and how can you test that of others?
I expect much of it is mastery of language: using mathematical language the correct way; having experience of doing the calculational bits of maths efficiently — algebraic manipulation, etc; being able to use (and learn) the libraries of a programming language; being able to use and extract common patterns of programming — learning the semantics of loops, etc, is not enough (one of the failings of formalist approaches is emphasising this) pragmatics and idioms are also essential. I expect a lot of people get stuck in maths and programming because they insist on doing everything from first principles, to search for some non-existent meaning, when it’s essential to use analogy and to take advantage of examples and counterexamples (of code, of mathematics) wherever possible.
Here’s some stuff from Psychological Knowledge by Kusch (1999). This is in the depths of an argument about how folk-psychology (e.g. talk about beliefs, wants, desires, etc) is a social institution, but try twisting it a bit…
“To follow a practice is to be able to distinguish between right and wrong ways of going on; it is to be able to characterise actions as belonging to a given practice; it is to be able to argue with others over whether or not a given way of `going on’ is acceptable; it is to be able to sanction others if they fail to go on in the right sort of way; and it is to be able to self- adjust one’s own doing in the light of others’ feedback. […] We would not normally say that a person possesses a concept unless the person understands the concept; and we would not normally say that the person understands the concept unless that person can engage in the sorts of activities that I have mentioned above as being involved in rule following.”
I think this is more than a written exam can test. Generally, questions in an exam are very close to questions given as examples, whereas real world problem solving can often involve spending time fighting with books, getting stuck for days, asking for help. People can do exceedingly well in exams without being able to argue convincingly for and against the techniques they’ve used. There’s no opportunity to test to see how well someone will interact and benefit from interaction with others — which is surely crucial for actual practice. I don’t mean in the fluffy “people skills” way that professional practice courses try to teach: I mean learning how to ask useful questions, understanding the relativity of premises and when to modify what you believe, reasoning credulously from what someone is saying before taking a skeptical stance, spotting rhetorical devices and ignoring them, and so on.
I liked this bit in the intro to Bridges from Classical to Nonmontonic Logic by David Makinson, on how to read the book:
“Do it pencil in hand; scribble in the margins. Take nothing on faith; check out the assertions; find errors (and communicate them to the author); pose questions.
“Notoriously, the best test of whether one has understood a definition is to be able to identify positive and negative examples. Nor has one understood a theorem very well if unable to apply it to straightforward cases and recognize some of its more immediate consequences. Without this one may have the illusion of understanding, but not the real thing.
“[…] As well as exercises, there are some problems. They are more demanding. In general they require more than checking positive and negative instances of a definition or straight applications with a couple of steps of argument. Their solutions may need perceptive guessing plus the ability to prove or disprove the guesses. Both the art of making good guesses and the ability to check them out are acquired skills, and grow with practice.”
Incidentally I think a lot of programming is the art of making good “guesses”, where a guess is probably the result of exposure to lots of examples, gaining lots of experience. Then solving the problem at hand is the skill of transforming the initial guess into something that works and justifying why it works.