To debug a program you actually verify that it's logic is correct (at least the good kind of debugging).
Consider a spectrum:
1. Natural language - we humans combine fragments of natural language easily and on an ad-hoc basis. We can get a fairly amount of use from reusing Shakespeare quotes and neologisms while spending rather little effort.
2. Trained programmers can reuse and combine general purpose libraries - with difficulty and often after considerable debugging.
3. AI algorithms like Neural Networks. These are just plopped in and tweaked, not combining seems possible.
It seem like "intelligent behavior" should be going more towards #1 but the process of Machine Learning seems to move things more towards #3. The "learn once, understand never" approach means that for each significant case, you'll need to do a re-tweaking and re-learning. The potential to get harder rather than easier over time might well be there.
Admittedly, all this is in manner of speaking but still, I would claim that most if not all the times you debug a program you are also debugging your mental concept of what the program does. By that fact that we can change our concepts, our minds are very "debuggable."
To debug a program you actually verify that it's logic is correct (at least the good kind of debugging).
Consider a spectrum:
1. Natural language - we humans combine fragments of natural language easily and on an ad-hoc basis. We can get a fairly amount of use from reusing Shakespeare quotes and neologisms while spending rather little effort.
2. Trained programmers can reuse and combine general purpose libraries - with difficulty and often after considerable debugging.
3. AI algorithms like Neural Networks. These are just plopped in and tweaked, not combining seems possible.
It seem like "intelligent behavior" should be going more towards #1 but the process of Machine Learning seems to move things more towards #3. The "learn once, understand never" approach means that for each significant case, you'll need to do a re-tweaking and re-learning. The potential to get harder rather than easier over time might well be there.