There are four main levers for improving an ML system:
1. You can change the training data.
2. You can change the objective function.
3. You can change the network topology.
4. You can change various hyperparameters (learning rate, etc.).
From there, I think it is better to look at the process as one of scientific discovery rather than a software debugging task. You form hypotheses and you try to work out how test them by mutating things in one of the four categories above. The experiments are expensive and the results are noisy, since the training process is highly randomized. A lot of times the effect sizes are so small it is hard to tell if they are real. The universe of potential hypotheses is large, and if you test a lot of them, you have to correct for the chance that some will look significant just by luck. But if you can add up enough small, incremental improvements, they can produce a total effect that is large.
The good news is that science has a pretty good track record of improving things over time. The bad news is that it can take a lot of time, and there is no guarantee of success in any one area.
1. You can change the training data.
2. You can change the objective function.
3. You can change the network topology.
4. You can change various hyperparameters (learning rate, etc.).
From there, I think it is better to look at the process as one of scientific discovery rather than a software debugging task. You form hypotheses and you try to work out how test them by mutating things in one of the four categories above. The experiments are expensive and the results are noisy, since the training process is highly randomized. A lot of times the effect sizes are so small it is hard to tell if they are real. The universe of potential hypotheses is large, and if you test a lot of them, you have to correct for the chance that some will look significant just by luck. But if you can add up enough small, incremental improvements, they can produce a total effect that is large.
The good news is that science has a pretty good track record of improving things over time. The bad news is that it can take a lot of time, and there is no guarantee of success in any one area.