This is the correct progression IMHO. I can tell you’ve been in industry because it mimics my experiences.
Always start with a simple model and see how far you can get. Most of the improvements I’ve seen comes from “working the data” anyway. You will be surprised how much you can improve model performance just by working the data, or improving the quality of the underlying data alone. Also simple models give you a “baseline”. What is the point of reaching for neural networks if you don’t have a baseline performance metric to compare against? XGBoost is a godsend. It trains extremely quickly and is surprisingly difficult to beat in practice.
As you say, constantly sharpen your saw with regards to probability theory and mathematics in general. There is simply no way around this in the long run.
Always start with a simple model and see how far you can get. Most of the improvements I’ve seen comes from “working the data” anyway. You will be surprised how much you can improve model performance just by working the data, or improving the quality of the underlying data alone. Also simple models give you a “baseline”. What is the point of reaching for neural networks if you don’t have a baseline performance metric to compare against? XGBoost is a godsend. It trains extremely quickly and is surprisingly difficult to beat in practice.
As you say, constantly sharpen your saw with regards to probability theory and mathematics in general. There is simply no way around this in the long run.