This has been about 50% of the time my experience as well. There are very good SWE who know how to use ML in real systems, and then there are the others who believe through and through it will replace well understood systems developed by subdomain experts.
As a concrete example, when I worked at Amazon, there were several really good ML-based solutions for very real problems that didn't have classical approaches to lean on. Motion prediction from grid maps, for example, or classification from imagery or grid maps in general. Very useful and well integrated in a classical estimation and control pipeline to produce meaningful results.
OTOH, when I worked at a startup I won't name, I was berated over and over by a low-level manager for daring to question a learning-based approach for, of all things, estimating orientation of a stationary plane over time. The entire control pipeline for the vehicle was being fed flickering, jumping, adhoc rotating estimate for a stationary object because the entire team had never learned anything fundamental about mapping or filtering, and was just assuming more data would solve the problem.
This divide is very real, and I wish there was a way to tease it out better in interviewing.
The lack of knowledge of and application of fundamental engineering principles is a huge issue in the Software world. While its great that people can pick up programming and learn and get a job, I have noticed this is often correlated with people not having a background in Hard Science and Mathematics. Even amongst CS graduates there are a lot of who seem to get through without any mathematical or engineering maturity. Having a couple people in a team with Physics, Mathematics, Mechanical or Electrical Engineering backgrounds, etc. can really be a big asset as they can fight back and offer a classical solution that will work nearly 100% of the time. Whereas someone who just did a Bootcamp and no formal scientific training seems less likely to be able to grasp or have prior knowledge of classical approaches.
I think that this is one reason Software has such a flavor of the month approach to development.
Your stationary plane example highlights a divide I've seen across my work experience in different domains; teams defaulting to ML when fundamental engineering would work better.
I'm curious: do you think there's any amount of high-quality data that could make the learning-based approach viable for orientation estimation? Or would it always be solving the wrong problem, regardless of data volume and delivery speed?
My sense is that effective solutions need the right confluence of problem understanding, techniques, data, and infrastructure. Missing any one piece makes things suboptimal, though not necessarily unsolvable.
> I've seen across my work experience in different domains; teams defaulting to ML when fundamental engineering would work better.
In my current field (predictive maintenance), there are (in)famous examples and papers using multi-layer deep networks for solving anomaly detection problems, where a "single" line of basic Matlab code (standard deviations, etc.) performs better than the proposed AI solution. Publish or perish, I guess...
One opportunity for human-designed systems to excel over machine learning is the case where treating ML as a black box has caused the designers to pose an impossible problem. From the parent comment, it sounds like each additional measurement was being related to a new estimate by the ML system, while the standard technique could integrate measurements over time (that's called filtering).
As a concrete example, when I worked at Amazon, there were several really good ML-based solutions for very real problems that didn't have classical approaches to lean on. Motion prediction from grid maps, for example, or classification from imagery or grid maps in general. Very useful and well integrated in a classical estimation and control pipeline to produce meaningful results.
OTOH, when I worked at a startup I won't name, I was berated over and over by a low-level manager for daring to question a learning-based approach for, of all things, estimating orientation of a stationary plane over time. The entire control pipeline for the vehicle was being fed flickering, jumping, adhoc rotating estimate for a stationary object because the entire team had never learned anything fundamental about mapping or filtering, and was just assuming more data would solve the problem.
This divide is very real, and I wish there was a way to tease it out better in interviewing.