I generally agree with the position that unit tests should be used with discretion, and that full coverage via unit tests often leads to thousands of low-ulitility or redundant tests, and so on. However I cannot agree with this:
>They also tend to slow down CI because they're slow and always flaky.
In my experience, unit tests are the most stable, the least flaky, because they touch the least code and often have very simple setup. An integration test might rely on four database tables being just-so, and go on to connect with two external services (and whether mocked, replayed, or live, flakiness may arise). That integration test is twenty times more valuable, but it is equally more likely to break for reasons tangential to its core assertions.
Oh, yeah I have experience with super flaky integration/UI tests too. I think a couple rules mostly keep things from getting out of hand in unit tests (never import `random`, use `freezegun`, etc.), but in my experience even this fails to corral the flakiness of 1000s of unit tests. I'm hopeful that property testing frameworks make a dent here, but I haven't had enough experience with them yet.
> An integration test might rely on four database tables being just-so, and go on to connect with two external services (and whether mocked, replayed, or live, flakiness may arise).
I actually feel most comfortable when my integration tests are essentially just API calls--or you could think of them as unit tests of the API. That way, if it's flaky in CI, it's flaky in prod too, and you know to fix it.
This is where mocks generally lead you astray, either they act like everything is fine, or they have some randomness/etc. built in and it causes flakiness. Any time you're testing a mock and not "real" code is a huge failure IMO; it debases the entire scientific process.
---
I guess I would summarize my testing position as "test all the API calls you support, pretty exhaustively". That's your spec. If you need some unit tests while you're developing something, definitely add them, but once you get things up to spec, just toss 'em. Otherwise you're binding future engineers to your implementation, which generally isn't helpful.
>They also tend to slow down CI because they're slow and always flaky.
In my experience, unit tests are the most stable, the least flaky, because they touch the least code and often have very simple setup. An integration test might rely on four database tables being just-so, and go on to connect with two external services (and whether mocked, replayed, or live, flakiness may arise). That integration test is twenty times more valuable, but it is equally more likely to break for reasons tangential to its core assertions.