The code in the article is about optimisation, in which case there should always be a slow reference implementation. Never try to write highly optimised code on the first go: make it work, make it correct, then make it fast.
Also, these aren't unit tests; they're property tests.
> The code in the article is about optimization [...]
Very true but that really should have been stated up-front by the author. Next thing you know "some guy at Valve said all unit tests with floats should test all 4 billion cases." You would be surprised at how much people take away from a skim over something - e.g. a poor understanding of Apps Hungarian created the monster that is Systems Hungarian.
> in which case there should always be a slow reference implementation.
Good point - however, keep in mind that bug fixes would need to be made in two places. Not only does it mean the maintainer needs to grok two pieces of code, but they may have to determine if the original code is providing bad truths. Hopefully there wouldn't be many places where this type of testing would be required as the low hanging fruit is often enough: however keep in mind certain industries do optimize to ridiculous degrees - and in some of those industries optimization can happen first depending on the developers [well-seasoned] intuition, and can be a frequent habit. The author's industry is a brilliant example of where the typical order of optimization (make it work, make it correct, make it fast) takes a back seat - life is incredibly different at 60/120Hz.
> Good point - however, keep in mind that bug fixes would need to be made in two places.
Only if the bug actually occurs in both implementations. This does happen sometimes, so it's indeed good to keep in mind - but my experience experimenting with these kinds of comparison tests hasn't borne this out as the more common case. YMMV, of course.
> Not only does it mean the maintainer needs to grok two pieces of code, but they may have to determine if the original code is providing bad truths.
I find automated comparison testing of multiple implementations to be extremely helpful in grokking, documenting, and writing more explicit tests of corner cases by way of discovering the ones I've forgotten, frequently making this easier as a whole, even with the doubled function count.
> Hopefully there wouldn't be many places where this type of testing would be required as the low hanging fruit is often enough: however keep in mind certain industries do optimize to ridiculous degrees - and in some of those industries optimization can happen first depending on the developers [well-seasoned] intuition, and can be a frequent habit.
I don't trust anyone who relies on intuition in lieu of constantly profiling and measuring in this sea of constantly changing hardware design - I've yet to see those who've honed worthwhile intuitions drop the habit :). Even in the lower hanging fruit baskets, it's frequently less a matter of optimizing the obvious and more a matter of finding where someone did something silly with O(n^scary)... needles in the haystack and where you'd least expect it.
That said, there are times when making it correct just doesn't matter. Sacrificing accuracy for speed can be an extremely powerful technique, as long as perfect accuracy isn't required (e.g. some graphics applications).
In some cases, yes. In others, you can only manage that on a micro-level.
An example that comes to mind: at some point I was working on an optimized paintbrush tool that was to give perceptually similar results to drawing the brush image repeatedly along a path. I replaced the individual draws with convolution, but there was no way to make this exactly correspond with the "over" compositing operator that it was replacing. So I used a post-processing step to adjust it.
In this case, the divergence from the reference implementation was so great that you couldn't really compare them. But the results looked good, and the performance gains were tremendous. The only way to compare it would have been to create a new reference implementation that imitated the new algorithm, which would have told me very little. Instead, I did lots and lots of real world testing to ensure there were no missed edge cases.
Also, these aren't unit tests; they're property tests.