My favourite part was the use of (graphite-like?) counters to monitor changes and make implicit assertions about relationships in the system (ie if we push that metric down, that metric will go up by same amount)

It's a really useful trick to stop yourself believing that the systems works the way you think it does just because you think it.