I actually tried this workflow with some ebay'ed microcassette recorders - even the really compact ones are bulkier (though not necessarily heavier) than a modern cellphone, plus being non-connected you basically have to set aside some time to transcribe and erase (doable, but if you're specifically using it as a memory assist then having to remember this isn't great.) They did do a good job of "one press and they're operating, and you can confidently believe it" (but you have to pick them up in the right orientation for that to work.)
A modern digital voice recorder would have been a better choice for the experiment perhaps? Pen form factor versions exist, e.g. Olympus VP-20, Philips Voicetracer DVT1600.
Of the few of those that have "one button recording" at all, they generally still have slide-and-hold or otherwise complicated mechanisms to power on the device before the one-button part works, or they have a slide-and-hold record switch that doesn't give good feedback that they're actually recording. (The VP-20 is like that.) Hmm, the voicetracer looks plausible, you just have to push the switch one way. (Most of the pen-form ones have the secondary flaw that they're trying to be stealthy, which is illegal in ⅔ of US states.)
The one consumer recorder that I've found that's really "grab it, press one button, start talking, and know you're recording" is the teenage engineering TP-7 - which is an amazing bit of engineering, but for $1500 it had better be.
