This is a really good insight - the trope of the "frustrated scholar-official" is extremely prominent in the Chinese literary tradition. That being said, the ideal of "becom[ing] hermits and enjoy[ing] their life happily after" was something of a literary conceit; in most of the actual literature written by people who tried to do this, there's an extremely strong tension between the idealized/romanticized apolitical world of the hermit, and the reality that farming was very hard labor and not something most literati particularly enjoyed. Probably the most famous example of this is the poetry of Tao Yuanming. (Note: I wrote my Master's thesis on depictions of eremitism in ancient/early medieval China.)