I think he meant "I must" in the incredulous sense, like it must happen to him that frequently as it is driving him nuts.