That's still a form of tracking. Maybe not enough to identify unique users in some use cases, but even just knowing someone has been here n times is enough if the user numbers are low enough that you can identify users by unique n counts and patterns of n (such as if one user is at 500 and another is at 490, if the second one is logging in daily while the first one hasn't logged in for a few months, and you see the 490 go 491, 492... when they go from 499 to 500, the chance when a 500 logs on tomorrow and becomes 501 it was the 490 account that has been logging in daily).
Must admit, I've never thought of "number of times I've visited your site" as PII. Number of times I've visited every site in my browser history, maybe, but not "number of times I've visited this specific site". I'm thinking about it, but I'm not immediately convinced.
That's because you're forgetting the temporal domain. As in GP's example, a count alone may not mean much, but a time series of counts will allow you to uniquely identify a subset of the users.
