Unfortunately not, GitHub tokens are (outside of GitHub actions generated tokens) scoped for your entire account, which sucks. It should be an advantage though.
I had the same reaction then I re-read the comment and realized it was correct.
The granularity of Personal access tokens scopes is focused on what kind of actions you can perform on which kind of objects, but you cannot limit it to one single repository or to one single organization you belong.
