A word of caution: I have tried the new tokens described below and they caused some vague issues (for example, plugins would sometimes no longer update). I will update this once I know more.
The fine-grained personal access tokens (currently in beta) from GitHub appear to solve the access control issues of the current system. The new tokens can be scoped to only a subset of repositories and be limited to read-only access of the Git repository and none of the other GitHub extras.