I think the idea is to use git submodules for vendor libs. Anything you're not modifying and a 3rd party is maintaining should be a submodule; that way you don't keep any of those changes in the repo.
When a new vendor release comes out, we build it with our "official" compiler settings for the different platforms, branch the headers from the source and combine them into a convenient "SDK" tree, sometimes tweak something here or there, and update the document. Amounts to several GB being checked in from multiple machines. Occasionally a developer (usually me, but others too) needs to commit changes back to our central repo vendor tree.
I suppose we could do that into a separate repository and then define parts of that as a git submodule.