I agree with your points. We have adopted a resource-based RBAC/ABAC and a policy language in simple YAML to address some of these challenges. In terms of performance, a side car architecture with SDKs give good response times. In such simple architecture the implementation of lists is the current challenge we are addressing. However, for majority of use cases abstracting the decision making logic to a centralized service and providing a simple API that addresses the question of can this principal, do this action on this resource with a true or false answer goes a long way.