I’m pretty sure Anthropic’s Claude is doing that.

https://scale.com/blog/chatgpt-vs-claude

Maybe an adversarial approach was used in training these models in the first place?

It was they were' trained using reinforcement learning with human feedback to create the critic.

I hadn't thought about human feedback being an adversarial system, but I guess that makes sense, since it's basically a classifier saying "you got this wrong".