There's an interesting interview with Sam Altman here where he acknowledges the model necessarily needs to understand and define off-limit topics in order to be told NOT to engage in those topics.

https://www.youtube.com/watch?v=WHoWGNQRXb0