I'm currently studying traffic with using reinforcement learning with:

- Microscopic simulator: https://sumo.dlr.de/index.html

- Flow (abstraction of RLlib and Sumo): https://flow-project.github.io/

Have a look at Sumo, its free, open source, and comprehensive. Also Aimsun [1] but I've never used it and don't think it's free although I'm unsure about that.

[1] https://www.aimsun.com/aimsun-next/