I've been seeing a lot more "ChatGPT only produces junk code" comments (case in point, this entire thread), and that hasn't really been the case either.
For majority of those benchmarks it produced the right code from the first go. It struggled more with Java VirtualThreads - probably fewer programs in the training set. It also had a tendency to overcomplicate things (adding unncecessary code). So there were a few iterations needed plus some hand edits.
