I am about 1 hour into using it with pi.dev. Do you have thinking on high? It is doing good but at one point i had to stop it and say 'you're overthinking this' haha
Yes full send mode on thinking. I have moved on from watching my agents and I don't really care how it thinks. I look at the end result and so far this thing has been blowing me away. No way this is as good as it is this small and fast. Outside Fable, this might be the best thing I've ever used.
"What we've done in this model is not necessarily add more intelligence, but improve the behaviors that lead to a more capable model: more verification, less taking things for granted, not declaring victory early, and being more persistent.”
Just by the (lack of) inter-model variance, I don't think SWEBench-Pro does a very good job of representing model capability. Terminal-Bench seems more challenging and separates the wheat from the chaff.
Also, *ops work, which in my experience can actually be more complicated than SWE is underrepresented there obviously.
