Mercor, Tacit Labs, Handshake AI...
I suspect companies like these play a big part in model improvements, generating high quality benchmark/task-focused data for training.
However, these do require educated, white collar, workers.
I'm curious about this because I've had Fable decompile games and help me understand what's going on inside the game itself and it never complained. I'm not sure what it takes to trip the "safety" guards but digging into game code and data files doesn't seem to be a barrier at all. I've used CC to build some personal game mods a few times now. Once for a game with no modding capability explicitly exposed.
Oddly not much to trip it. I once ran “touch AKAMAI.md” and then had Claude run git status and it fell back to Opus as if I had just broken into the pentagon itself.
> A narrow set of frontier LLM development tasks, such as distributed training infrastructure, ML accelerator design, and kernel development for certain non-standard chips.
My work is mostly on the Nvidia b200, which apparently gets flagged as non-standard.
Opus 5 works, but sometimes I do wonder if it's surreptitiously trying to sabotage the efforts -- possibly deliberately, but more likely by something like Fable's initial launch, which did come with secretly degraded performance when detecting kernel work. Anthropic was open at the time that such a mechanism existed, but disabled it due to backlash. More likely than not, this is just paranoia on my end...
I think I'd rather have the model stop once the obvious solutions failed and ask me. It can suggest more creative ideas, but I don't necessarily want it to try implementing them.
A few years ago, on July 4th one of my mom's dogs freaked out, somehow managed to escape, and got hit by a car before my mom found her. She loved that dog, regularly attending nose work competitions with it.
One of your pets getting a seizure must be harrowing for both you and the dog.
My female Weimaraner was terrified of thunderstorms and fireworks. Once, she managed to squeeze herself under my Mini Cooper. I didn't know at the time that panic vests were a thing. The breed is a gun dog ¯\_(ツ)_/¯. I forget how we managed to get her out from under there, but it took a bit. My Akita doesn't care, he checks the balcony, yawns, and back to his toys. But he is a afraid of air balloons and blow up Christmas Santas. Go figure.
When claude goes down a wrong path, I tend to clear context and write a new prompt that helps guide it down the correct path.
Whatever thinking or context that led it there has inertia and tends to be sticky, otherwise.
Pretty annoying when it brings those up again later from memory...
Exactly. How can "we" develop and encourage benchmarks for multi-turn user assistance?
That is what I want.
I feel like the models and harnesses push much too hard against this workflow -- that they push you towards letting go and vibe coding, with only your discipline (and desire for a quality and maintainable product) holding it back.
As a baseline, I run all models in Q8 [0] because I want to be confident that when I observe a problem, the root cause is not due to the quantization. However, in this specific case, I use Q8 on the mac and Q4 on the RTX machine because the latter does not fit the full context at Q8. So far, I don't have conclusive evidence that the Q4 quantization affects the quality in a significant way for this model and the tasks that I am using it for.
27B seems surprisingly resiliant to quantisation. Though my evals showed there was some impact to coding ability from 8 bit to 4 bit, it was less than I would've expected: and it was on task types that you've said above that you don't really do with these!
However, these do require educated, white collar, workers.
reply