Yes, it sets the reviewer's expectations around how much effort was spent reviewing the code before it was sent.
I regularly have tool-generated commits. I send them out with a reference to the tool, what the process is, how much it's been reviewed and what the expectation is of the reviewer.
Otherwise, they all assume "human authored" and "human sponsored". Reviewers will then send comments (instead of proposing the fix themselves). When you're wrangling several hundred changes, that becomes unworkable.
Phone companies are required to make sure 911 works on their phones. Random people on the internet aren't required to make sure 911 works on random apps, even if they look like phones.
Documentation rots a lot more quickly than the code - it doesn't need to be correct for the code to work. You are usually better off ignoring the comments (even more so the design document) and going straight to the code.
I maintain you’re either grossly misappropriating the time and energy of new and junior devs if this is the case on your project, or you have gone too long since hiring a new dev and your project is stagnating because of it.
New eyes don’t have the curse of knowledge. They don’t filter out the bullshit bits. And one of the advantages of creating reusable modules is you get more new eyes on your code regularly.
This may also be a place where AI can help. Some of the review tools are already calling us out on making the code not match the documentation.
No, they're 100% correct. This has been my experience at every place I've worked at in SV, from startup to FAANG.
You write the code so you can scan it easily, and you build tools to help, and you ask for help when you need it, but you still gotta build that mental map out
The last time I tried AI, I tested it with a stopwatch.
The group used feature flags...
if (a) {
// new code
} else {
// old code
}
void testOff() {
disableFlag(a);
// test it still works
}
void testOn() {
enableFlag(a);
// test it still works
}
However, as with any cleanup, it doesn't happen. We have thousands of these things lying around taking up space. I thought "I can give this to the AI, it won't get bored or complain."
I can do one flag in ~3minutes. Code edit, pr prepped and sent.
The AI can do one in 10mins, but I couldn't look away. It kept trying to use find/grep to search through a huge repo to find symbols (instead of the MCP service).
Then it ignored instructions and didn't clean up one or the other test, left unused fields or parameters and generally made a mess.
Finally, I needed to review and fix the results, taking another 3-5 minutes, with no guarantee that it compiled.
At that point, a task that takes me 3 minutes has taken me 15.
Sure, it made code changes, and felt "cool", but it cost the company 5x the cost of not using the AI (before considering the token cost).
Even worse, the CI/CD system couldn't keep up the my individual velocity of cleaning these up, using an automated tool? Yeah, not going to be pleasant.
However, I need to try again, everyone's saying there was a step change in December.
I did my own experiment with Claude Code vs Cursor tab completion. The task was to convert an Excel file to a structured format. Nothing fancy at all.
Claude Code took 4 hours, with multiple prompts. At the end, it started to break the previous fixes in favor of new features. The code was spaghetti. There was no way I could fix it myself or steer Claude Code into fixing it the right way. Either it was a dead-end or a dice roll with every prompt.
Then I implemented my own version with Cursor tab completion. It took the same amount of time, 4 hours. The code had a clear object-oriented architecture, with a structure for evolution. Adding a new feature didn't require any prompts at all.
As a result, Claude Code was worse in terms of productivity: the same amount of time, worse quality output, no possibility of (or at best very high cost of) code evolution.
Are you able to share your prompts to Claude Code? I assume not, they are probably not saved - but this genuinely surprised me, it seems like exactly the type of task an LLM would excel at (no pun intended!). What model were you using OOI?
The exact same prompt ? Everything depends on the prompt and it’s different tools. These days the quality and what’s build around the prompt matters as much as the code. We can’t feed generic query.
Similar happened to me just now. Claude whatever-is-the-latest-and-greatest, in Claude Code. I also tried out Windsurf's Arena Mode, with the same failure. To intercept the inevitable "holding it wrong" comments, we have all the AGENTS.md and RULES.md files and all the other snake oil you're told to include in the project. It has full context of the code, and even the ticket. It has very clear instructions on what to do (the kind of instructions I would trust an unpaid intern with, yet alone a tool marketed as the next coming of Cyber Jesus that we're paying for), in a chat with minimal context used up already. I manually review every command it runs, because I don't trust it running shell scripts unsupervised.
I wanted it to finish up some tests that I had already prefilled, basically all the AI had to do was convert my comments into the final assertions. A few minutes later of looping, I see it finishes and all tests are green.
A third of the tests were still unfilled, I guess left as an exercise for the reader. Another third was modified beyond what I told it to do, including hardcoding some things which made the test quite literally useless and the last third was fine, but because of all the miscellaneous changes it made I had to double check those anyways. This is about the bare minimum where I would expect these things to do good work, a simple take comment -> spit out the `assert()` block.
I ended up wasting more time arguing with it than if I had just done the menial task of filling out the tests myself. It sure did generate a shit ton of code though, and ran in an impressive looking loop for 5-10 minutes! And sure, the majority of the test cases were either not implemented or hardcoded so that they wouldn't actually catch a breakage, but it was all green!!
That's ultimately where this hype is leading us. It's a genuinely useful tool in some circumstances, but we've collectively lost the plot because untold billions have poured into these systems and we now have clueless managers and executives seeing "tests green -> code good" and making decisions based on that.
What model, what harness and about how long was your prompt to fire off this piece of work? All three matters a lot, but importantly missing from your experience.
The severity of the DoS depends on the system being attacked, and how it is configured to behave on failure.
If the system is configured to "fail open", and it's something validating access (say anti-fraud), then the DoS becomes a fraud hole and profitable to exploit. Once discovered, this runs away _really_ quickly.
Treating DoS as affecting availability converts the issue into a "do I want to spend $X from a shakedown, or $Y to avoid being shaken down in the first place?"
Then, "what happens when people find out I pay out on shakedowns?"
If the system "fails open" then it's not a DoS, it's a privilege escalation. What you're describing here is just a matter of threat modeling, which is up to you to perform and not a matter for CVEs. CVEs are local properties, and DoS does not deserve to be a local property that we issue CVEs for.
There is _always_ fraud, and you can't stop it all. All you can do is try to minimize the cost of the fraud.
There is an "acceptable" fraud rate from a payment processor. This explains why there are different rates for "card present" and "card not present" transactions, and why things like Apple Pay and Google Pay are popular with merchants.
You are really running with scissors there. If anyone with less scrupulous morals notices, you’re an outage away from being in deep, deep shit.
The best case is having your credit card processing fees like quadruple, and the worst case is being in a regulated industry and having to explain to regulators why you knowingly allowed a ton of transactions with 0 due diligence.
> Treating DoS as affecting availability converts the issue into a "do I want to spend $X from a shakedown, or $Y to avoid being shaken down in the first place?"
But that is what security is in the real world anyway. Once you move past the imaginary realms of crypto and secure coding that some engineers daydream in, the ultimate reality is always about "do I want to spend $X dealing with consequences of ${specific kind of atack}, or $Y on trying to prevent it" - and the answer is to consider how much $X is likely to be, and how much it'll be reduced by spending $Y, and only spending while the $Y < reduction in $X.
> Treating DoS as affecting availability converts the issue into a "do I want to spend $X from a shakedown, or $Y to avoid being shaken down in the first place?"
> Then, "what happens when people find out I pay out on shakedowns?"
What do you mean? You pay to someone else than who did the DoS. You pay your way out of a DoS by throwing more resources at the problem, both in raw capacity and in network blocking capabilities. So how is that incentivising the attacker? Or did you mean some literal blackmailing??
If you're submitting a CVE for a primitive that seems likely to be useful for further exploitation, mark it as such. That's not the case for ReDOS or the vast majority of DoS, it's already largely the case that you'd mark something as "privesc" or "rce" if you believe it provides that capability without necessarily having a full, reliable exploit.
Don't you need something like a network clock to get deterministic replay?
It can't use immediate return on replay, or else the order will change.
This makes me twitchy. The dependencies should be better modelled, and idempotency used instead of logging and caching.
reply