Hacker Newsnew | past | comments | ask | show | jobs | submit | d2p's commentslogin

Odd that the tool call failure rate is so high (5%) for the OpenAI provider than Azure (0.2-0.5%).

> There is a limit of 450,000 tokens per minute. I hit this limit in about 90 seconds

I'm confused. If it's 1500t/s, isn't that only 90k per minute? How do you hit a 450k/minute limit?


Cached tokens count towards the limit as well. For example, if your context window is 50,000 tokens, it takes 9 requests to reach that limit without generating a single token.

Cached tokens counting toward the limit is ridiculous.

then it's basically useless lol, wtf, this has to be a defect


> and use your traffic to (opt in) train you a model.

Is there more info on this? I'm curious exactly what it is. Is it fine-tuning/LoRA on some base model? Don't cloud providers encrypt reasoning now - does that prevent this?


Does this support Linux yet? When I previously looked it did not (the reason being that they were already using VMs on Windows/macOS but not on Linux). Every time I see an announcement I think "great, they must've added Linux now then", but the linked pages always have Windows + macOS instructions but not Linux.

All the open GH issues about supporting Linux that I subscribed to have gone unresponded to.

OpenShell looks like a good alternative, but it still has "Do not use in production" plastered all over the website, which doesn't fill me with confidence yet


It must be a joke that this tool is not supported on linux yet, although docker is built on top of linux containers. Shame on docker.


It is supported on Linux…

https://docs.docker.com/ai/sandboxes/#get-started has instructions for Ubuntu.


I think Linux has a better solution than Docker.

I wrote a tool to use `bubblewrap` to containerize any agent (at least all the agents I've used a couple of times), and bind mount the system stuff read-only, so the agent has your "usual" environment, but they can only see the project. Their history persists (either through a bind mount or a "shadow" copy of the history that only the wrapped agent sees), the agent can still create and manage containers of its own using podman's rootless mode, etc. It's nearly instant to start because it's just a namespace (plus a few copied files for the container support and session history); no container needs to be built/fetched/updated/whatever. bubblewrap is extremely well-tested as it is used by flatpak and several other large projects, so I trust it quite a bit (more than I trust Docker).

https://github.com/swelljoe/flar


bubblewrap may work well for you and your specific workflows/projects but not in an enterprise setting where everyone already has a different setup on the host and needs something different inside the container. It's impossible to deploy a solution like that with bubblewrap -- configuration itself is going to be a nightmare. Which is why Docker Sandbox is aimed at teams/enterprises.


Yeah, Podman would be a better basis for that kind of use case. I'd built an early implementation of `flar` with Podman first, but it was more annoying than simply having my regular dev environment instantly available in the container. But if you need a bunch of different dev environments, instead of just your usual one, then sure, a bunch of different custom containers makes sense.

But, Docker is rarely the right way to manage containers on Linux, IMHO.


"Docker Sandbox" is not docker. Completely different (and almost unrelated) products.


Bubblewrap is not nearly as secure as a proper VM.


I was able to install it on Fedora from the .rpm distributed on their GitHub releases: https://github.com/docker/sbx-releases/releases


I also hit the same issue recently. No Linux and no Windows on arm. AI sandboxing has a lot of options but none feel complete just yet. It's hard to commit to something, especially if reviewing tools to aide in company policies.

Regardless, I'm hoping something that isn't behind a login screen is going to win out.


I'm fairly certain that docker sandbox is based on https://github.com/containerd/nerdbox which you can run on Linux.


I work on OpenShell and we definitely see folks using it for production use cases. We're updating the language on our docs to reflect that. I'd recommend giving it a shot!


I have - and it's probably my preferred of all the sandboxes I've tried out. But when it comes with this disclaimer about being Alpha and not to use in production, it makes it seem like you don't think it's ready yet :)

Btw - any plans for a web app to manage network approvals etc.? It feels so archiac monitoring in the console (honestly, I don't know what the obsession is with terminal UIs lately... Yes it's great that they work via SSH, but if I can SSH to a box I can also probably connect a browser to it )


I clicked through and it showed Qwen at the top at 55.4 compared to 55.3 for Opus Max. I have a screenshot.

Then I clicked away and back, and now it goes Qwen second, with 58.4, to Opus Max at top with 59.2.

I have screenshots of both. The description above the chart is the same in boh cases:

> Artificial Analysis Agentic Index > Represents the weighted average of agentic capabilities benchmarks in the Artificial Analysis Intelligence Index (GDPval-AA v2, ³-Banking)

What happened? How can the scores change so much in a few seconds?



> HLE, AA-LCR and AA-Omniscience are now graded by GPT-5.6 Luna (medium), replacing GPT-4o, Qwen3 235B A22B 2507, and Gemini 3 Flash Preview respectively. These checks are now unified under a more capable modern model, selected for strong agreement with human judgment in our grader validation

Interesting that they chose a nano-sized model from OpenAI to be a grader for benchmarks involving knowledge and hallucination.


What's interesting is that if you ask 5.6 Sol or Opus 5 they will tell you it's a bad idea to have the reviewer be the dumber of the set as it can't judge them properly to decide who is right, and thus if one is better because it found an answer that's better but contradict the obvious it would be biased against. I know because I just had a consensus conversation with them this afternoon about a design that was similar (though about something completly different than judging agentic quality or whatever).


Fixed the result, eh? In both senses of the word.


Can someone please explain what changed, when it happened, and whether it was surreptitious?


I have been suspicious of these AI leaderboard sites for some time now, and this only increases that suspicion.


In that case they should clearly label that this is a new benchmark.


What was the change?


Hey! George from the Artificial Analysis team here. We published an update today that does result in a change of the order, Qwen3.8 Max to second rather than first. The methodology change was an already planned upgrade to our equality checking/grader models, and brings the latest ³-Banking version to Artificial Analysis. Regular updates are normal for us to keep our benchmarks up to date.

The order changes but I think the story discussed in this thread holds - this is a very impressive release and Qwen3.8 Max is a huge step up in agentic capabilities.

Relevant blog post (also linked to by others): https://artificialanalysis.ai/articles/artificial-analysis-i...


You gotta admit the timing looks very suspicious.


Luna pricing was just cut by 80% https://www.eesel.ai/blog/gpt-5-6-pricing and as the blog post states is a more accurate judge than the previous methodology.


> You gotta admit the timing looks very suspicious.

Do you mean the timing looks like: "We're SV tech-bros. Our benchmarks showed a chinese model above what's considered the best model at the moment. So we quickly modified the benchmark so that our SV tech-bros don't look like they're losing to a chinese model"?

That's indeed a bit fishy.


They could just have avoided all of this by not publishing the benchmark until the new methodology update.


They should probably freeze the results before publishing.


Welp. That didn't last long


Same, they just updated it. Hacker news effect?


Looks like the default chat template was updated on HF to enable this by default shortly after you posted this :-)


> A flag called DO_NOT_TRACK sounds like a good idea, but also suggests the default is CONSENT_TO_TRACK=1, and I find that creepy.

It could also be used to prevent showing an opt-in notification at all even in software that requires opt-in.


It is a bit sad that in privacy enthusiastic consent is understood as failing to shout 'NO' in the right way.


> At best, they're effective at presenting people who don't know they're sick from infecting others.

I don't have numbers, but I'm pretty sure the number of people infected by people that didn't know they were sick is significant. Imagine if all those sick people wore masks (or some substitute, like a scarf over their mouth/nose).

If we can agree that wearing a mask (or substitite) if you might be sick is a good idea, but you don't know if you might be sick, that seems to translate to wearing a mask (or substitute) is a good idea.

> that only works if you can mask a significant portion of the population without a shortage of masks for people who we know need them.

I agree it's important to reserve masks for those who need them most, but that doesn't mean we have to teach people they are not effective. We could instead teach them how "non-masks" can make very reasonable substitutes. And let's not forget that reducing the spread has a real impact on the need for healthcase workers to have them.


> I don't have numbers, but I'm pretty sure the number of people infected by people that didn't know they were sick is significant.

Look at the number of asymptomatic or pre-symptomatic cases (approx 50%) there are in the two full population studies on the Princess Cruise ship, and Vò (in Italy).


> That's not true.

How so? Everything I've seen from them, including the Gov't site only say to self-isolate. They said that in the near future they may tell households to isolate, but right now only the person with symptoms.

> "people with confirmed or possible coronavirus (COVID-19) infection"

https://www.gov.uk/government/publications/covid-19-stay-at-...


> browser extension

Using browser extensions to block ads is much higher risk than doing DNS blocking. Most ad blockers have full access to all web pages, which essentially means they could trivially scrape your usernames/passwords for your email/banks/etc or perform actions on your behalf.

There's room for this to go bad (AdBlocker dev turns bad, or sells extension to a bad guy for a wad of cash, or extension has security vulnerabilities, or keys for publishing extension are not propery secured) so while DNS-level blocking might not work as well, it's definitely not an obviously-worse solution.

(though FIWI PiHole in the past had some really agressive default lists which stopped my from using it - though I set it up again recently and it's been much better - I haven't had any broken websites besides Amazon's own sponsored product links at the top of their own search results pages).


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: