Because, one way or another, we will need to do that for LLMs to be useful. Whether the facts are in the training data or the context knowledge (RAG provided), is irrelevant. And besides, we are supposed to trust that these things have "world knowledge" and "emergent capabilities", precisely because their training data contain, well, facts.
All privatizations of public infrastructur and services are failures, simply due to the discrepancy between the interests:
The public wants: Cheap, reliable, quality services.
A corporation wants: Maximized Revenue.
The two inevitably clash. No matter what business, no matter what country. Don't believe me? Go and find a single instance of a service that used to be provided completely by the government, that got BETTER FOR USERS (that is: The Public, not investors) after being privatized. I'll wait.
Public utilitis HAVE TO be run by the public, meaning the government. In pretty much every instance where this isn't the case, the provided service is more expensive, and/or less effective for the people.
And that's why I am sick and tired of the old trope "bUt pRivAtE sEcToR mOrE effIciEnt!" Sure. That would be the same private sector that caused almost every non-war related major economic crisis, is it not? I think the "efficiency argument" is already a moot point.
Why would public ownership deliver those benefits? Walmart is delivering cheap, quality and reliable service in retail. Soviet stores we not better than American ones.
>Don't believe me? Go and find a single instance of a service that used to be provided completely by the government, that got BETTER FOR USERS (that is: The Public, not investors) after being privatized. I'll wait.
You won't have to wait long: flying. Shocked? You shouldn't be. In the 70s, you'd have to pay thousands of dollars to fly across the country. Now you can get a flight between NYC and London for $200.
Competition dynamics for air traffic are different. Railway infrastructure can only support a very limited number of competing companies before you run out of capacity, and especially on classic mixed traffic routes it is very easy to run out of capacity. Once that happens, rail operators have to start competing for paths instead of directly for passenger, which can definitely lead to misalignment of incentives.
On top of that, there's the problem that it's physically impossible to run competing services at exactly the same time. This means that for customers with schedule constraints (i.e. you need to arrive somewhere specific by a certain time at latest), competition becomes much less effective, because you're no longer able to freely choose among the competing operators, but are instead forced to simply take the train that arrives at the right time.
A similar thing is for journeys that involve changes of trains – it's physically impossible to have attractive connections (i.e. without hanging around the station for ages) between more than one or at most (if that) two operators per route, because trains running along the same line always have to be separated by at least two or three minutes.
With air traffic it's different. While airport and air space capacity isn't quite unlimited, it's still not as limited in the way a mixed traffic railway is. Plus a much higher proportion of air traffic is holiday traffic and other long-distance journeys where even a few flights per day would be considered frequent service, so competition is much less limited by that fact.
Air travel wasn't provided by the government before deregulation. The airlines have always been private companies. They were heavily regulated and run as public utility. Deregulation increased competition but nothing was privatized, the CAB was eliminated.
Air travel is also not privatized these days. The airports are nearly all owned by government entities.
It basically provides a bunch of graphs that confirms that flying is safer than ever, even as more people than ever fly at cheaper fares than ever. Embedded within the actual data is an unsubstantiated rant about enshittification and the danger of flying (despite the evidence they chose indicating the opposite), and a tangent about github.
This is only "true" because anything run efficiently by the private sector is so demonstrably better than any communist-fantasy alternative that no one bothers arguing any more.
Should Tesco be run by the government? I use Tesco more than I take trains.
May be of note also that the best train in London (the Elizabeth line) is run by a tendering process
> May be of note also that the best train in London (the Elizabeth line) is run by a tendering process
But that tendering is just internal between the state and the operator – fares and service levels on the other hand are set by TfL (and probably also the DfT to some extent, especially on the GEML and GWML sections).
Yeah, sorry, but if the user friendly alternative to the VT100 emulators is anything like Jupyter, then I'll happily keep using alacritty, konsole and xterm, thanks.
> It's absolutely insane to me how bad the user experience is with video nowadays
Has nothing to do with video per se. Normal embeddings, using the standard `<video>` element and no unnecessary JS nonsense, still work the same way they did in the 90s: Right click the video and download it, it's a media element like any other.
The reason why user experience is going to shite, is because turbocapitalism went to work on what was once The Internet, and is trying to turn it into a paywalled profit-machine.
I've always found it insane how much software development web sites are willing to undertake, just to avoid using the standard video, audio, and img HTML elements. It's almost hilarious how over engineered everything is, just so they can 'protect' things they are ultimately publishing on the open web.
Plain <video> elements are easy to download, but not great for streaming, which is what most people are doing nowadays. Much of the JS complexity that gets layered on top is to facilitate adaptive bitrate selection and efficient seeking, and the former is especially important for users on crappier internet connections.
I'm not a fan of how much JS is required to make all that work though, especially given the vast majority of sites are just using one of two standards, HLS or DASH. Ideally the browsers would have those standards built-in so plain <video> elements can handle them (I think Safari is the only one which does that, and they only do HLS).
I totally agree. And much of the JS complexity on smaller niche video sites aren’t even implemented properly. On some sites I just open developer console, find the m3u8 file URL and cookies in the request, and download it to view locally.
Browsers generally do allow native seeking if the video is properly encoded and the site supports such niceties as Accept-Range: bytes.
> still work the same way they did in the 90s: Right click the video and download it, it's a media element like any other.
I’m so confused reading these comments. Did everyone forget RealPlayer? Flash videos? All of the other nonsense we had to deal with to watch video on the internet?
RealPlayer was 1995, so a few years later, and arguably was a start of the trend of enshittification. Flash videos was around the times things really got bad.
Technically, you can profit off of ad revenue and subscriptions without exploiting the labour of your workers, so in this particular case it has nothing to do with the economic regime. Enshittification is its own thing.
The problem with a standard video element is that while it's mostly nice for the user, it tends to be pretty bad for the server operator. There's a ton of problems with browser video, beginning pretty much entirely with "what's the codec you're using". It sounds easy, but the unfortunate reality is that there's a billion different video codecs (and a heavy use of Hyrum's law/spec abuse on the codecs) and a browser only supports a tiny subset of them. Hosting video already at a basis requires transcoding the video to a different storage format; unlike a normal video file you can't just feed it to VLC and get playback, you're dealing with the terrible browser ecosystem.
Then once you've found a codec, the other problem immediately rears its head: video compression is pretty bad if you want to use a widely supported codec, even if for no other reason than the fact that people use non-mainstream browsers that can be years out of date. So you are now dealing with massive amounts of storage space and bandwidth that are effectively being eaten up by duplicated files, and that isn't cheap either. To give an estimate, under most VPS providers that aren't hyperscalers, a plain text document can be served to a couple million users without having to think about your bandwidth fees. Images are bigger, but not by enough to worry about it. 20 minutes of 1080p video is about 500mb under a well made codec that doesn't mangle the video beyond belief. That video is going to reach at most 40000 people before you burn through 20 terabytes of bandwidth (the Hetzner default amount) and in reality, probably less because some people might rewatch the thing. Hosting video is the point where your bandwidth bill will overtake your storage bill.
And that's before we get into other expected niceties like scrolling through a video while it's playing. Modern video players (the "JS nonsense" ones) can both buffer a video and jump to any point in the video, even if it's outside the buffer. That's not a guarantee with the HTML video element; your browser is probably just going to keep quietly downloading the file while you're watching it (eating into server operator cost) and scrolling ahead in the video will just freeze the output until it's done downloading up until that point.
It's easy to claim hosting video is simple, when in practice it's probably the single worst thing on the internet (well that and running your own mailserver, but that's not only because of technical difficulties). Part of YouTube being bad is just hyper capitalism, sure, but the more complicated techniques like HLS/DASH pretty much entirely exist because hosting video is so expensive and "preventing your bandwidth bill from exploding" is really important. That's also why there's no real competition to YouTube; the metrics of hosting video only make sense if you have a Google amount of money and datacenters to throw at the problem, or don't care about your finances in the first place.
Chrome desktop has just landed enabled by default native HLS support for the video element within the last month. (There may be a few issues still to be worked out, and I don't know what the rollout status is, but certainly by year end it will just work). Presumably most downstream chromium derivatives will pick this support up soon.
My understanding is that Chrome for Android has supported it for some time by way of delegating to android's native media support which included HLS.
Desktop and mobile Safari has had it enabled for a long time, and thus so has Chrome for iOS.
Any serious video distribution system would not use metered bandwidth. You're not using a VPS provider. You are colocating some servers in a datacenter and buying an unmetered 10 gigabit or 100 gigabit IP transit service.
Running in tmux, marking anything on my terminal immediately puts it into the tmux buffer, without me having to click anything on the keyboard. Pressing middle-mouse pastes it.
> I don't think there's evidence that this issue would persist after continuing to scale models to be larger and doing more RL
And how much larger do we need to make the models? 2x? 3x? 10x? 100x? How large do they need to get before scaling-up somehow solves everything?
Because: 2x larger, means 2x more memory and compute required. Double the cost or half the capacity. Would people still pay for this tech if it doubles in price? Bear in mind, much of it is already running at a loss even now.
And what if 2x isn't good enough? Would anyone pay for a 10x larger model? Can we even realistically run such models as anything other than a very expensive PoC and for a very short time? And whos to say that even 10x will finally solve things? What if we need 40x? Or 100x?
Oh, and of course: Larger models also require more data to train them on. And while the Internet is huge, it's still finite. And when things grow geometrically, even `sizeof(internet)` eventually runs out ... and, in fact, may have done so already [1] [2]
What if we actually discover that scaling up doesn't even work at all, because of diminishing returns? Oh wait, looks like we did that already: [3]
Scaling applies to multiple dimensions simultaneously over time. A frontier model today could be replicated a year later with a model half the size, with a quarter of the FLOPS, etc. I don’t know the real numbers for optimization scaling, but you could check out NanoGPT speedrun [1] as an example.
The best solution in the meantime is giving the LLM a harness that allows tool use like what coding agents have. I suspect current models are fully capable of solving arbitrary complexity artificial reasoning problems here, provided that they’re used in the context of a coding agent tool.
> Scaling applies to multiple dimensions simultaneously over time. A frontier model today could be replicated a year later with a model half the size
Models "emergent capabilities" rely on encoding statistical information about text in their learnable params (weights and biases). Since we cannot compress information arbitrarily without loss, there is a lower bound on how few params we can have, before we lose information, and thus capabilities in the models.
So while it may be possible in some cases to get similar capabilities with a slightly smaller model, this development is limited and cannot go on for an arbitrary amount of time. It it were otherwise, we could eventually make a LLM on the level of GPT-3 happen in 1KB of space, and I think we can both agree that this isn't possible.
> giving the LLM a harness that allows tool use like what coding agents have
Given the awful performance of most coding "agents" on anything but the most trivial problems, I am not sure about that at all.
Some problems are just too complex and the effort to solve them increases exponentially. No LLM can keep up with exponenentially increasing effort unless you run them for adequatte number of years.
What? Fundamentally, information can only be so dense. Current models may be inefficient w.r.t. information density, however, there is a lower bound of compute required. As a pathological example, we shouldn't expect a megabyte worth of parameters to be able to encode the entirety of Wikipedia.
> They are good at repeating their training data, not thinking about it.
Which shouldn't come as a surprise, considering that this is, at the core of things, what language models do: Generate sequences that are statistically likely according to their training data.
This is too large of an oversimplification of how an LLM works. I hope the meme that they are just next token predictors dies out soon, before it becomes a permanent fixture of incorrect but often stated “common sense”. They’re not Markov chains.
Sure, but a complex predictor is still a predictor. It would be a BAD predictor if everything it output was not based on "what would the training data say?".
If you ask it to innovate and come up with something not in it's training data, what do you think it will do .... it'll "look at" it's training data and regurgitate (predict) something labelled as innovative
You can put a reasoning cap on a predictor, but it's still a predictor.
They are both designed, trained, and evaluated by how well they can predict the next token. It's literally what they do. "Reasoning" models just buildup additional context of next token predictions and RL is used to bias output options to ones more appealing to human judges. It's not a meme. It's an accurate description of their fundamental computational nature.
Yes. That's not the devastating take-down you think it is. Are you positing that people have souls? If not, then yes: human chain-of-thought is the equivalent of next token prediction.
The problem is in adding the word "just" for no reason.
It makes the statement of a fact a type of rhetorical device.
It is the difference between saying "I am a biological entity" and "I am just a biological entity". There are all kinds of connotations that come along for the ride with the latter statement.
Then there is the counter with the romantic statement that "I am not just a biological entity".
Oh how the turn tables :D
reply