> I bought the cheapest compatible BTicino intercom device (BT 344232 for 32 €) that I could find on eBay, then soldered in 4 wires and added microcontrollers to make it smart. It now connects to my Nuki Opener Smart Intercom IOT device, and to my local MQTT Pub/Sub bus (why not?).
Yeah, when you run ≈10k agents instead of ≈10, you need a different solution :)
I’m curious what gVisor is getting you in your setup — of course gVisor is good for running untrusted code, but would you say that gVisor prevents issues that would otherwise make the agent break out of the kubernetes pod? Like, do you have examples you’ve observed where gVisor has saved the day?
I've used both gVisor and microvms for this (at very large scales), and there are various tradeoffs between the two.
The huge gVisor drawback is that it __drastically_ slows down applications (despite startup time being faster.)
For agents, the startup time latency is less of an issue than the runtime cost, so microvms perform a lot better. If you're doing this in kube, then there's a bunch of other challenges to deal with if you want standard k8s features, but if you're just looking for isolated sandboxes for agents, microvms work really well.
since we allow agents to execute arbitrary python, we treat every container as hostile. we've definitely seen logs of agents trying to crawl /proc or hit the k8s metadata api. gvisor intercepts those syscalls so they never actually reach the host kernel.
The reason why virtualization approaches with true Linux kernels is still important is what you do allow via syscalls ultimately does result in a syscall on the host system, even if through layers of indirection. Ultimately, if you fork() in gVisor, that calls fork() on the host (btw fork() execve() is expensive on gVisor still).
The middle ground we've built is that a real Linux kernel interfaces with your application in the VM (we call it a zone), but that kernel then can make specialized and specific interface calls to the host system.
For example with NVIDIA on gVisor, the ioctl()'s are passed through directly, with NVIDIA driver vulnerabilities that can cause memory corruption, it leads directly into corruption in the host kernel. With our platform at Edera (https://edera.dev), the NVIDIA driver runs in the VM itself, so a memory corruption bug doesn't percolate to other systems.
> Ultimately, if you fork() in gVisor, that calls fork() on the host
This isn't true. You can look at the code right here[1], there is no code path in gVisor that calls fork() on the host. In fact, the only syscalls gVisor is allowed to make to the host are listed right here in their seccomp filters[2].
I was more specifically referring to the fact that to implement threads in gVisor, it calls to the go runtime, which does make calls to clone() (not fork()), but I see the pushback :)
I think it's a small distinction. fork() itself isn't all that useful anyways.
However, consider reading a file in gVisor. This passes through the IO layers, which ultimately will end up a read in the kernel, through one of the many interfaces to do so.
You might want to add a little note to that effect to your link blog :)
I have added year indicators to my blog (such that old articles have a prominent year name in their title) and a subscribe note (people don’t know you can put URLs into a feed reader and it’ll auto-discover the feed URL). Each time, the number of people who email me identical questions goes down :)
GNOME’s “proper wayland implementation” also does not work with my monitor, as I explained in the article:
> By the way, when I mentioned that GNOME successfully configures the native resolution, that doesn’t mean the monitor is usable with GNOME! While GNOME supports tiled displays, the updates of individual tiles are not synchronized, so you see heavy tearing in the middle of the screen, much worse than anything I have ever observed under X11. GNOME/mutter merge request !4822 should hopefully address this.
"Jen-Hsun Huang certifies that he is the president and secretary of NVidia Corporation, a California corporation." - ARTICLES OF INCORPORATION OF NVidia Corporation, 1993, filed with the California Secretary of State and available online.
"The name of this corporation is NVIDIA Corporation." - 1995 amendment.
No, the lesson of “separate display server from window manager” was very clear when Wayland was started. People have been discussing this over the years ever since. (See also “client-side decorations” for another part of this issue that was heavily discussed.)
I seem to remember reading in an old paper (1990s?) that the asynchronous nature of the connection between the X server and the window manager results in essentially unfixable protocol-level races.
Wayland has a philosophy of "every frame is perfect", which means fixing every race condition. However, X11 doesn't have this philosophy. If the window manager is slow and doesn't respond to a notification that a window has been resized, drawing the new window content over the old borders is the correct thing to do. What sense does it make to freeze the whole display just for a window border?
Similarly, tearing gets pixels to the screen faster.
Reacting somehow to user input, even if not perfect, is more important for me...
That's why we have HW cursors, and frame interpolation in games, etc...
Perfect frames is what Mac and Windows provide and what Linux should also aim for. Border tearing is a display bug, correctness should come first, Wayland's approach is right. X was designed for CPU and IO constraints that no longer apply. _Graceful_ degradation of slow UI should lower the frame rate, not compromise rendering of individual frames.
Well, exactly. Windows does try to provide perfect frames. If the display glitches, it's considered as bug rather than the momentarily acceptable degradation that OP was praising X for.
Tearing and hidpi is why I left Linux for Windows between 2012 ans 2022. Once wayland was good enough I returned. Tearing is awful, and should be opt in (which wayland provides), not opt out.
Conversely, I much prefer lowest latency at the cost tearing; when I'm forced to use windows I generally disabled the compositor there too whenever i could (I certainly don't use one under Linux and that's one of my reasons for being there). I find macOS unuseable, even on brand new top-end mac studios the input lag and slow reaction of the OS to... any user input, is frightening when you're used to your computer reacting instantly.
The races I recall being described were substantially worse, but that’s largely beside my point.
My point is that, now that bare fillrate and framebuffer memory haven’t been a limiting factor for 15 to 20 years, it is a reasonable choice to build a desktop graphics system with the invariant of every frame being perfect—not even because of the user experience, but because that allows the developer to unequivocally classify every imperfect frame as a bug. Invariants are nice like that. And once that decision has been made, you cannot have asynchronous out-of-process window management. (I’m not convinced that out-of-process but synchronous is useful.) A reasonable choice is not necessarily the right choice, but neither is it moronic, and I’ve yet to see a discussion of that choice that doesn’t start with calling (formerly-X11) Wayland designers morons for not doing the thing that X11 did (if in not so many words).
To be clear, I’m still low-key pissed that a crash in my desktop shell, which was deliberately designed as a dynamic-language extensibility free-for-all in the vein of Emacs or TeX, crashes my entire graphical session, also as a result of deliberate design. The combination of those two reasonable decisions is, in fact, moronic. But it didn’t need to be done that way even on Wayland.
> To be clear, I’m still low-key pissed that a crash in my desktop shell, which was deliberately designed as a dynamic-language extensibility free-for-all in the vein of Emacs or TeX, crashes my entire graphical session, also as a result of deliberate design. The combination of those two reasonable decisions is, in fact, moronic. But it didn’t need to be done that way even on Wayland.
This is not a given, though. It's entirely feasible to have a stable "display manager/server" with an extension/plugin system where your WM of choice would sit. Crashing the latter should not bring down the former.
> There are many race conditions that must be dealt with in input and window management because of the asynchronous nature of event handling. [...] However, not all race conditions have acceptable solutions within the current X design. For a general solution it must be possible for the manager to synchronize operations explicitly with event processing in the server. For example, a manager might specify that, at the press of a button, event processing in the server should cease until an explicit acknowledgment is received from the manager.
I went and checked, and it was an SPE article from 1990[1]: “Why X is not our ideal window system” by Gajewska, Manasse, and McCormack. Section 3 has extensive diagrams of various races, including ones between clients and the window manager. Was even discussed here at one point[2], it turns out. Where I came across it I’m not sure (on X.org[3] possibly?).
I covered that in the X-Windows Disaster chapter of the Unix Haters handbook, and many HN posts, and other articles. So it's quite disappointing and underwhelming that Wayland failed to learn or apply any of these lessons. All this stuff was widely discussed long before Wayland was designed in reaction to X11, without considering any alternatives than Windows and Mac.
>There is a certain justice in The UNIX-HATERS Handbook's description of my efforts:
Ice Cube: The Lethal Weapon
One of the fundamental design goals of X was to separate the window manager from the window server. “Mechanism, not policy” was the mantra. That is, the X server provided a mechanism for drawing on the screen and managing windows, but did not implement a particular policy for human-computer interaction. While this might have seemed like a good idea at the time (especially if you are in a research community, experimenting with different approaches for solving the human-computer interaction problem), it can create a veritable user interface Tower of Babel.
If you sit down at a friend’s Macintosh, with its single mouse button, you can use it with no problems. If you sit down at a friend’s Windows box, with two buttons, you can use it, again with no problems. But just try making sense of a friend’s X terminal: three buttons, each one programmed a different way to perform a different function on each different day of the week — and that’s before you consider combinations like control-left-button, shift-right-button, control-shift-meta-middle-button, and so on. Things are not much better from the programmer’s point of view.
As a result, one of the most amazing pieces of literature to come out of the X Consortium is the “Inter Client Communication Conventions Manual,” more fondly known as the “ICCCM”, “Ice Cubed,” or “I39L” (short for “I, 39 letters, L”). It describes protocols that X clients must use to communicate with each other via the X server, including diverse topics like window management, selections, keyboard and colormap focus, and session management. In short, it tries to cover everything the X designers forgot and tries to fix everything they got wrong. But it was too late — by the time ICCCM was published, people were already writing window managers and toolkits, so each new version of the ICCCM was forced to bend over backwards to be backward compatible with the mistakes of the past.
The ICCCM is unbelievably dense, it must be followed to the last letter, and it still doesn’t work. ICCCM compliance is one of the most complex ordeals of implementing X toolkits, window managers, and even simple applications. It’s so difficult, that many of the benefits just aren’t worth the hassle of compliance. And when one program doesn’t comply, it screws up other programs. This is the reason cut-and-paste never works properly with X (unless you are cutting and pasting straight ASCII text), drag-and-drop locks up the system, colormaps flash wildly and are never installed at the right time, keyboard focus lags behind the cursor, keys go to the wrong window, and deleting a popup window can quit the whole application. If you want to write an interoperable ICCCM compliant application, you have to crossbar test it with every other application, and with all possible window managers, and then plead with the vendors to fix their problems in the next release.
In summary, ICCCM is a technological disaster: a toxic waste dump of broken protocols, backward compatibility nightmares, complex nonsolutions to obsolete nonproblems, a twisted mass of scabs and scar tissue intended to cover up the moral and intellectual depravity of the industry’s standard naked emperor.
I just wanted to say thanks - I read the unix-hater's handbook about 15 years ago and it taught me a LOT, maybe even changed my life. The chapter on X was particularly memorable and entertaining. Thanks!
> I bought the cheapest compatible BTicino intercom device (BT 344232 for 32 €) that I could find on eBay, then soldered in 4 wires and added microcontrollers to make it smart. It now connects to my Nuki Opener Smart Intercom IOT device, and to my local MQTT Pub/Sub bus (why not?).
reply