I don't see the issue. Just let wrong code do wrong things But let it do the expected wrong thing, rather than changing the code to something unexpected.
> The best way to program GPUs is face up to the reality that they are not the same machine as the CPU, write your kernels in separate files, and launch them manually,
Yes, I also prefer doing it that way, but in Cuda with the driver API. Allows you to handle kernels like shaders, including editing and hot-reloading at runtime.
The reason I'm sticking with CUDA is because it's by far the most convenient API to use, without nonsense like 50-liners to alloc memory or the need to manage descriptors, bindings, queue families, etc.
> The reason I'm sticking with CUDA is because it's by far the most convenient API to use, without nonsense like 50-liners to alloc memory or the need to manage descriptors, bindings, queue families, etc.
I was there when the OpenCL committee was deciding on that sort of stuff.
As I recall, and it's been two decades and a lot of sleepless nights since then, there was real pushback at the time against OpenGL-style default bindings. So folks didn't want to establish an implicit command queue or any other default objects attached to other objects. Part of it is because OpenGL was perceived as clumsy and passé, some of it was because it is not friendly to multi-threaded applications.
Those first meetings were a shitshow full of tension, implicit threats from Apple, and backroom deals. Kudos to Neil Trevett for chairing the group; I I bet it wasn't fun for him either.
That's unfortunate. Cuda has shown that, when done right, defaults and a convenience layer can make for a well received API without sacrificing performance.
Gif is still there because it's still often the only option for frictionless video embedding. Embedded videos usually come with a video player that hardly ever quite works out well, often not starting automatically, not looping, or showing video player UI elementa that shouldnt be shown.
The problem with the open standards is that their dev UX is absolutely horrible. You can't neglect usability, and then be surprised that there are no users.
Note that Vulkan is only somehow better than anything else out of Khronos, because at least for that API someone is paying LunarG for the Vulkan SDK, and NVidia has a vested interest in now pushing slang as the shading language.
The moment LunarG no longer gets sponsored it will be back to "here is a standard, all the best, the community will provide the tools".
On the SYSCL front, that is mostly driven by Intel, and their own OpenAI tooling, there are hardly any offerings for SYCL from anyone else. Thus it is really only pseudo-open.
Consider that for the others you are supposed to build your own SDK from random pieces of libraries, not all of them playing well together.
In that regard, the Vulkan SDK is definitely an improvement.
However using Vulkan on Android is actually hardly any different, given that there is no SDK as such, only the libs and headers get dumped into the NDK, there is a github with samples from Google and that's it.
So bad that they failed to convince anyone to drop OpenGL ES from Java/Kotlin, and use NDK alongside C or C++ for Vulkan, thus now there is WebGPU implementation for Java and Kotlin consumption.
I do agree with the Vulkan code itself, related extension spaghetti, and overall state of drivers, which has become visible enough to have been part of Vulkanised talks for two years in a row now.
This is at the core of the matter for me, and my knowledge is too weak to understand why this is the case. I don't enjoy the idea of relying on Nvidia's stack for GPU compute, but the alternatives I've tried (e.g. Vulkan compute) are higher friction to use. I am trying to reconcile why; Nvidia shouldn't have this moat.
My software is labeled "CPU only unless using an nVidia GPU". I would prefer to strikethrough "nVidia". Incidentally, this means no more Mac support.
All commercial APIs have this moat, which is why other than Carmack famously advocating for OpenGL, the large majority of other devs never cared, they implement an API agnostic engine, use the best API for each platform, and move on with the rest of the game.
Even Carmack later admitted that DirectX had become much better than OpenGL, but they weren't going to change given the studio culture.
Vulkan compute is not really designed or intended to be a CUDA competitor, its feature set is much more restricted, and Vulkan host side code is much more verbose than CUDA. OpenCL or SYCL are much closer in features to CUDA. I found that when using SYCL on Nvidia, debugging symbols etc can be passed through and you can use tools like NSight Compute to profile it as if it were CUDA.
I tried getting LLMs to add proper Vulkan support to ik_llama.cpp, which have very good support for CUDA and CPU. The models do an admirable job; they don't care much about poor DX.
Few problems I noticed:
* coopmat2 from nvidia is the classic embrace, extend, extinguish. No point to ask the models to translate from CUDA to coopmat2. Instead, the models can understand the existing CUDA and CPU kernels, and adapt them accordingly to non-nvidia devices.
* However, the standard API is also lacking. The models struggled to make prompt processing compute-bound on strix halo when the graph is complex. Upfront standard API might just be an evolution dead end.
On the other hand, despite my complain about the standard API, the models were able to come up with cooptmat1 kernels that run dsv4 flash faster than whatever the guys at antirez/ds4 can come up with using rocm, on a strix halo, with the added benefit that I can also pair the strix halo with an egpu to drastically speed things up.
From what I can tell, coopmat2 can get to about 75~90% of cuda performance on a single device, and there is no good way to do direct communication across devices. It is fair to say that nobody would replace cuda with coopmat2? That looks like a EEE project that can assigned to a couple of nvidia engineers, to fragment the ecosystem.
coopmat2 features will eventually be rolled elsewhere. coopmat also started as an NVIDIA extension.
The client use cases that coopmat was intended for are customer machines, not multi-GPU, which is broadly seen as a datacenter feature instead. That said coopmat orthogonal to this.
So when I said "a couple of nvidia engineers", I indeed meant Jeff.
VK_KHR_cooperative_matrix - embrace?
VK_NV_cooperative_matrix2 - extend?
I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
> I am pretty sure VkImportSemaphoreFdInfoKHR, mentioned in https://github.com/ggml-org/llama.cpp/issues/22648, works across multiple AMD devices, but somehow doesn't work across multiple nvidia devices.
p2p is disabled on nvidia customer cards, vulkan device groups are shipped for the RTX 6000s
> Added support for creating Vulkan logical devices from multiple physical devices on select cards via VK_KHR_device_group_creation. This feature can be enabled by setting the environment variable
__VK_ENABLE_DEVICE_GROUPS=1.
It could easily be a competitor to Cuda, if it just made things easier. Like, why does it take 50 lines of code to allocate memory in vulkan, and just one single line in cuda? Vulkan should just provide a single-line gpuMalloc convenience function. And not just for allocation, for all the other nonsense as well.
I'm generally suspicious of the effectiveness of ads. It always amuses me when Ibbuy something, and then I start getting ads for the thing I already bought.
I've had great experienced with it. It's almost certainly not as full-featered as premiere pro, but it runs buttery smooth and robust, while Premiere Pro kept crashing frequently and was a bit tedious to use for simple tasks.
Not at all. I specifically bought a MacBook Air M4, first Mac product in my life, for its fantastic performance in a passively cooled notebook. I have zero loyalty to Apple and usually use Windows. If someone else starts beating them perf wise, I'll happily switch again.
I've never understood this argument. Even if self-hosted things were offline more often (which I've never found to be the case, I've had Forgejo + runners running for a year now with no downtime), the real benefit is that you yourself can work to bring it back online when it does, rather than waiting on a large, slow-moving organization to figure out what slopped PR caused their global service serving ungodly amounts of RPS to go offline again.
Except it's not down more often and it's not a heavy lift. It's not a mail server, it's a Postgres database, a Golang app, and backups (and some providers will just handle that for you!).
Why? Like I wrote in a separate comment, it's not a mail server, it's a Postgres database, a Golang app, and backups (and some providers will just handle that for you!)
What do you mean "why"? Setting up and tinkering with systems is work I loathe to do, obviously. I want to code, not host coding repositories. I did it in the past because I had to, and now I don't have to do it because github does it for me. It's great.
Except when Github’s down, of course. I’m saying it’s incomprehensible to me that people would rather put their trust into an organization as enshittified as Microsoft where they get their codebases added to an AI, have an awful experience (the code review UX, for example), and deal with consistent downtime, rather than put a few hours every so often into something that they loathe to do. That does not seem like a good trade-off at all.
Self-hosting a service like GitHub that operates at GitHub scale is difficult.
Self-hosting a service like GitHub that operates at the typical small/medium company's scale is trivial.
A single machine (with separate runners for CI) will cover many companies' needs. It being a single machine eliminates a lot of the complexity and failure modes associated with a distributed system and makes backups/restores/maintenance easy.
reply