> Sadly, multithreading is an afterthought for Emacs
It is, but it's usable. I'm actually amazed that, even after three major versions, the built-in threading is not used by the community.
Yes, the threads currently are not usable for number crunching in the background. And yes, there are bugs, and trying to do many things from the background thread doesn't work, sometimes in unexpected ways. You can still block the main thread from the background thread since some things block the event loop, no matter where they were started.
But, the threads do give you independent control flows. Whatever you cannot do in the background, you can offload to the main thread with a timer and a queue of lambdas.
The built-in threads are very, very bare-bones - it's around 15 functions, for threads, mutexes, and condition variables. They are very limited by their "mostly cooperative" nature. However, with a bit of sugar, they are usable for at least one thing: async processes and network communication.
In a background thread, you can "block" to wait for a child process to do something. It's natural and requires no macrology (async.el...). The same is true for network communication. You can block and wait for a response while the rest of Emacs does whatever. With just two functions, you can write code without blocking as if you used `call-process`. Sequential actions - call this, wait for it to finish, call that, wait for it to finish, etc. - can now be coded in a sequential way, without having to worry about callbacks, sentinels, and a poor-man FSM implementation that invariably appears in Elisp that doesn't use threads.
The threads built into Emacs, currently, are closer to green threads or coroutines, functionally, than to OS-level threads. But that's still a huge help in a bunch of important and pervasive scenarios. It's really strange that nobody seems to realize this.
With threads (as they are), the Continuation Passing Style compiler macro (in generator.el), and dynamic modules (for actual parallelism where needed) Emacs now has everything it needs to make it non-blocking by default. Of course, that would entail rewriting everything on top of these abstractions, so it's unrealistic - but for new code and packages? I think we're just one package (along the lines of dash, s, etc.) away from convenient concurrency and parallelism in Emacs. The problem, of course, is that someone needs to design and code that package...
"Yes, the threads currently are not usable for number crunching in the background. And yes, there are bugs, and trying to do many things from the background thread doesn't work, sometimes in unexpected ways. You can still block the main thread from the background thread since some things block the event loop, no matter where they were started."
Many dynamic languages have bodged on threading over the past decade. (I don't think dynamic languages are intrinsically unthreadable or anything, but their interpreters were pretty deeply based on not having threads.) What that has shown is that 90%-effective threading is useless, and 99%-effective threading is superficially appealing but always, always blows up at any sort of scale.
You really need threads that don't come with all those caveats.
I expect it will get there, but another thing we've learned from previous efforts is that telling the community it's ready before it's ready causes "$LANGUAGE threading" searches to be filled with posts telling people how bad it is, even years and years after it has actually been fixed. It's probably a blessing in disguise it's not something the community is pervasively trying to use.
Well, threading - shared-state parallelism, more precisely - is hard to do well, and retrofitting it into a program that wasn't designed with that kind of threading in mind is even more challenging. I don't think any popular languages solved this, save for Java. Especially in the recent versions, with the new virtual threads - as much as I dislike Java, I have to say they did an excellent job on this. Other languages and platforms (that I know of; what's the .NET story here?) are all shitshows to varying degrees, trying to catch up and failing over and over again.
I think the only sensible way to offer parallelism in Emacs is to exclude the "shared-state" part, the way Racket (places) and OCaml do it. I think Python also tries to do it with subinterpreters? Let another instance of the interpreter run in the same process and communicate via message passing. That's probably still a huge undertaking, but at least it seems more viable than going through the whole codebase and adding locks everywhere...
Still, the "threads" in Emacs, as incomplete and half-baked as they are, can be useful. And if nobody uses them, there's no incentive for the developers to improve them. So I think we need at least some early adopters if we want the threading support in Emacs to get better.
Generally, I don't feel like in 2023 concurrency and parallelism are problematic areas anymore aside from existing aged stacks. From a perspective of mainly .NET ecosystem resident, it has been a shitshow outside of it for a long time indeed with many architectural choices throughout the industry paying for the Java sins (e.g. Kafka) and imposing limitations that seemed nonsensical and embarrassing even 7 years ago.
We're talking about parallelism here, not concurrency. async/await solve concurrency, not parallelism (on their own). Kotlin coroutines solve parallelism only because they piggyback on Java threads. I'm not sure about Go, but it's probably M:N concurrency (so with parallelism) like what you get on the BEAM. Then again, on the BEAM you don't get to "share" anything (other than binaries, IIRC).
I'd say concurrency is largely a solved problem, yes; limited parallelism (e.g., with message passing) also mostly works. We don't need to worry about the "C10K problem" anymore. But shared (mutable) state parallelism is, I think, still far from solved - if it can ever be "solved", which is a pretty big assumption :)
It is, but it's usable. I'm actually amazed that, even after three major versions, the built-in threading is not used by the community.
Yes, the threads currently are not usable for number crunching in the background. And yes, there are bugs, and trying to do many things from the background thread doesn't work, sometimes in unexpected ways. You can still block the main thread from the background thread since some things block the event loop, no matter where they were started.
But, the threads do give you independent control flows. Whatever you cannot do in the background, you can offload to the main thread with a timer and a queue of lambdas.
The built-in threads are very, very bare-bones - it's around 15 functions, for threads, mutexes, and condition variables. They are very limited by their "mostly cooperative" nature. However, with a bit of sugar, they are usable for at least one thing: async processes and network communication.
In a background thread, you can "block" to wait for a child process to do something. It's natural and requires no macrology (async.el...). The same is true for network communication. You can block and wait for a response while the rest of Emacs does whatever. With just two functions, you can write code without blocking as if you used `call-process`. Sequential actions - call this, wait for it to finish, call that, wait for it to finish, etc. - can now be coded in a sequential way, without having to worry about callbacks, sentinels, and a poor-man FSM implementation that invariably appears in Elisp that doesn't use threads.
The threads built into Emacs, currently, are closer to green threads or coroutines, functionally, than to OS-level threads. But that's still a huge help in a bunch of important and pervasive scenarios. It's really strange that nobody seems to realize this.
With threads (as they are), the Continuation Passing Style compiler macro (in generator.el), and dynamic modules (for actual parallelism where needed) Emacs now has everything it needs to make it non-blocking by default. Of course, that would entail rewriting everything on top of these abstractions, so it's unrealistic - but for new code and packages? I think we're just one package (along the lines of dash, s, etc.) away from convenient concurrency and parallelism in Emacs. The problem, of course, is that someone needs to design and code that package...