Most here seem to know that the motivation for adding DLLs to unix was to make it possible for the X windowing system to fit in the memory of a computer of that time, but many comment writers here seem not to know something that the participants in the discussion that is the OP all knew:
Plan 9 has an alternative method for sharing code among processes, namely the 9P protocol, and consequently never needed -- and never used -- DLLs. So for example instead of dynamically linking to Xlib, on Plan 9 a program that wanted to display a GUI used 9P to talk to the display server, which is loosely analogous to a Unix process listening on a socket.
Problem with that is that it is less efficient than a shared library but also susceptible to the same issue, namely that if you change the interface then things break.
Edit: It does of course get you extra functionality, like being usable over a network connection. Trade offs.
Directly dispatching a function via a call per the ABI of your computer vs producing a serialized message stream, sending it to another process, deserializing a stream and then calling the function.
Yes, it's less efficient.
The better question might be "did it matter in practice?".
I once replaced an export operation that depended on COM calls to Excel with some text manipulation code and brought the run time down from 30 minutes to an hour to just a few seconds. So in at least that case it did matter in practice.
There are examples of very efficient IPC, just not in UNIX-land. If your IPC system is integrated with the scheduler, the difference between a dynamic library call and IPC can be minimized into irrelevance.
I am repeating stuff I learned over the years from internet discussions, e.g., on the 9fans mailing list, rather than from direct experience in writing GUIs in Plan 9 and in X. I think when the decision was made to add DLLs to Unix, Xlib, the library a program would use to talk over the socket, was itself too big to fit in memory if a separate copy got statically linked to every program that displays a GUI. (The Wikipedia page for Xlib says that one of the two main aims of the XCB library, and alternative to Xlib, were "reduction in library size".)
I'm not advocating for removing DLLs from our OSes, BTW. Nor am I advocating for Plan 9.
COM uses vtables and normal calls via function pointers, unless the caller and the callee are in incompatible contexts that require marshaling (remoting, different processes, or different threading apartments in the same process).
Plan 9 has an alternative method for sharing code among processes, namely the 9P protocol, and consequently never needed -- and never used -- DLLs. So for example instead of dynamically linking to Xlib, on Plan 9 a program that wanted to display a GUI used 9P to talk to the display server, which is loosely analogous to a Unix process listening on a socket.