I might strengthen your argument even more and say it's largely a non-technical problem. We have had the tools necessary to build good software for a long time. As others have pointed out, I think a lot of this comes down to incentives and the fact that no one has demonstrated the tradeoff in a compelling way so far.

I find it really interesting that no one in the future of programming/coding community has been able to really articulate or demonstrate what an "ideal" version of software engineering would be like. What would the perfect project look like both socially and technically? What would I gain and what would I give up to have that? Can you demonstrate it beyond the handpicked examples you'll start with? We definitely didn't get there.

It's much harder to create a clear narrative around the social aspects of engineering, but it's not impossible - we weren't talking about agile 20 years ago. The question is can we come up with a complete system that resonates enough with people to actually push behavior change through? Solving that is very different than building the next great language or framework. It requires starting a movement and capturing a belief that the community has in some actionable form.

I've been thinking a lot about all of this since we closed down Eve. I've also been working on a few things. :)

I'll take this opportunity to appreciate C# in VS as a counterexample to the article. Fast as hell (sub-second compile times for a moderately large project on my 2011 vintage 2500k), extremely stable, productive, and aesthetically pleasing. So, thanks.

It's very hard for me to get away from C# because it's just so crazy productive. The tooling is fanstastic and the runtime performance is more than good enough.

One thing I found was that surprisingly the C# code I write outperforms the C++ code I used to write at equal development times.

I was good at C++, but the language has so many footguns and in general is so slow to develop in that I would stick to "simple" and straightforward solutions. I avoided multi-threading like the plague because it was just so hard to get right.

Meanwhile in C# it's just so easy to sprinkle a little bit of multithreading into almost any application (even command-line tools) that I do it "just because". Even if the single-threaded performance is not-so-great, the end result is often much better.

Similarly, it's easy to apply complex algorithms or switch between a few variants until something works well. In C++ or even Rust, the strict ownership semantics makes some algorithm changes require wholesale changes to the rest of the program, making this kind of experimentation a no-go.

The thing that blows my mind is the "modern" approach to programming that seems to be mostly young people pretending that Java or C# just don't exist.

Have you seen what JavaScript and Python people call "easy?" I saw a page describing a REST API based on JSON where they basically had thousands of functions with no documentation, no schema, and no typed return values. It was all "Just look at what the website JS does and reverse engineer it! It's so easy!"

I was flabbergasted. In Visual Studio I can literally just paste a WSDL URL into a form and it'll auto-generate a 100K-line client with async methods and strongly-typed parameters and return values in like... a second. Ditto for Linq-2-SQL or similar frameworks.

I've also been lurking on the FoC community, and hadn't seen much on an articulation on the social and incentive structures that produce software. Do you think they'd be receptive to it?

And by "social and inventive structures", I'm assuming you're talking about change on the order of how open source software or agile development changed how we develop software?

While agile did address how to do software in an environment for changing requirements and limited time, we don't currently have anything that addresses an attention to speed of software, building solid foundations, and incentives to maintain software.

What would a complete system encompass that's currently missing in your mind?

This is very much a social and political problem. Will be interesting to see if us technical folks can solve it.

I think you will see great change if you were to look at the personalities around one opportunity.

Because it's never problems really, it's perceived that way though.

A certain challange needs a specific set of personalities to solve it. That's the real puzzle.

Great engineers will never be able to solve things properly unlessed given the chance by those who control the surroundings.

We seek how we should develop, what method should be used, is it agile or is it lean? But maybe the problem starts earlier and focusing on exactly what methods and tools to use we miss out on the most simplest solution even beginners can see.

For example I am an architect, I tend to not touch the economics in a project. It's better fitted for other persons.

While not having read much about team based development I do want to be directed to well read literature about it. Maybe it's better called social programming, just another label of what we really do.

The one I miss the most at work is my wife. She clearly is the best reverse of me and makes me perform 1000x better. I find that very funny since she does not care about IT at all.

Maybe we should enforce some guidelines, and sponsor some programs to address these issues.

There's ways to develop working software, but not if it's all locked behind closed OSes and other bullshit.

The stuff I write I don't think is that bloated, but like most things these days the stuff I write pulls in a bunch of dependencies which in turn pulls in their own dependencies. The result, pretty bloated software.

Writing performant, clean, pure software is super appealing as a developer, so why don't I do something about the bloated software I write? I think a big part of it is it's hard to see the direct benefit from the very large amount of effort I'll have to put in.

Sure I can write that one thing from that one library that I use myself instead of pulling in the whole library. I might be faster, I might end up with a smaller binary, it might be more deterministic because I know exactly what it's doing. But it'll take a long time, might have a lot of bugs and forget about maintaining it. Then end of the day, do the people that use my software care that I put in the effort to do this? They probably won't even notice.

I think part of it is knowing how to use libraries. It's actually a good thing to make use of well-tested implementations a lot of time rather than re-inventing the wheel: for instance it would be crazy to implement your own cryptography functions, or your own networking stack in most cases. Libraries are good when they can encapsulate a very well-defined set of functionality behind a well-defined interface. Even better if that interface is arrived at through a standards process.

To me, where libraries get a bit more questionable is when they exist in the realm of pure abstraction, or when they try to own the flow of control or provide the structure around which your program should hang. For instance, with something like Ruby on Rails, it sometimes feels like you are trying to undo what the framework has assumed you need so that you can get the functionality you want. A good library should be something you build on top of, not something you carve your implementation out of.

A good compromise would be to replace bloated modules with alternatives that are leaner and have fewer nested dependencies.

Most developers I have known want to work on the new great new thing. They don't want to spend a great deal of time on the project either. Forget about them wanting to dedicate time to software maintenance. Not sexy enough.

Ok but why ? And what can we do to improve things? Promote maintenance, but I think one of the issues is that you can show something new, it's much more difficult to show that something could have changed (failure, difficulty to grow), but didn't.

To the extent it's in your power as a developer and a team member, don't tolerate low-performance code from yourself or your co-workers.

In my experience, a lot of performance problems boil down to really stupid problems, like simple code using the wrong data structure out of convenience (e.g. linked lists instead of arrays for lots of randomly-accessed data), or structured in a bad way (e.g. allocating a lot of small pieces of memory all the time). Often times, there are cheap performance wins to be had if you occasionally run the product through a profiler and spend couple of hours fixing the most pressing issue that shows up. Couple of hours isn't much; there's enough slack in the development process to find those hours every month or two, without slowing down your regular work.

I agree with your point of developers being responsible for the performance.

But I have a different experience (probably because we work in different areas):

Most the performance problems of the products I ever worked were purely systemic.

They boiled down to technologies and architectures having been chosen for "organizational" rather than technological reasons.

And "organizational" is in quotes because sometimes it was just blackmail: I worked with two developers who quit in protest after the prototype they wrote in Scala was deemed not good enough and dropped for... being too slow, ironically.

This has been a major frustration for me as a UI developer on the current application I work on. The UI is often hamstrung by how the backend API was implemented. There are frequently cases where we stitch together pre-existing API functionality to make something work in a far-from-ideal manner just because it would take longer to do it right and no one is interested.

I've seen a similar thing happen. It all started with good intentions, like only having simple endpoints that do "one and only one thing".

In the end the backend was pure and beautiful, but the the frontend devs had to perform joins in the the client and make 21 API calls in a 20-item list and then everything goes to hell.

Ah, I can tell you such stories of a stack that evolved solely out of incompetence...

It's not a technical problem at all. It's an economy problem.