I think your compilers example doesn't fit. They are actually a rather straightforward thing to build, many undergraduates build one as part of their studies, some hobbyists build compilers that get used by thousands of people and in fortune 500 companies core infrastructure. There's little rigor involved.
Among the few software systems that need rigor are control systems for physical installations and trading/finance systems for example.
Also many production-grade compilers (GCC/G++, Clang, OpenJDK, V8, and almost every new language that's come out since the 90s) are open-source. You can go read the commit logs & source code to see how they work, if you're diligent and willing to slog through them. There are certainly tricks that professional compiler writers use that aren't covered in textbooks (the big ones center around error-reporting, incremental compilation, fancy GC algorithms, and certain optimizations), but you can always go consult the source to learn about them.
I thought the thread was really about domains where the bulk of knowledge is locked up in industry rather than being about rigor, but I'd put control systems in that category as well. Also information retrieval (Google's search algorithms are about 2 decades ahead of the academic state-of-the-art...the folks at Bing/A9/Facebook know them too, but you aren't going to find them on the web), robotics, and aerospace.
im generally talking intel c++ compilers, etc. When I was doing EDA, we used to fork out a lot of cash for these compilers and these guys used to work very closely with us to optimize for certain kinds of code styles - for example loop unrolling on HP-UX, etc. I dont know if this is still a thing, so i might be mistaken.
Among the few software systems that need rigor are control systems for physical installations and trading/finance systems for example.