This has not been my experience of using technology effectively. Without an understanding of the implementation details, you inevitably use something inefficiently or for not quite the right purpose.
I cannot imagine anyone using a database effectively on any significant amount of data without understanding indexes, how different joins work, why join order is important, what effect join orders have on performance, etc. Get to a certain scale and it's not enough to know about indexes; you need to understand the structure of b-trees, disk I/O performance, how CPU cache performance affects b-tree navigation even when index is cached in memory, how to use compound indexes effectively to reduce random access through the index, etc.
The constraints of CPU and memory never go away, and if you're trying to scale something, you're going to be limited on either or both of those resources. That in turn forces you to understand execution and memory behaviour of the abstractions you're working with. All abstractions leak when pushed.
> I cannot imagine anyone using a database effectively on any significant amount of data without understanding indexes, how different joins work, why join order is important, what effect join orders have on performance, etc.
But conversely you probably didn't have to understand what filesystem the database runs on, whether it is in a RAID array, whether the network connection was over Ethernet or T1, etc.
All abstractions leak. The question is how leaky they are. In my experience Haskell abstractions are much less leaky than most.
I did and do; when performance at the database level isn't right, you need to look into OS stats, I/O stats, disk stats (the SAN is an area ripe for unexpected contention, RAID5/6 act like a single spindle and inhibit random access in ways that RAID1 or RAID10 don't, stripe length in RAID5/6 bloats the size of small writes, etc.), but I had to stop somewhere :)
So I'm confused - how many years did you have to study before you felt confident enough in your grasp of all the underlying concepts to write your first "Hello World"?
Thunk leaks happen, but they aren't that scary. They all pretty much have one of a handful of root causes.
It's definitely a wart of laziness, but it's also pretty easy to avoid.
As far as abtraction goes, laziness doesn't leak. In fact, it's a big reason Haskell performance composes. The head . sort example is contrived, but it holds true in more complicated & useful examples as well!
I have to understand all the intricacies of the system, so people who choose to treat databases as black boxes don't have to. There is only so far this abstraction holds.
Fun aside, coworker recently had a fun issue where the database server would halt due to running out of disk space... except the disk had over 100 GB left.
Turns out NTFS has a maximum number of file fragments a file can have, and if it exceeds that it will refuse to grow the file (for obvious reasons).
This interesting, I work as a data engineer but never had to really understand how joins or indexes work because I work 99% of the
time with denormalized data. I also do not know much about b-trees. I think you come from the database developer point of you rather than the database user point of view.
There is also other aspect of this, even if I do not understand joins or b-trees I can measure performance so I can figure out which combintion of joins are the faster. The reason that I prefer performance testing over getting to know the theorethical background for certain things is because in many cases your performance also depend on the implementation details, not only on which tree data structure backs your implementation.
It's exactly because the performance depends on the implementation details that when you understand implementation details, it can guide what you test and help find an optimization peak faster - or come up with a theory of another peak more distant, and journey through the performance valley to find it.
Implementation details follow the contours of the fundamental algorithms and data structures. You understand the outline of the implementation details - the bones, if you will - from that. And you can dive into the source (or disassembler - I've done that on Windows) to fine tune knowledge of other specifics, with e.g. gdb stack traces on the database engine when it is abnormally slow to focus attention.
Without having gone to battle together over some performance problems, or me writing significantly longer battle-story posts, we're probably not going to be able to communicate effectively here.
<rant> I work with a team that has a 16 TB database, who doesn't understand how it works. They seem really surprised when their queries run slow. </rant>
This is exactly right. I use lenses all the time, but I have absolutely no idea how they're actually implemented, nor do I need to know.
This is abstraction. If there's one thing Haskell does well it's abstraction.
EDIT: It's really bizarre. We see these same responses to all the Haskell-or-Idris-or-whatever threads -- I wonder if there's some imposter syndrome going where "I can't immediately read/write Haskell" somehow morphs into "Haskell is useless". IME it's really rare for people who actually program in Haskell to have serious issues with the language. Yes there are issues from a smaller ecosystem, package management was bad (Stack fixed that), etc. etc. but there are very few fundamental problems with the language. Something so small as just Pattern Matching is a huge increase in productivity. Thankfully, quite a few languages have adopted pattern matching these days (Scala, Rust, TS, maybe even C++23?).
(The really big payoff comes from granular effects, but I'm sure the rest of the world will realize in about 20-30 years' time. The Erlang people already have, albeit in a different way.)
People look at weird syntax and discussions about things that have no resemblance to the problem they are facing, and conclude it must not be useful for anything real. Yes, those problems have no resemblance to any real problem because of abstraction, but most people's experience with abstraction is in a Java-like language where nothing good ever gets out of it.
You don’t need to understand the internals of a thing, until you do. Everything works fine as described in documentation until it doesn't for your use case. You might be lucky and find help from stackoverflow, otherwise you need someone who really grok it.
Haskell's internals are actually pretty easy to inspect!
- Haddock hyperlinked source makes it easy to understand libraries if you need to dig into internals
- ghci makes it easy to interactively learn the language - and a new codebase!
- Haskell can dump the stages of its codegen pipeline (Core, STG, Cmm)
- Profiling and Event logging exist and are easy to use
- You can even write an inspection test to assert certain behavior holds. For instance, that no allocations occur or that some abstraction (e.g. Generic) is truly erased by GHC's optimizations
When things go wrong, you might not have time to do so. When my project started to leak memory at enormous rate, I was able to find the issue quickly enough. But if I didn't know how all those things work, I would spend weeks or months learning all those things. Restarting application every 10 minutes for a week is not a good idea.
How is this different for Haskell than with anything else? If you want to be an expert in something, anything you do need to put in the work and learn it inside-out. There is no royal road. I fail to see how this is specific to Haskell...
There was this piece of common knowledge floating around a number of years ago about how you need to know at least 1 level of abstraction beneath you well, and have a working knowledge of the second one below it, to use your tools effectively. I don't recall where the advice floated around or came from but it was something along those lines, and it's pretty true.
I actually think you do need to understand rendering logic to some extent to use CSS effectively.
For example I have seen many having a hard time understanding why it is trivially easy to align an element to the top of the screen but tricky to align something to the bottom of the screen - something which would be symmetric and equally simple in a typical application GUI framework.
But understanding how layouts are generated makes this clear.
IMO the conceptual level below CSS is in the design sphere - the box model itself. You would not believe the number of people who hack together properties until they get something that looks right, without understanding the box model, who think they're really good frontend developers.
Only if you have infinite memory. In theory there's no difference between folding left and folding right (if associative), in practice there is a right way and a wrong way.
It depends on your needs. This is neither constrained to Haskell specifically nor functional programming more generally.
If you were building a website for your local Italian restaurant, what would your needs be? Do you need an ElasticSearch cluster to handle customer menu item queries? Do you need a database at all?
In Haskell's case it's best to avoid lists entirely, as they're _usually_ not the optimal data structure. But best for whom? Does the beginner care that a set operation would be more effective than a list operation for their given use-case?
In my day job, I frequently generate records returned from a database along with local changes to be posted later, and say compute the sum of one of the columns. That sounded like something I'd use folding for, with my limited knowledge, so I was just curious at which point (order of magnitude) I'd have to worry about doing it this way or that.
But if lists are not to be used, what should I use for the above? And will the data structure you propose be fine with either fold?
Again, it depends. A left fold might be fine (and likely will be). Lists might be fine (and likely are). I don't know anything about the size of your data, or about the business you're writing software to support (who knows, maybe you're writing a FPS game, or some real-time aviation software!).
At this point (taking into account the extent of your knowledge as you yourself described it), my advice would be to just use basic folding and lists. At some point in the future, if you observe some performance issue in exactly that area, you might remember this and think "hmm, memory consumption is bit excessive here; maybe I'll try a foldr instead", or "finding the intersection of these two big lists is a little slow; maybe I'll convert them both to sets instead?"
> So do other languages. Why is Haskell special in this regard?
I never suggested other programming languages don’t also have value. Again, if you want to educate yourself further on a specific technology’s benefits, I invite you to make use of a search engine instead of sea-lioning on a technical forum.
> None that seem to address the problem of diminishing returns.
That’s your opinion. Nobody is forcing you to like Haskell. You are free to just ignore it.
