They are right about null. To say it is a billion dollar mistake is completely over blown. Maybe it was at the time but now a null pointer exception is extremely easy to find and fix. It doesn't compare to the other complex bugs one has to deal with most of the time.
>They are right about null. To say it is a billion dollar mistake is completely over blown. Maybe it was at the time but now a null pointer exception is extremely easy to find and fix.
The whole idea is not to have to "find it and fix it" at runtime... Whether it's "easy" or not it's a moot point, as it is after your program just crashed or got into an undefined state, while your server is running or your desktop user is using it...
Yup, plus depending on the case it might be anything up to insanely hard to find that bug (imagine >100k line code base where will causes a subtile wrong state once in a thousand requests if build with optimizations under load which causes a chain of other subtile errors crashing the production server once a hour).
C will not throw an exception when you try to dereference a null pointer. C does not have exceptions. Instead, dereferencing a null pointer results in undefined behaviour, which is often tough to deal with. Having an exception thrown is a lot nicer. In C the compiler is free to make optimizations, based on the programmer's vigilance about which pointers may or may not be null, which you would have never expected on your own[1]. Debugging these issues is hell.
I think their point was that in modern languages (not C), having null in the language isn't so bad, as attempts to dereference null are handled fairly safely, generally with exceptions.
Same thing goes with modern languages throwing on signed overflow, where, again, C gives you the horrors of undefined behaviour.
I think the main issue is that the program should never reach a state where you're in a position to dereference a null. If it does, that means you didn't handle an error condition earlier in the program. Sure you can handle the null safely but that doesn't handle the real cause of the error.
The solution is for the language to offer 'option types', which essentially force you to check for null. Zig does this, and the result is much nicer than how things work in C: https://ziglang.org/documentation/master/#Optionals
More importantly, they let you safely not check for null (because with null-types, you should really always be handling it, because anything can go from never-null to null-in-this-one-case without warning)
And options help with solidifying input contract as well as the output. Knowing what's an acceptable input via a type system is just as relevant as knowing what is a potential output.
While an exception might be marginally better than undefined behavior, its actual occurrence can still leave your database in an inconsistent state or have other random detrimental effects. By the time you see the exception reported the damage has already been done (program flow interrupted unexpectedly). Due to the large amount of possible program states you would have to do an impossible amount of testing to make sure such surprises won't happen once the software gets into the hands of a user.
>an exception might be marginally better than undefined behavior
The difference isn't marginal, it's night-and-day. Undefined behaviour means the program can behave in unpredictable ways, either now, or at some other time. If you're lucky, your whole process explodes immediately, but that's not guaranteed. Undefined behaviour is the root of many a nightmarish hidden bug.
> its actual occurrence can still leave your database in an inconsistent state or have other random detrimental effects
Not if your exception-handling code is correct, surely?
I agree though that there are good arguments to be made against exceptions as they exist in many languages, particularly regarding how it interferes with control flow 'from a distance' as it were. I defer to the excellent Raymond Chen: https://devblogs.microsoft.com/oldnewthing/?p=36693
The interesting question is not what you do once a catastrophe has happened (program has reached invalid state) but how to avoid it in the first place. Exceptions are better than undefined behavior in coping with the former, but they still don't help you with the latter. The idea to “make invalid states unrepresentable” would give you an actual shot at this.
> Exceptions are better than undefined behavior in coping with the former, but they still don't help you with the latter
That's not right. Exceptions are thrown reliably and are always 'noisy'. Undefined behaviour may go completely unnoticed for a long time. Code that wrongly throws exceptions tends to get fixed.
I've seen undefined behaviour in code samples in respected technical books. The code happened to work fine when using the MSVC C++ compiler and x84/AMD64 targets, and will probably continue to do so.
> The idea to “make invalid states unrepresentable” would give you an actual shot at this.
That's a pretty good summary of what type-systems research is aiming for.
Not in cases where the optimizer did something tricky based on the assumption that null pointer dereferences never happen (which is allowed because they are undefined behavior so any behavior is correct).
Oh wow, I didn't know you were on here! I'm a huge fan of D so I just wanted to say thanks for all the work you've done. D is an absolute pleasure to work with.
This does not save you if that null pointer is supposed to point to an array, and subsequent index operations move it out of the protected first page (or the first couple, I forget how modern Unix does it).
This is indeed how certain language runtimes generate null pointer exceptions. It would be too slow to add comparisons against null to every use of a pointer. So they handle SIGSEGV and raise the exception if the faulting access is on the null page.
Interesting side question: if you have a struct bigger than a page, and you dereference the end of it... How big does it have to be to not get this behavior?
It's less about null specifically and more about taking advantage of a type system. A null pointer, or, really, any pointer address used as a special value, is an in-bound value. This makes the type system unaware of it.
By using optionals, the idea of a special "missing" state becomes an out-of-bound value, meaning the type system can participate in helping the programmer properly deal with missing values when the time comes.
So it doesn't even have to be null. Let's say you're writing a function that returns an integer, or MAX_INT indicating that something failed. That's the same problem. The error indicator is an in-bound value, which means the type system is unaware that MAX_INT is special, and won't be able to catch mistakes, such as using the return value without checking if it is the special value. It would also be a subtle bug if the integer size changed from int to long, and now MAX_INT isn't even the correct value anymore. If the type were an optional, everything would have continued working correctly.
Sentinel values are things the compiler has no idea about and I agree with you here. If the type system has no idea about these sentinel values, the compiler cannot help you with those edge cases.
One aspect I think many people are bringing up, implicitly, is that they want a language that enforces a form of "correctness" so that you cannot put your programming into an "invalid state". I understand the appeal of this but it is not free, it does come at a cost.
This implicit view assumes that all values are a form of "object" and "singular". Assuming pointers are "references" to a "singular object", rather than just a pigeonhole to a piece of memory which may have a type associated with it. This may seem like it is expressing he same concept, but the former is kind of like Aristotelian Object Orientated Ontology (OOO) naïvely applied to things which are not actually objects in the real world. Thinking of these things as "objects" is an abstraction and is not actually what is happening. It may have a lot of utility, but it is not the reality at hand.
In many regards, most forms of Object Orientated Programming (OOP) in reality are an application of OOO (even the term "virtual" is from Aristotle). And Rust's ownership and lifetime semantics are another application too. But explaining this is in itself a long article. I know I won't convince people here about this, and that's absolutely fine as I don't expect to. It's just interesting to read many people's implicit assumptions about what how things "ought to be" with regards to a programming language.
Ownership semantics also suffer from the misapplication of what ownership is. Property cannot own property. In this case, it just becomes a hierarchical dependency system of responsibility. So "ownership" itself is the wrong term.
"This object owns this object which in turn is owned by another object."
but rather
"This object is responsibly for this object which in turn is responsible for this object."
In sum, you are artificially applying a hierarchy were one has not arose naturally, and adding the artificial concept of an "object" as an abstraction where there is no real "object" in reality. All of these abstractions that we apply to programming are just tools that we (hope to) get utility from.
In order to interpret anything, we must use a model. Some models are better than others, and some models are downright wrong.
If when you say “null pointer exception” you’re thinking of Java: do you find any value in nullable / non-null annotations? Or if you’ve used Kotlin, its non-nullable references?
I find them very useful. I don’t just want null pointer exceptions to be easy to debug, I want to avoid them completely in the first place.
You're right. The mistake is not in having null pointers, it's lacking non-nullable references which is the problem. If you can write a function which only accepts a non-null reference, then you don't have to check in every function called from it.
+1. I think this is something that gets lost in discussions around null. Generally, everyone complaining about C's handling of nulls will recommend languages that make nullable-pointers opt-in by having non-nullable references and making them the default. The argument people make is not that C should somehow drop the entire concept of null; the argument is that people should use languages where a type has to opt-in to accepting null.
A really effective way to handle nullable fields is an Optional<T> which forces the user to handle a null value for that field, this is very useful when working in a team and dealing with objects that someone else spent time creating, it won’t get rid of null but it definitely hints the user at “hey this field may be null so I should go ahead and handle this before I even get to testing my code”
Edit: You would want to implement these in your getters for fields that may return null
Note that in C++ today, static analysis tools + classes such as `std::unique_ptr`,`gsl::owner` and `gsl::non_null` (similar to the Kotlin mechanism you described), it is possible to avoid the possibility of null pointer dereferencing.
That's not to say the same thing is possible in C, but one _can_ go a long way with static analysis at least, and suspicion vs provability regarding pointer dereferences.
You can configure your IDE to warn of incorrect usage, and you can configure your compiler to flag warnings as errors. That will catch a lot of potential problems at compile time. It’s not perfect but it can be lot better than nothing.
Java annotations are better than Optional when you’re interoperating with Kotlin code, such as when you’re partway through converting a large codebase to Kotlin.
@NonNull etc are not completely meaningless ... if you run SpotBugs it can be set up to fail your build if the contract specified in the annotations isn't met. I love static analysis!
I have to agree (ignoring the whole C doesn’t throw exceptions).
I have been looking at our bugs, logs and exceptions recently of the past 6 years and an enormous amount of bugs are caused by methods/functions that have multiple parameters with the same type (Java).
This happens because (my theory) we use java and java doesn’t have type aliases or value types as well as easy destructuring. It also doesn’t have named parameters (well there is a compiler option to retain the parameter name but it’s not like ocaml label parameter or python kwargs).
So often times in boring business programming you are dealing with methods with 5 to six strings so it’s very easy to mix up the parameter order.
Very few “hard” bugs were caused by NPEs where as the previous problem caused serious pain.
Agreed. I've been saying that for too many decades.
My version of that is "The big safety questions are 'how big is it', 'who owns it', and 'who locks it'. C helps with none of those."
Most of the things done with pointer arithmetic are really array slices without the right syntax. If you're doing something with pointer arithmetic that can't be represented an array slice, you're probably doing something wrong. I once proposed adding slice syntax and array sizes to C. This was discussed on comp.lang.c at some length, and looks backwards compatible. But the political problems are huge.
Politics are a big issue regarding security improvements with C, every attempt to improve the language's security has failed, including the now optional Annex K.
Hardware solutions like Solaris SPARC ADI, ARM MTE used by iOS and a requirement for future Android ARM devices, CHERI CPU seem to be the only way to tame those developers.
Sadly Intel dropped the ball on MPX, leaving x86/x64 as the only CPU not pursuing such kind of endeavours.
So memory tagging seems to be the only way.
Thankfully C++ community did not inherit this part of C culture, and several activities are in process to at least minimize the security impact of such features, e.g. lifetime profile, reducing the amount of UB, security guidelines, library types instead of raw C ones, ....
This is why Odin does not have first class pointer arithmetic and has first class support for slices. I have found in practice that I don't actually need pointer arithmetic most of the time as slices solve that function.
The reason pointer arithmetic is so useful in C is because of its lack of array types, like slices. `x[n]` is the same as `*(x + n)` which means in C, you can do the "wonderful" trick of `n[x]`.
I feel like the resistance is people think pointers are C's secret sauce. Reality is the only advantage is it makes it easy to write bran damaged unopimized compilers for C. Which no one does. I remember back in the 90's my boss gave me shit for using array indexes instead of pointers. Being a brat I looked at the assembly and there was no difference.
Ditto the cultural proscription on passing/returning small stucts by value. In practice it's no worse than passing the arguments separately. And likely easier for the compiler to optimize around.
Ditto the cultural proscription on passing/returning small stucts by value.
Which really should be a compiler decision. Depends on the target CPU. If you pass something as const ref, the compiler should copy it if that's faster. For AMD64, it probably is. If something was just written, it's in the L1 cache and is really cheap to copy. On-chip data buses today can be as wide as 64 bytes. Anything not bigger than that is better copied.
NULL / nil isn't bad; however any case where encountering one is a problem is most often an issue of under-specified program design.
The better question is, what were you expecting there and how can it be described without a bare pointer? I often find a list is better, particularly in languages with syntax sugar for iterating through a list.
Agreed. Non-null pointers also come with their own set of problems:
- Increased language complexity. Initialization of (arrays of) structs with non-null pointer members is more complicated because there is no straightforward "default" value that can be used.
- Performance tradeoffs. Initialization of arrays and other containers is more costly for non-null pointers (or structs containing them) because we cannot simply zero out a block of memory.
- In memory unsafe languages it is possible for a non-null pointer to become null if its memory is overwritten (e.g. in custom allocator scenarios (also mentioned by the author of the article), interop scenarios, etc.). The type system now makes a false guarantee, and it becomes possible for a "non-null" pointer to trigger a null pointer crash. (It's worth mentioning that enums generally have a similar problem).
- I'd like do a more rigorous evaluation on this, but my gut feeling is that most null pointer bugs that I've encountered usually happen in situations where the pointer would have to be declared as nullable anyway, because I'm using null to represent a possible state.
I don't believe any of those tradeoffs are true for rust, which has references that may never be null (in safe rust).
I'll respond to each in turn:
- language complexity for arrays of nullable things
In rust you can easily type 'let v: Vec<Option<&MyStruct>> = vec![None; 10]'
Having a vector of 10 'None' option types is no more difficult than anything else. Also, Option<T> implements the default trait. It defaults to 'None.
- Performance
Rust's 'Option<Box<T>>' takes up just as much space as 'Box<T>'. The null value for the pointer is used as the None type for the option. In that case it's a zero-sized zero-overhead type. You can read about that here, and in various other places: https://doc.rust-lang.org/std/option/#options-and-pointers-n...
- In memory unsafe languages it is possible for a non-null pointer to become null
Yeah. In memory unsafe languages. Don't use one of those. In Rust, Haskell, etc the Option type can't lie like that without explicitly opting in to memory unsafety. Which hardly anyone does.
- my gut feeling is that most null pointer bugs that I've encountered usually happen in situations where the pointer would have to be declared as nullable anyway
Null pointer exceptions happen when a pointer is nullable, but some location forgets that it is. If the type-system encodes this in the form of Option<T>, it's impossible to forget that. If I have an 'Option<String>' and I have a function that expects a 'String' (but not a null string), with the option type I know I have to do something like 'call_func(val.unwrap_or(""))', or I have to match, or I have to do 'let Some(val) = optional_val { call_func(val) }' before I can use that thing.
The fact that I have to do a null-check is built into the language and the compiler yells at me if I forget about it.
Being able to encode in the type-system that null should encode some valid state (e.g. 'None' in the option type) vs null-able pointers, where you use pointers that may be null even when you don't want nulls and the compiler can't check you work.. It's night and day difference.
I'm curious; I had an idea (in fact, it's part of my Master's degree project) to create a language that brings over concepts from functional programming to systems programming. What would be beneficial for that, and what wouldn't be?
clarity: this is arguably not FP specific, but I think that having declarations for binary layouts would save a lot of very confusing shifting and masking
concurrency: to the extent that you can get by with pure functions, they also tend to be more trivially multithreaded. you also have quite a bit more flexibility wrt things like STM, or lazy evaluation, or other more novel notions f concurrence scheduling
regions: I think GC is pretty much a no-no, at least for some critical sections. but there is a lot you can do with static analysis to both make allocation more statically safe and more performant than generic heaps
closures: again, not specific to FP..but using closures makes asynchronous programming really quite nice
runtime: having a proper runtime with maps, higher order functions and and real string functions when not in the performance path really does save a lot of time.
immutability: not sure its a win, but I find it super instructive to think about what actually has* to be mutable in a big system..C really loses the distinction since with the exception of the verb and viral const, everything is mutable by default. you can also* play lovely tricks with explicit time ala MVCC and Daedelus
even the sum of that I think doesn't really justify a project...but if you're doing it anyways...why not try to make something a little nicer than the huge and difficult to debug standard C business
I don't have much to add but I do think you are correct about binary layouts. Correct about closures. I think with closures you can fix a lot of C's jankiness. Immutability. I've never been happy with 'const' as a 'bandaid' for mutability issues.
