Lua doesn't distinguish between, "the key is not in the table," and, "there is a value for that key, and the value is nil." Setting the value for the key to nil simply erases it from the table.
You can just create a sentinel value and use that. eg. `local mySentinel = {}`. Then insert that value when that's the behavior you want. `nil` on its own rarely makes for a good semantic understanding of what value is actually in the table anyways.
Also, in Lua 5.4, the behavior is different -- you need to do `t[key] = undef` (which is the only context you can use `undef` in). And `nil` can actually be a value in a table. Not saying what's better or worse, but that the behavior in 5.4 is different.
Do libraries like Penlight know about your sentinels or ‘undef’? Maybe in ten years they will use undef, or maybe you whipped up your own functional library and keep it optimized and full-featured. But now we have what we have.
Besides, please tell: if you receive an empty object as a value in JSON, how do you preserve it in your ‘sentinel’ approach? Gotta need another special value for that.
For the JSON case, it doesn't make sense to use an empty object as a sentinel, for sure. JSON parsers usually have their own sentinel for `null`. Lua has a lot of issues, but `nil` being not-present hasn't been an issue that has come up a lot in practice. If you have any specific cases where that has been an issue, would be interesting to hear about them.
As I wrote in the first comment, the use-case is e.g. API integration, which is what ‘scripting’ often is, these days. More widely, any transformations or filtering or helper code where you accept structures and pass them on after mild fiddling—again, quite typical use for Python or JS if you ever pipe data between scripts in the terminal or over the network. You don't put your data schema in each of those scripts and maintain them on every change at the source. Imagine implementing `jq` in Lua with all those ‘sentinels’ throughout the whole function library.
The issue is that interoperability is crippled. The fact that you haven't so far run into these use-cases doesn't help with the problem that an entire, very common, class of functionality can't be done without tracking those nulls at each and every step with some crutches that bring their own troubles.
Yeah I actually don't think Lua is great for those things, tbh. Lua is not a good tool for manipulating JSON--JavaScript is probably sensible for that (it is JavaScript's object format, after all). And yeah I don't think it's good to use a bunch of external libraries in Lua, since as you say there's not a common ground for how data is organized. The libraries I rely on are are more native integration things like lpeg, luasocket, luasec, love2d, ... They've all been quite good. And yeah you keep saying 'sentinels' in quotes but honestly it's worked fine when using lua-cjson. There's always an impedance mismatch between JavaScript's object format and Lua's though. Lua also allows non-string keys which are great in Lua but not a thing in JavaScript, and in practice I've found the array/object-table mismatch more problematic than `nil`/`null` and so on.
I've mostly applied Lua for embedding in game engines, and to that end there's nothing that comes close, short of rolling your own VM that's exactly built the way you want. The main way to use it I think is to embed it somewhere and build up the set of utilities / libraries that you need to script over. But yeah it's not great for API glue code bc. APIs tend to bias toward JavaScript / JSON and other ways of doing things.
Well, my lament in the first place is that Lua could be a good and fast general-purpose scripting language. Instead, I can have ‘good’ but then I'll have to sit around twiddling thumbs while Python or JS start up. Especially Fennel runs circles around ClojureScript in terms of toolchain and build-cycle lightness.
JSON is pretty non-specific to JavaScript, which is probably a part of its popularity. And it's not the only serialization format. Null works about the same way in the majority of popular languages—so you're bound to need it for interoperability if you go outside C.
By libraries, I mean Lua libs like Penlight. Once you touch functional programming and function composition, you want structures to be preserved as they are, unless explicitly mangled.
> And yeah you keep saying 'sentinels' in quotes
It's because I haven't seen that notion before, at least under this name.
Have you tried lua-cjson, out of curiosity? They use a cjson.null (or something like that) value for nulls in JSON. But yeah agree with what you've said in general, if I had to do some API JSON manipulation Lua isn't what I'd jump to, even though I'm pretty familiar with it. Do Python or JS actually take that much longer to start up for a script? Node starts up pretty quickly on my system.
Now that I think of it, cjson.null might be workable even with data-transformation libraries, since it's some Lua meta-voodoo-value that can't come in the input from other systems. But then, of course, there's the problem that other data-exchange libs invent their own voodoo-values for null, like ngx.null. Precisely because they need it for actual integration use-cases.
As for the performance, I happen to use an underpowered machine, which is how I discovered the drastic difference with Lua. However, performance still matters beyond just ‘pretty quickly’ if you consider desktop software, especially productivity software—which has to be snappy. Scripting your productivity software is a textbook case for dynamic languages, and almost the exact niche for Lua. Even on my machine, there was a moment when I had to make sure that Alfred actually called my Lua script—because the results popped up immediately (despite going through the command-line interpreter, not Luajit). Meanwhile, people are making productivity software with Electron or Python, which to me verges on ridiculous.
Yeah, I definitely care about performance more in that tradeoff. In my experience when performance matters, these details about using such a null representation and what different representations lend themselves also tends to matter. vs. gluing libraries to libraries willy-nilly. At that point this null issue is far from the main thing that comes up bc. you care about data layout enough that you have control over that representation.
One example I'm running into in practice these days is--I'm using chipmunk2d game physics engine in a library. It does heap allocations, while all the component data for other stuff in my game tends go be well managed into pools that I can walk through contiguously. Even if I use a custom allocator for it, it uses pointers to refer between objects internally so I can't "move" instances of its eg. `cpBody` object around in memory. Would've been so nice to just own its data and let the library be layout-agnostic but alas.
But yeah those are the data-data glues I've been interested in--libraries that kind of own representation but not location I guess.
For your particular issue though, I think it really is bc. Lua is a language that out of the box doesn't tend to own representation, to the point of not eg. having its own class system etc, and it mostly shines when used in ways that play into that strength (eg. you want to wrap your own object ontology for ppl to script). The best libraries for Lua have tended to the invent their own representation and expect you to care about and own the mappings between libraries. esp. for JSON, the impedance mismatch with Lua tables puts it in this weird uncanny valley.
Tbh my fav method of using JSON is to not have an in memory document format at all and just treat it as reader / writer objects directly into domain data so that you don't keep paying this intermediatw representation conversion cost. eg. https://github.com/beached/daw_json_link is interesting here In my own game stuff I just have game components accept reader/writer objects to read from / write to and each component tends to have domain logic about what it means to be missing something, what some defaults are, etc.
> when performance matters, these details about using such a null representation and what different representations lend themselves also tends to matter. vs. gluing libraries to libraries willy-nilly
Well, Lua juuuust about hits the sweet spot of a good ‘layman’ dynamic language with great performance, and my hope is that perhaps it could be moved a bit to the generic-scripting-language side without losing performance. Basically, since libraries invent magic values to use instead of null, presumably Lua could provide such a value for them all to agree on.
> for JSON, the impedance mismatch with Lua tables puts it in this weird uncanny valley
Not seeing the mismatch here: to my knowledge, tables can be employed as (untyped) arrays or dictionaries, exactly the structures in JSON—and what I've long been using in PHP, JS and Python. If I'm not forgetting something, null is the only thing missing from making this triumvirate an integrated quartet.
> Tbh my fav method of using JSON is to not have an in memory document format at all and just treat it as reader / writer objects directly into domain data so that you don't keep paying this intermediate representation conversion cost
As we're discussing this on a post about a Lispy thing, I personally can't back your approach, for I'm lately buying into functional transformations big-time. Here I'm more on the convenience side: it seems to work fast enough for interactive desktop cases, though I've heard that the in-place method works wonders for busy web apps. In fact, afaik some high-level languages/environments kinda do the in-place thing implicitly by not copying strings between structures or when extracting substrings, and possibly by doing COW. Dunno if Lua does any of that (afaik it copies pointers between structures, not the data—this behavior is pretty much expected of runtimes these days).
However, I occasionally do wish that Clojure-style lazy structures were more widely employed, so I could use transformations without worrying that they might crunch some stuff needlessly.
https://luafun.github.io/intro.html is the best I've found for performant stream-y transforms. But I tend to just believe in always writing for-loops because they are how computers actually work and IMO it's easier to reason about what they are doing (of course, encapsulating logic per element and so on into their own functions). C++'s STL stuff I think is actually also good / ok here and one exception I make because you can see the code generated and things get inlined, and things like `std::remove_if` or `std::lower_bound` are actually well implemented. The rest of the functional shenanigans are cool I think for their theory (eg. as far as applicative functors for parsing and whatnot); but I think if you care about performance, the data layout in memory for cache utilization seems to be what matters now. Clojure is ok yeah but IMO you should know that it's impl'd with structural sharing + GC and decide that that's the behavior you want. Here's eg. a library that does persistent data structures of that sort in C++: https://sinusoid.es/immer/ (no transients for maps though)
Re: the Lua / JS mismatch -- I think the main thing is that you kind of have to decide if a table is an array or not, and in a lot of cases there's no sensible thing here necessarily, especially if there are a lot of `nil`s in the array -- so is it a sparse JSON array or an object with those keys (stringified -- which is gnarly) or what? etc. You can see settings for this in lua-cjson [1], and the fact that the settings exist is the issue. It's not as bad for pure JSON as much as when you start wanting to do things like generate a diff from changes in the Lua side (that's when you start getting sparse arrays) and apply those to the same document in JS, or the other way round and so on. I wrote a real-time automatic diff sync'ing thing once that was something like this; and that was the main place where there the impedance mismatch came up. nil-null basically didn't matter -- I decided on one conversion and stuck with it (I basically decided we would not use `null` for anything -- just use something that represents the actual semantic thing you want instead).
Also FWIW, I don't find CL super 'functional'. Scheme and Clojure a little bit, sure. The most functional stuff I feel like is Haskell / ML. If it's just about closures--I feel like every language except C, Zig and Ada has closures these days.
Lua strings are pretty simple -- just an array and immutable (there's no COW because there is no W) and always interned. You have to build a new string with the modification you want. For a 'builder' approach usually the pattern is to accumulate a table then call `table.join`. Usually I manage fine with `str = str .. 'new stuff'` though unless it's in some hot code path.
Yeah, I can see that. But I have to say: I think the mistake was on JSONs part. That's what `nil` should mean, imho: the value of anything which has no other value.
An object { "foo" : 12, "bar" : null } could just be { "foo" : 12 }, and if I was expecting a "bar" field, well, I can set it to null/nil myself.
Other people have pointed out ways to work around it, so I won't go down that road myself.
In Lua, nil is not a value. It's an all-encompassing void in which variables are suspended when not having some other value—simultaneously existing and non-existing: both manifest variables or fields of this reality and the infinite multitude of potential variables of other timelines. And you can't tell which reality you're in.
So it's useless for interfacing with ‘null’s of other languages.
In Lua, nil is assuredly a value. `return type(nil) == 'nil'` is a valid program which returns `true`.
It doesn't have the semantics you're used to from other languages. I happen to prefer Lua's approach, I think the distinction between undefined and null is clunky and I'm glad that `undef` didn't make the cut in 5.4.
You presumably disagree, which is fine.
There is one rough edge I've encountered, which is down to a serious wart in the standard library: `table.insert(tab, value)` inserts the value at the end of the array portion, which in Lua we express as `#tab + 1`.
But `table.insert(tab, index, value)` is what's used to insert into another slot in the array portion. This is a serious mistake which only prevails for historical reasons.
I beg of you: never shift the meaning of parameters around when you add more of them. Nothing good will ever come of this!
As a result, `table.insert(a, b, nil)` doesn't do the same thing as `table.insert(a, b)`. In the three parameter version, `b` is an index, in the two parameter version, it's a value.
It's not possible to write this sort of monstrosity in pure Lua, where nil is well behaved; but `table.insert` is a builtin, implemented in C, or whatever the host language of your particular Lua implementation happens to be.
In C, there's a difference between a `nil` on the parameter stack, and a shorter parameter stack. A Lua programmer will never see this: passing a third `nil` or just calling a function with two parameters will have the same effect.
You can actually implement that sort of logic in pure Lua too using `...` in your parameter list and then using `select('#', ...)` in the function body. For example:
> function f(...) print(select('#', ...)) end
> f(1, 2)
2
> f(1, 2, nil)
3
I didn't mean "it is impossible to build logic incorporating the number of parameters passed to a function", but rather "there will be no difference between calling an ordinary Lua function with (a, b, nil) vs (a, b), unlike table.insert, without extensive effort"
Sooner or later, a budding Lua programmer makes an index an optional, last parameter to a function, and then passes it through to `insert`.
If what you're inserting happens to be a number, this silently does the wrong thing. Suffice to say I won't ever forget this particular wart on what's become my favorite language.
You seem to be frustrated that Lua doesn't work like one other programming language: Javascript.
That's fine. Probably shouldn't use it. The nil handling is one of my favorite things about Lua, so let me offer you an alternative: instead of being grumpy about the language not working the way you'd like, instead embrace what it is, and work with it.
On the contrary, apparently you're unaware that a ton of languages and serialization formats have ‘null’. And there's nothing special about nil in Lua, it's just called ‘undefined’ in other dynamic languages (possibly with minor differences). Lua doesn't have a proper ‘null’.
But yeah, this discussion is pointless since you're obviously not concerned either about the use-cases of exchanging data with languages that are not like C, or about keeping structures ‘schemaless’ in Lua code.
null and nil are effectively synonyms, and I'm having difficulty understanding what about Lua's use of nil has caused problems for you.