Most Optional implementations are kinda terrible, though. They cost more than they save. If I have a function with a param and I release it with it required, but later I change it to be optional (loosening a requirement) does rust require everyone calling that function the old way to change their code? If so that not an improvement!
You're basically just regurgitating a recent Rich Hickey talk. Interested readers can probably find it on YouTube.
Rich is certainly a brighter individual than me, but some of his points are either him missing the point or being intentionally misleading.
For example, he discusses how Either types aren't true sum types because Either<A, B> isn't the same type as Either<B, A>. So he disparages people who say that Rust/Scala/Whatever have sum types. He's missing the point because 1) All Either implementations I've seen have the ability to swap the arguments to match another Either with the types backwards, so it's a sum type in practice, and 2) Clojure has none of it, so why criticize the typed languages by saying their type systems aren't perfect when your language's type system isn't helpful at all? Throw the baby out with the bath water?
To your specific point (which is also one of Hickey's), yes, it does kind of stink that loosening a requirement forces consumers to update their code. However, that minor downside does not mean that Optional is "not an improvement". It's still a HUGE improvement over Java's absurd handling of null (IIRC, Clojure is the same as Java there).
Also, maybe changing something to optional isn't really "loosening" the requirements. It's just changing the requirement. If the parameter changed to optional, don't you want to be alerted to that? Why is it optional now? What will it do if I pass None/null? Maybe I actually would prefer that to the way I called the old version.
It just never struck me as offensive to have to change my code when I upgrade a dependency. I have trouble sympathizing with that mindset.
Edit: And what is the Clojure alternative? You can loosen requirements, but really, you never had enforceable requirements anyway. Is it apples to apples to talk about a typed language loosening its contract?
> "Either types aren't true sum types because Either<A, B> isn't the same type as Either<B, A>."
that's such a weird argument! did he also complain that tuples aren't true product types because (A, B) isn't the same as (B, A) ? why would they be the same, and not just isomorphic?
I haven't watched the talk recently, but my feeling at the time was that he was just being pedantic about the definition of a sum type. Kotlin's nullable types would be example of true sum types because they are symmetric. But you can only make a sum of `T + null` and not a more generic `T + U`.
His real point, I believe, was that the `Either` implementations weren't as good as true sum types because of ergonomics. It's part of his philosophy/bias that type systems get in the way and therefore cause more harm than good.
I don't really grok his point most of the time. It just feels foreign to me to not want as strong a type system as possible. But a lot of really smart guys feel that way: Him, Alan Kay, etc. I suspect that they're able to track much more stuff in their heads at a time than I am.
The point is Hickey brings up important points about language design as its experienced by devs actually using the language. Hardly anyone discusses this. Furthermore, you seem to be making my argument for me when you claim that Clojure doesn't have types so why complain about types. In Clojure you could write a type system to do all that, probably in a a dozen hours (the language is programmable after all), but it would be an academic exercise to most which is the point Rich is trying to make when he disparages other type systems.
I think its worth noting that in Rust you can create a function that can take either Option or the value itself, if that is what you really want to do:
Yeah that's not good design. Lowering a requirement should not make callers complying to a stricter one have to change anything. But, ohhh.. right only the type changed. so everybody stop what you're doing and start over.
I strongly disagree, but this is why it's great we have a ton of languages! To me, forcing you to handle it is the exact point of an option type.
The type wasn't the only thing that changed, the possible values have changed. You may be getting more values than you were previously. This means that some of your assumptions may be incorrect. Of course your code needs to change.
I might be misunderstanding, but I think you are talking about slightly different points here. It seems to me that the critique of an explicit Option type (that acts sort of like a box, in contrast to Kotlin's T? vs T) applies to when you pass in the Option as a function parameter to a function that previously expected it to always be T instead of Option<T>. In that case you as a caller are never "getting more values than you were previously", but you can now certainly pass in more values than you could before.
Forcing callers to refactor their calls to use a new Option<T> type as a parameter simply amounts to a change in the type signature, but since the function is more liberal than before, it cannot break your assumptions (at least not assumptions based on the function type signature).
(For what it's worth, I do find Kotlin's T? to be more elegant than the Haskell/Rust-style Option/Some type. But then again, Kotlin is not fully sound, so there's that. Dart's implementation of T? will be fully sound though, so there are definitely examples of languages going that route.)
That is true! You're right that the perspective can be different.
You could write <T: Into<Option<i32>> if you wanted, and your callers wont change.
Frankly, using options as parameters is just not generally good design, so this issue doesn't really come up very often, in my experience. There are exceptions, but it's exceedingly rare.
Ah, I see where the misunderstanding is. You can make it so you change only the function signature and behaviour or you can make it so you have to also change the function call site.
Ever since https://github.com/rust-lang/rust/pull/34828 you can transform any `f` that takes a `T` into an `f` that takes an `Option<T>` without any of the call sites changing.
Your function `get_string_raw` which just handles `i32` can be transformed into a function `get_string` which handles `Option<i32>` without the thing calling changing how it calls the function. And the new `get_string` can accept `Some(i32)` or `None` or just `i32`.
Of course, this is slightly broad for brevity: you can now pass in anything that can become an `Option<i32>` but you can just define a trait to restrict that if you wanted.
You can get that sort of covariant effort that you wanted.
Well, yes, of course. The thing you could previously rely on being present can no longer be guaranteed to be present - that _should_ require code in the calling function to change.
To be clear, you're talking about the function signature changing from
fn(a: int)
to
fn(a: Option<int>)
?
Technically, yes, all callers would have to update, but practically, you'd just define
fn(a: int) { fn_2(Some(a)) }
to avoid the breakage. That is, you're essentially telling the compiler how to loosen the requirements. Ergonomically, this seems rather fine. Especially if this means you gain (some) protections from the much more problematic case of restricting the requirements.
How often do you believe this really happens in practice? And does that truly outweigh the benefit of being able to define a precise contract on your APIs?
How many times have you written a function and a version later said "Oh, wait. I guess I don't actually need that required Foo parameter! I used to, but now I don't!"
> How often do you believe this really happens in practice?
Regularly if you're doing refactoring of code. Otherwise code becomes unchangeable because it's too big of a burden once it's clear it needs to change.
> And does that truly outweigh the benefit of being able to define a precise contract on your APIs?
I would point you to the XML standards which allowed people to do exactly that, and instead JSON won.
Are we talking about a published library or your internal-only code? If the former, I sympathize with the argument that relaxing a requirement should not force consumers to change their code. If the latter, then I find it much harder to sympathize. You're already refactoring your code, what is a few more trivial syntactic changes? You could almost do it with `sed`.
> I would point you to the XML standards which allowed people to do exactly that, and instead JSON won.
You know- this is an interesting point. And I guess I'm consistent because I absolutely hate JSON. I've only had to work with XML APIs very few times, but every time, it was perfectly fine! I could test my output against a DTD spec automatically and see if I did it right. It was great. JSON has JSON Schema, but I haven't bumped into in the wild at all. So it seems like "we" have definitely chosen to reject precision for... easy to read, I guess?
You might really enjoy going and reading about CORBA and SOAP -- two protocols that have tight contracts. I'm sure you can still find java/javascript libs that will support both. And if you really really want, you can put them into production -- CORBA like it's 1999 while singing to the spice girls.
And what you'll find is that the tighter the contract, the more miserable the change you have to make when it changes. It's one thing if it's in one code base, it's another if it affects 10,000 systems.
I'll admit that I've never deployed a service with 10,000+ clients.
And CORBA (after looking it up) seems to include behavior (or allow it, anyway) in the messages. That's about much more than having a precise/tight contract on what you're sending. It's much more burdensome to ask someone to implement so much logic in order to communicate with you. I'm fine with the contracts only being about data messages.
SOAP is closer to what I'm talking to. Or even just regular REST with XML instead of JSON.
I'm asking genuinely, how would life be worse between a REST + XML and a REST + JSON implementation of some service? In either case, tightening a contract will cause clients to have to firm up their requests. In either case, loosening requirements (making a field optional, for example) would not require changes in clients, AFAIK.
The only difference that I see is that one can write JSON by hand. And that's fine for exploring an API via REPL, but you surely don't craft a bunch of `"{ \"foo\": 3 }"` in your code. You use libraries for both.
It just seems insane that we don't have basic stuff in JSON like "array of things that are the same shape".
> And CORBA (after looking it up) seems to include behavior (or allow it, anyway) in the messages. That's about much more than having a precise/tight contract on what you're sending.
The IDL (interface description language) for CORBA is a contract. It defines exactly what can or can't be done. It's effectively a DTD for a remote procedure call, including input and output data. (Yes it can do more than that, but realistically nobody ever used those features)
A WSDL for SOAP is similar. CORBA is basically a compressed "proprietary" bitstream. SOAP is XML at it's core with HTTP calls.
> I'm asking genuinely, how would life be worse between a REST + XML and a REST + JSON implementation of some service?
So REST+XML vs REST+JSON alone (no DTD/XSD/schema) would be very similar -- other than the typical XML vs JSON issues. (XML has two ways to encapsulate data -- inside tags as attributes and between tags. Also arrays in XML are just repeated tags. In JSON they are square brackets []).
But lets say you need to change the terms of that contract (new feature usually), will code changes be required on client systems?
* If you used a code generator in CORBA with IDL the answer is yes, there will be code changes required.
* If you used a WSDL and added a new HTTP endpoint, the answer was no. If you added a new field to an existing endpoint, the answer was yes. (See [2])
* If you used a DTD/XSD, the answer is usually yes, since new fields will fail DTD validation using an old DTD -- that is if you validate all your data upon receipt before you process it.
And this was fine for services that didn't change frequently or smallish deployments.
In large systems, schema version proliferation became a nightmare. Interop between systems became a pain of never ending schema updates and deployments, hoping that you weren't going to break client systems. And orchestrating deployments across systems were painful. Basically everything had to go down at once to update -- that's a problem for banks, say.
What's sad to me is that was well known back in 1975. [1] When SOAP was developed around 2000 they violated most aspects of this principle.
> but you surely don't craft a bunch of `"{ \"foo\": 3 }"` in your code. You use libraries for both.
In python, JSON+REST is:
resp = requests.post(url, data={"field":"value"})
What I find really appealing in REST+JSON is that validation just happens on the server side, and that's usually good enough. Sure there's swagger, but that's a doc to code against on the client side.
I don't feel that schemas and the need for tight contracts are all bad. I think if your data is very complex, a schema becomes more necessary than not when documents are bigger than a 1MB, say. I also think it's fine if your schema changes rarely. And yeah, if you need a schema for tight validation, JSON kinda sucks.
But that's the question, do you really need tight validation, and therefore coupling, or is server-side validation good enough? And in most cases people tend to agree with that.
> If you used a DTD/XSD, the answer is usually yes, since new fields will fail DTD validation using an old DTD -- that is if you validate all your data upon receipt before you process it.
I'm not sure I follow. DTD, as far as I know, allows both optional elements as well as attributes. If you add a feature, a client with the old version should continue to work correctly if you add optional elements. If they are NOT optional, then the client will fail regardless of whether you did XML+DTD or JSON, because your API needs that data and it simply wont be there.
What am I misunderstanding?
> What I find really appealing in REST+JSON is that validation just happens on the server side, and that's usually good enough. Sure there's swagger, but that's a doc to code against on the client side.
As a client, you don't have to validate your request before you send it. But it's nice (and probably preferable) that you can.
requests is not built-in to Python, right? So you are still using a library to JSONify your data. If you were to use urllib, then you'd have to take extra steps to put JSON in the body: https://stackoverflow.com/questions/3290522/urllib2-and-json
What's more, you still are not crafting the JSON yourself if you call json.dumps on a dictionary.
But, yes, crafting a dictionary with no typing or anything is still many fewer keystrokes than crafting an XML doc would be, even with an ergonomic library. But again, how much are you doing what you typed in your real code? That looks more like something I'd do at the REPL.
> If you add a feature, a client with the old version should continue to work correctly if you add optional elements. If they are NOT optional, then the client will fail regardless of whether you did XML+DTD or JSON, because your API needs that data and it simply wont be there.
Sure but, that begs the question, How is that better than JSON exactly? Maybe strong typing? And why isn't just sending a 400 Bad Request enough if the server fails validation?
I mean you could say well, "I know the data is valid before I sent it". But you still don't know if it works until you do some integration testing against the server -- something you'd have to do with JSON, anyway. XML is only about syntax, not semantics.
From what I've seen, XSD's tend to promote the use of complex structures, nested, repeating, special attributes and elements. And if you give a dev a feature, s/he will use it. "Sure, boss, we can keep 10 versions of 10 different messages for our API in one XSD" But should you?
JSON seems to do the opposite, it forces people to think in terms of data in terms of smaller chunks say. Yes you can make large JSON API's that hold tons of nested structures, but they get unwieldly quickly. And most devs would just break that up in different API's, since it's easier to test a few smaller messages than one large message.
> As a client, you don't have to validate your request before you send it. But it's nice (and probably preferable) that you can.
If you unit test your code, good unit tests serve as validation -- something you should be doing anyway. If you fail validation on your send, you have a bug anyway -- it's just you didn't get a 400 Bad Request message from the server. But to the user/dev, it's still a bug on the client side.
> requests is not built-in to Python, right?
Yes. But there's a lot of stuff not in the standard library that should be. The point is normal day to day code can be just a one-liner using native python data types.
> What's more, you still are not crafting the JSON yourself if you call json.dumps on a dictionary.
Sure, maybe a technicality here. If I type this, is it python or JSON?
{ "field": [ 1, 2, 3 ]}
Well, the answer is that both will parse it. json.dumps() just converts it to a string. No offense here, but I see it as a distinction without a difference.
