Am I reading this right that it's a binary format for real-time streaming data, similar to Avro, but can include arbitrarily deep nested structures unlike Avro?
Ion has a binary format but is not specifically about real-time streaming. It is a JSON replacement.
Ion originated 10+ years ago from the Amazon catalog team - the team that kept data about the hundreds of millions of items available on Amazon. Nearly every team in the company called the catalog to get information about items all the time - scanning the entire catalog, parts of the catalog, millions of individual item lookups every second, etc.
They did the math and some very large percentage of network traffic in Amazon Retails data centers was catalog data. If that data, currently in XML or JSON format, was sent in a more compact format it would save some ridiculous millions of dollars every year. So Ion was born and eventually open sourced.
Why do you single out avro & not any of the hundreds of other ser/de systems? Is that what you know best? Is there something specific about avro that makes it feel particularly similar? https://github.com/maximveksler/awesome-serialization
People keep inventing new ones because the old ones suck or they think the old ones suck. Look at all the discontents around JSON (no comments!), people react violently when people try to apply a little extra like JSON-LD. Then there are all the things like YAML, TOML and such that try to be a little better but are widely thought to be a little worse. (And that's just the human readable data formats) Then there is always
I work in an a shop where Kafka and Avro are everywhere. If I worked someone else I might make reference to something else if it was front-of-mind all the time.
