> Because gun is not a database (NoDB), it is a persisted distributed cache.
This I believe.
> The fatal flaw with databases is that they assume some centralized authority. While this may be the case initially when you are small, it always ceases to be true when you become large enough that concurrency is unavoidable.
Partially true. Though that's not necessarily a "fatal flaw", and calling it such is troubling. Yes, concurrency is unavoidable when you become large enough but you also want your data to be, well, consistent and persistent, but then you go on...
> No amount of leader election and consensus algorithms can patch this without facing an unjustified amount of complexity. Gun resolves all this by biting the bullet - it solves the hard problems first, not last.
Where in the code, pray tell, is it solving these problems? The fact that you also claim to be an AP system (and conflate this with ACID) makes me strongly wonder what your notions on Consistency actually are.
"Just a cache" needs some consistency as well, I'll point out, but you may not care as much about stale reads.
> It gets data synchronization and conflict resolution right from the beginning, so it never has to rely on vulnerable leader election or consensus locking.
From what I'm starting to understand you're, at best, shuffling that off to S3 or "other storage engines" -- you've still got to pay the cost. You can't really claim to do linearizability without, well, actually doing linearizability.
So, maybe it's a cache, sure. And you seem to like to work on the developer API, nothing wrong there. But there's nothing new under the sun and I'm really skeptical that hard distributed database problems are solved in one large JS file.
Clarification, I said GUN is NOT acid compliant from your "usual" understanding of the term, since GUN is AP. Most people assume acid means CP.
ACID is very vague though, and I'd like to explore it more by writing tests to either confirm or deny whether GUN supports it or not (would you be interested in helping build those tests?). I also want to get some Jepsen like tests up as well.
Data convergence (data sync) is guaranteed by the Hypothetical Amnesia Machine algorithm, which is completely deterministic and idempotent. There is some details on it in the wiki, let me know if you have any questions. I also did a tech talk on it.
In NO way does gun rely on S3 for consistency. That would be horrible. Check out the algorithm and slam me with questions/critiques. Thanks for looking. :)
The problem with AP is that you completely lack consistency guarantees, and most people will agree that's a bad thing. That you advocate that notion so heavily, and without scholarly research makes me really, really nervous.
From what I'm seeing in the function you reference (with very few details in the wiki) there's no accounting for, well, `P` -- what happens when a message does not make it, which happens all the time.
What I see is sort of, if you squint, a vector sequence, but one that takes nothing about actual distributed systems (and their unreliability) into account. It also completely trusts no errors have happened in states it's receiving.
This all through some really messy code...
You also claim to not use clocks, but the first thing you do depends on a clock:
```
var serverState = Gun.time.is();
var incomingValue = Gun.is.soul(deltaValue) || deltaValue;
var currentValue = Gun.is.soul(current[field]) || current[field];
var state = HAM(serverState, ...
```
Where Gun.time.is() does in fact call current time
```
Util.time.is = function(t){ return t? t instanceof Date : (+new Date().getTime()) }
```
So one thing that will break you right away will be clock skew. That's a critique I get simply and for free. I also see nothing about guaranteeing that states applying to each other are actually what they expect to be. Which, you know, would depend on consistency....
The conflict resolution guarantees eventual consistency, but not strong consistency.
Why did I choose that route? Globally locking data is something the universe doesn't even do, we are bound by the laws of physics and can't get faster than the Speed of Light to transmit information. So you have to make a trade off.
Now, the cool thing is that you can use AP systems to creating globally locking/strong consistency behavior ontop. I do plan for there to be some plugins/modules that handle this for you, as long as you are aware you're moving into either a very potentially SLOW system, or a centralized one.
Also, CRDTs are something that are really important to look into. They're a good reason to try parting ways with CP systems, because they have some fantastic guarantees on data integrity.
We're working (as I previously mentioned) on getting academics and papers involved and written. I don't have that much funding yet, though as it is a very expensive and long winded process, but very important.
Next issue: LARGE DISCLAIMER: GUN is NOT relying on TIMESTAMPS alone to converge data. THE ACTUAL ALGORITHM is a state machine operating within a boundary function, the boundaries are defined by sort values, which I then use a combination of vector clocks and timestamps. But it is not timestamps alone, timestamps have massive vulnerabilities to them.
CLOCK SKEW will NOT break data sync. I've demonstrated syncing happen across machines with bad drift. But you are very right: I need to get docs and videos and evidence of this up ASAP.
Pardon for the caps, I just want people to skim and know for sure that I am not ignorant of these things. I'm not trying to sound angry. I'm super happy people care about these things.
So I tried it. I successfully lost data, and reported conflicting/inconsistent data, with a one-line change to your example to-do list. Instead of generating a key that won't conflict, I force it to write to the same key. Yes, this means it's one list item. That's fine, I just need to make this list item inconsistent between two machines.
And then I simulate a network partition (virtual machine network disconnect. pull the virtual plug. (EDIT: and repair it!))
This is enough to cause a seconds-long lag before the "okay" case comes through. And a local write during that lag? Completely lost. Two windows. Virtual machine. Game over.
This is because your data is overwriting itself, and I don't even care which version I got. I just needed it to not (locally) be the correct one, for any amount of time.
It effectively becomes "last write wins" and, well, there's nothing new under the sun.
Curiouser and curiouser, the "last write wins" happens in about five seconds. Which happens to be approx. the clock skew. (Called it.) There's some timestamp reliance for sure.
(I also got exactly the overwriting "always win" behavior you discussed in the other comment as being a flaw of vector clocks. From where I'm sitting, this smells like a pretty garden-variety vector clock.)
Thank you for trying it! You should also post this to the Issues on the repo. I'd like to get some more details though...
On Keys: keys are unique, meaning they can only point to one thing. When you "forced" it to write to the same key, you changed references, your previous data is still stored, but you'll only be able to find it by looking it up by its soul (its ID) or by scanning for it.
Were you experiencing something different than what I just mentioned above?
On Network Partition: The clients store their updates in LocalStorage before they send the peers the update or get an ACK. What caused those peers (tabs) to lose their data?
For clock skew, please see this reply: https://news.ycombinator.com/item?id=9077969 . Most "last write wins" algorithms require a centralized server (a Single Source of Truth) or they diverge on separate machines. That doesn't happen with GUN, you have no SST and they converge (even if it "looks" like last write wins) on both machines in an eventually consistent manner.
I'd like to follow up with these things, either here, over email mark@gunDB.io, on the gitter chat, or on github.
Is the Hypothetical Amnesia Machine algorithm backed by any scholarly research, or can you at least cite some papers with similar techniques? Blog posts and javascript are nice, but I have a certain fondness for LaTeX-generated PDFs whenever data integrity is involved, e.g. HyperDex's pretty excellent papers:
No papers yet, but I've been working on building up connection with academics and hiring them. So hopefully expect to see something published, but the process takes a while.
Meanwhile I'm actively working on building towards a simulation system and an actual high-scale deployable battle testing environment. Think of these as taking the theory from scholastic research, and actually implementing them in practical settings that anybody can run.
I'd also like to get a TLA+ specification going. If you have any experience in this stuff, please please please contact me mark@gunDB.io because this is important to me.
When I read your responses, I can‘t shake the feeling I'm talking to some snake oil salesman in a nice suit.¹
And to be honest, GUN‘s docs sound similar. Heavy on how to use, and how awesome everything is, but as soon as one tries to understand stuff, it‘s either WIP or “team up with me/us!”. eyebrow rises
And the claim of “building up connection with academics and hiring them” falls perfectly in line with this. Why the hell can‘t you describe what you did by yourself? If it's so awesome, why don‘t you just die to explain it to everyone who asks? Or, $DEITY forbid, should the “academics” lend some credibility to GUN, even if it's with just their title? What's this HAM about?
Maybe it's just me, but all this with a rather complex naming convention (souls...) and code...
eh, I'll go with “show us teh algoz”. Or describe it.
¹) To illustrate: « I'm actively working [...]» – I'd like to see you passively working.
No snake oil. It is a state machine operating over a boundary function. However words like that sound super jargony which sounds vague, despite the fact that people spend their entire lives working on just these problems sets and their nuances.
I'm happy to discuss the workings, and I'd encourage you to try and use GUN and see if it can withstand your concurrency attacks. Challenge accepted?
If people aren't interested in reading the materials I've provided that go over the concepts and algorithms, there is nothing I can do to "prove" anything to them, they just remain agnostic.
Did you read the link to the other comment and my reply? I'd really appreciate it if you could critique it.
You're happy to discuss the workings? How about writing them down somewhere...? All your documentation, such as it is, describes things using terms that you never actually define or explain. Your code is just as bad. Worse in fact, because it introduces yet further terms that are not in the documentation.
Your Conflict-Resolution-with-Guns page simply says 'see gun.HAM' for the explanation. No indication where this can be found. It isn't in the source repository and it isn't in the wiki. A google search reveals nothing.
The 'algorithm' presented on How-to-Create-GUN is meaningless because you don't define any of the return values. I can see how it maps some input values to some output values, but nowhere do you say what any of those return values actually mean, what I should do with them, or why they are useful.
e.g. return {amnesiaQuarantine: true} ... what does this mean? What should be done with that return value? What is an amnesiaQuarantine?
e.g. return {quarantineState: true} ... what does this mean? How does it differ from amnesiaQuarantine: true? What is a quarantineState? How should I react to receiving this return value?
Your documentation says a lot, but doesn't actually define anything, and is ultimately meaningless. This is why people are giving you a hard time and asking so many questions.
Most people reading will not know: what amnesiaQuarantine is, what amnesiState is, what the Hypothetical Amnesia Machine thought experiment is, what a boundary function is (there are multiple definitions - what are you using?), what 'converge: true means', what 'incoming: true' means, what state: true means (given that you say other 'state' variables are times* -- how the hell does a boolean represent a time, what 'you have not properly handled recursion through your data' means. What is a 'soul'? What happens to the data when particular values are stored? Where are things stored? What is the data flow? How are things shared? How does sync happen?
Imagine you don't know what any of your terminology means - like everybody reading your documentation. Treat each term like an undefined variable. Now try to understand your document. You can't. Those undefine terms are never 'set' anywhere. It doesn't make any sense. As soon as it gets close to actually explaining anything it just handwaves, or leaves you with undefined terminology.
You don't define what kind of persistence you implement or what consistency guarantees (worse: your explanations do not seem consistent). You don't define how your conflict resolution works (the 'explanation' given is tantamount to Star Trek technobabble). You don't define how data is transferred. Your slides are useless without any notes.
In your code you say that ACID is vague. It really isn't. Your explanation of how you meet ACID is extremely vague however, using what appear to be truisms and contradictions, and yet more undefined terms that seem to have little to do with anything mentioned in the documentation. Your code is poorly structured, and badly commented. It uses 'cool' sounding gun-related terminology ('shot', 'roulette', etc.) without defining what the hell those things mean. There is nothing in the code that actually seems to do anything with consistency
Your HAM algorithm - the very crux of your system as stated in your documentation, remains unexplained, and WORSE.. has a TODO: comment noting that it might not work and needs further investigation. This comment also mentions rollbacks.... yet nowhere else in the code or documentation says anything about rollbacks, and it's not clear why rollbacks would even be needed according to the (vague) explanation of HAM.
Your further explanations in these comments STILL do not actually describe precisely what HAM is or how it works. If you cannot do this in a simple and elegant manner, then NOBODY will be able to use or trust your database system.
If you want anybody to take you seriously, you must write a simple and concise explanation of HAM, including definitions of all your terms.
Frankly, it is so vague, and so unclear how it works that I am starting to think this is the product of some kind of mental illness...
Sorry to be so harsh, but nobody seems to be getting through to you.
EDIT: I'm reminded of Einstein's quote: "If you can't explain it simply, you don't understand it well enough."
Return values, with comments explaining their purpose, here: https://github.com/amark/gun/wiki/How-to-Create-GUN (I know you referenced this already, but did you read the comments explaining each return value? Edit: upon further reading your comment, it looks like you did, you just didn't like them. Perhaps I should make them more concise)
Persistence, currently S3 or localhost-testing-only disk. Persistence is a plugin.
ACID: Please link me to your favorite explanation of ACID that is clear and concise. I'll try and base my reply off that. I haven't found any good ones. GUN is AP, not CP.
People are taking me seriously, enough that I have contributors and funding. Some people don't take me seriously, and I'm trying hard to open up to them and be honest.
Do I need better documentation? Yes. Do I have documentation? At least some, yes.
You still have not described HAM except in the vaguest of terms. You have addressed very few of my questions.
What is the Hypothetical Amnesia Machine thought experiment? Where have you described this, or where can a description be found? What do the return values mean? The comments are very little help. What situations do they cover?
How does this relate to your algorithm? Please explain the algorithm in simple terms, with precise definitions.
Your slides provide NO USEFUL INFORMATION WHATSOEVER. If you cannot see that someone who doesn't already know what HAM is will be TOTALLY UNABLE to understand your slides, then you have a serious problem seeing things from another's point of view and should get somebody else to do your documentation for you.
Believe me, it's not from lack of trying on my part. I'm not stupid. I'm an experienced developer and familiar with the internal workings of many different database systems. It's my job and my hobby. I have maintained and contributed to several database systems. Your slides are intriguing but meaningless to me.
Your list of definitions ('Semantics') redefines many things that already have perfectly good definitions, and declares new terminology for concepts that already have perfectly good labels.
Many of the definitions are vague or even nonsensical/self-inconsistent.
For example: "soul': is the practically unique, immutable identifier for a node".
OK, so it's an identifier for a Node. So it's a Node ID. Why don't you just call it that?
But what does 'practically unique' mean? Something is either unique, or it isn't. It might be unique in a particular context, e.g. only in one instance of the database, or application, or server, ... or... what?
And what's a 'node'? "A group of no, one, some, or all fields, as they change over time." Well, you've redefined a perfectly good piece of jargon with a new and vague description. Node seems like a really bad word for this. In what way is a set of fields anything like a 'node' in the general sense? How does a node capture things over time? Is it a list, a history, an event log....?
"A group of no, one, some, or all" is better known as a 'set'. This is universally-accepted mathematical terminology. Except you've redefined that too.
And if something is a set of fields.... hey, how about calling it a field set? You know, like everybody else does...? Oh, no, let's call it a node instead....
My favourite: "Sent: proof that a message was received, might contain data that needs no receipt." The more you study this sentence, the more nonsensical and ambiguous it becomes. For a start, why not call it 'Received'? Or even 'Receipt', because that's the common noun for an item showing proof of receipt. Except, that you might need to prove receipt of data that needs no receipt... It is a ridiculous definition.
I'm sorry, but I can't take you seriously.
Frankly, it sounds like you yourself don't understand the domain and concepts you are describing, and are handwaving to cover your lack of knowledge. The fact that you provide your own terminology for things that could quite easily be described in standard terms betrays a lack of theoretical background, and ignorance of the state-of-the-art.
I'd venture a guess that your being REALLY, REALLY bad at explaining things may be correlated with the fact that you're apparently really good at describing tiny things in the most grandiose and self-aggrandizing terms. This seems to be ubiquitous across all your github projects. Redefining things unnecessarily, solving things that already have simple solutions, describing toy apps as radical revolutionary game-changers. I suspect your inability to explain things stems from this narcissism/egocentrism.
I am not quite sure where to begin. I read your wiki. The casual disregard for years of distributed computing research struck me as a bit scary. It would be fine if the website did not claim "Data integrity is now a breeze." but it does. It turns out data integrity in a distributed setting is actually quite a hard problem given the number of boundary cases that exist. For that reason experienced practitioners building distributed algorithms start with proofs, not try to come up with them later.
From reading your page on conflict resolution, which is quite light on details, it seems like you want to have a transaction log but unless the data is purely commutative that is impossible in an AP environment.
If you are going to reference a "Hypothetical Amnesia Machine" it would be helpful to at least define it. Looking at the code you hand someone two versions of their data and ask us to merge it.
It's nice that you are enthusiastic but it might be nice to build some proofs (or use someone elses) before making claims that our distributed state problems are solved.
Why can't a transaction log be commutative? You just have to make sure to be explicit about the ordering of events (like by using linked lists or progressively incremented hashes). This is the realm for CRDTs and stuff though, which GUN core doesn't touch.
Yes the HAM deals with merging any two logs/streams/history, any two snapshot/states, but also merging any log/stream/history with any snapshot/state.
This is important, because it allows you to merge more than just two, you just have to do it serially (in any order). That merge algorithm guarantees the deterministic resolution (see the comment link I posted above).
It relies on the data being commutative. If the data is not commutative then we need, wait for it, drum roll please, some consistency, which by your own admission is not provided. The consistency is the hard part which is why there is an entire field of study on just this problem.
I fail to see how your merge algorithm is deterministic in case of failures.
Maybe I wasn't clear... GUN does have Eventual consistency, but not strong/global consistency.
Aka GUN is AP and Eventually consistent. You manually at the application layer can decide to lock, sacrificing Availability, and get strong consistency.
The merge algorithm works in an Eventually consistent case, but obviously is too naive for global Consistency, you'd need some form of consensus.
I've demonstrated GUN before handling conflict resolution across machines with significant drift. I need to get a video of this and more docs out on it.
I'll be working on getting a recording of the tech talk up, more blogs/documentation on the algorithm specifically. And as others have mentioned, some actual academic papers (but that could be a while).
Yes, I've spent the last 4 years doing consulting work and research on them.
Warning: I have a very different approach though, more on the side of bittorrent and bitcoin, than what you are going to find in your traditional databases (CP, Master-Slave, Consensus/PAXOS/etc.).
If you are a distributed systems person also, I'd really like to talk. If you're armchair/backseat scoffing, then I would still love to talk show you how the algorithms work.
Sorry if you were offended by my question; I didn't intend any disrespect, but I hope you'll forgive me if I say that this HN submission is very confusing. The webpage claims that Gun is both the "easiest database ever" and "not a database" (interesting, then, that the website's URL is gundb). It claims that the problem with databases is that they assume there's a "centralized authority", which is sort of absurd; the overwhelming majority of businesses and organizations that use databases have at least some data that they need to absolutely 100% guarantee is safe and consistent, and distributed algorithms with leaders are the easiest way to capture that. Also, what is "vulnerable" about consensus algorithms? Does your distributed database really have no way to reach consensus?
Persistence is solved with "any S3 like service"? So what does that mean, using Gun is going to tie me to another unrelated Database as a Service that I'm going to have to pay Amazon for?
I'm sure this tool offers something interesting that other tools can't match, since you made it and put time into it, but the existing documentation isn't capturing that yet. Write up another blog post that describes the details, the use cases, the guarantees, etc.; i.e. the actual hard technical details, rather than the PR-speak, and I'll happily take a second look.
Yes, I got caught red-handed with my NoDB/gunDB marketing speak. Pretty embarrassing, but the point is that it is a distributed persisted cache, so you get the benefits of a DB without having to manage or maintain a DB.
You are right, most businesses that run that type of logic probably have the money to afford configuring master-slave based systems. They probably should not move over to GUN.
No, you do not have to use S3 for persistence, you can also use your disk. Persistence is a plugin in GUN, so you could build your own module that uses anything to store data.
However, there are lots of interesting advantages to distributed/decentralized master-master systems. And I'm trying to make those algorithms available to common man.
Thanks for the encouragement, I'll be adding more docs and blogs and stuff.
Yes, it is very similar. I'll actually be building some CRDT plugins ontop of GUN core. CRDTs usually deal with specific data types. Interested in helping?
> Because gun is not a database (NoDB), it is a persisted distributed cache.
This I believe.
> The fatal flaw with databases is that they assume some centralized authority. While this may be the case initially when you are small, it always ceases to be true when you become large enough that concurrency is unavoidable.
Partially true. Though that's not necessarily a "fatal flaw", and calling it such is troubling. Yes, concurrency is unavoidable when you become large enough but you also want your data to be, well, consistent and persistent, but then you go on...
> No amount of leader election and consensus algorithms can patch this without facing an unjustified amount of complexity. Gun resolves all this by biting the bullet - it solves the hard problems first, not last.
Where in the code, pray tell, is it solving these problems? The fact that you also claim to be an AP system (and conflate this with ACID) makes me strongly wonder what your notions on Consistency actually are.
"Just a cache" needs some consistency as well, I'll point out, but you may not care as much about stale reads.
> It gets data synchronization and conflict resolution right from the beginning, so it never has to rely on vulnerable leader election or consensus locking.
From what I'm starting to understand you're, at best, shuffling that off to S3 or "other storage engines" -- you've still got to pay the cost. You can't really claim to do linearizability without, well, actually doing linearizability.
So, maybe it's a cache, sure. And you seem to like to work on the developer API, nothing wrong there. But there's nothing new under the sun and I'm really skeptical that hard distributed database problems are solved in one large JS file.