In another decade or so the world might replicate half of the very nice internal tools Google has.
Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple aggregations. That was pretty handy. I really wish Google would open source some or most of this stuff. It’s not like keeping it closed source creates any kind of insurmountable competitive advantage, especially compared to the advantages that would accrue from broader adoption of protobufs.
Other tools and features that don't exist outside:
- a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should.
- load balancing dashboards showing traffic between frontends and backends
- Kubernetes: server-specific custom hyperlinks on dashboards (e.g. links to POD_IP:PORT/stats, /debug, etc. for each individual pod you are looking at)
- Kubernetes: multiple Docker images in the same container or pod. E.g. the first container could be your code, while the second one might be data or the JVM runtime, etc., without having to bundle them together or doing costly copies in init containers.
> - a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A. I don't think Envoy has this, but it should.
Great! I suspected it had to be in there, but it was hard to figure that through the docs. I didn't think of 'shadow' as a keyword, only 'mirror'. Thanks. One more reason to adopt Envoy and Istio.
Yeah, the docs are definitely an area of Envoy that could be improved. All the information is there, somewhere, but finding what you're looking for can be quite challenging.
That's an external controller, which is what most people are doing themselves these days, reinventing the wheel each time. Borg has long had an automatic rollback feature on updates, tuned through a few settings on top of the health check machinery. I'm in the camp believing that a basic implementation should be built-in, since health checks are already there. An implementation of this was started, but it has stalled. External controllers should be still allowed, for more advanced scenarios like rolling back upon detection of regressions in latency and similar.
Also, setting up Spinnaker is pretty much as complicated as Kubernetes itself. :-)
How does Borg express updates? With Kubernetes only deployments have rollout/rollback, and you need something like Helm -- which is frankly not that great -- to handle groups of resources that are to be updated and versioned together, e.g. configmaps and services.
AFAIR, you can only update jobs (collections of single containers) and allocs (pods inside of which jobs will schedule and run: you need to state that job J lives inside alloc A, otherwise it will run in an anonymous, implied alloc). There aren't other objects you can manipulate, actually.
Configmaps don't really exist, although something similar is achieved with a job that has a second package holding just the data. This is why I think multi-image containers should be implemented, but also a reason why they haven't been yet: configmaps cover some use cases. When a job replica (task) gets updated or rolled back, both packages change in sync. On Kubernetes, you'd use version numbers in the configmap name (but you need to worry about garbage collecting unused ones).
Services live outside of Borg entirely. GSLB has its own push mechanisms and only consumes Borg's lists of containers that comprise a given Borg job.
From the paper:
> A user can change the properties of some or all of the
tasks in a running job by pushing a new job configuration
to Borg, and then instructing Borg to update the tasks to
the new specification. This acts as a lightweight, non-atomic
transaction that can easily be undone until it is closed (com-
mitted). Updates are generally done in a rolling fashion, and
a limit can be imposed on the number of task disruptions
(reschedules or preemptions) an update causes; any changes
that would cause more disruptions are skipped.
Some task updates (e.g., pushing a new binary) will al-
ways require the task to be restarted; some (e.g., increasing
resource requirements or changing constraints) might make
the task no longer fit on the machine, and cause it to be
stopped and rescheduled; and some (e.g., changing priority)
can always be done without restarting or moving the task.
> a tee loadbalancer for gRPC, forwarding the same requests to both A and B backend pools, but only returning results from A
I would call that a "(live) traffic replayer" rather than a load balancer. "load balance" implies to me that the upstream traffic is divvied up among the downstream sinks, not that the upstream traffic gets copied to multiple downstreams.
It's a load balancer, because, typically, there is more traffic than a single server can handle. That's why I mention pools: it sends traffic to one of {A1, A2, ...} and one of {B1, B2, ...}. Using your terminology, that instance of the load balancer will take the upstream traffic and divvy it up twice, among two separate groups of downstream pools. A load balancing tee proxy/LB is also useful where B is a new implementation that is supposed to handle more traffic. You could e.g. bring up 10 A replicas and 5 B replicas and compare their performance metrics. That setup is closer to reality than one instance of A and one of B (if you want to test soft stickiness, for example).
Anything at Google that does not support load balancing is doomed to melt fairly quickly.
Image based volumes (second last to bullet) has long been blocked on the container runtime having a really clean way to enable and keep the container filesystems mounted. Definitely something I want to see fixed since otherwise you just end up doing hacky copies via emptydir.
Most of this is public (see the Slicer paper for dynamic sharding) or implemented in some other form by Envoy and other tools (dark launches, traffic visualization) or not that complicated (Kubernetes has manual rollbacks already and automating them is a natural extension). None of those are Spanner territory. Even LOAS is getting discussed and open sourced.
When I was at Google, I kept an eye on the open sourcing of RecordIO. Apparently there was no desire not to open source it: it was simply that nobody had the time to disentangle and/or clean it up for release.
I think the decision not to open-source RecordIO is likely related to legacy baggage that's baked into the format. The LevelDB format above avoids that.
It doesn't appear that the headers for this are public though.
Pretty neat work. Especially the corruption detection/skipping and seek support. That part was super ugly in RecordIO proper, and relied on stars properly aligning and the absence of cosmic radiation. RecordIO being the default format for everything at Google, it's not something they can really fix though. Taking care of concatenation is a nice touch as well.
As someone who has spent quite a bit of time at Google working on a high performance file format (not RecordIO):
1. I'd also add LZ4 and/or Snappy for the cases where they are more Pareto-optimal (i.e. fast, network attached, remote storage, such as SSD Colossus, or its external proxy: SSD Persistent Disk).
2. IMO HighwayHash is overkill here, and the author should have used CRC32C instead. You don't particularly care about collisions in this case, you're detecting data corruption. CRC32C is perfect for that, and it's hardware accelerated in almost all recent Intel and ARM CPUs, and it's half the size on disk.
3. It'd be pretty cool to introduce some kind of metadata which would tell the user what type of message is encoded in the file. This is not something RecordIO has, but internal tools can guess most of the time because they have all the proto definitions at their disposal. There's no need to store it in every header, just the first one. I would advise against storing the full schema (that can get very gnarly in the presence of proto dependencies and extensions), but just have something lightweight, i.e. message name and perhaps SCM revision number or hash in the file header, so that the user (or the external system consuming the files) could somewhat reliably establish what the format is later on, when the proto definition drifts. Otherwise, this being a binary serialized file format, it's very easy to end up in a situation where you have some files from years ago and you no longer know how to read them. And yes, I'm aware that SCM hash can change if history is edited.
Interesting. I wonder how different that is from RecordIO. Also, whether there'll be a Go implementation.
[Edit, after looking a bit.]
Pretty different. If I remember correctly, RecordIO is re-synchronizing, whereas Riegeli seems to break things up into 64KB chunks, splitting messages across chunks if necessary.
IIRC (but memory has faded considerably), RecordIO also did support something to aid compression across records (rather than just offer per-record compression). There was some gnarly code in it to that effect where there could be a compressed subset of several records within the file. But I might be wrong.
Nah. I’m interested in quickly querying on-disk data specifically, ie proto-based application logs and the like (another thing the world needs to adopt more broadly imo).
Of note, Prototool has a binary-to-json command, so assuming your Protobuf files are in path/to/proto/files, and you had a newline-separated log file of Protobuf messages foo.bar.Baz, you could do:
Why not use SQLite[1] for storing this data?
Storing structured data in binary format, and being able to run SQL queries on it, is already possible with SQLite right?
While not exactly what you are describing, I work for another company that uses protobufs extensively and we have some nice internal tools similar to what you describe. I really wish we could open source those too. I feel like the wheel is reinvented a lot with protobuf in several of the large companies who use it.
I‘m smelling the SQL case could be reasonably easily thrown together with PostgreSQL and a custom Foreign Data Wrapper based on protobuf-c (prior art: cstore_fdw by the Citus folks). Proto definitions then should compile rather cleanly to table definitions, at least one level down (PG isn‘t so good with nested structures).
The main thing stopping this endeavour is probably that to the best of my knowledge, there isn‘t any standardization in the Protobuf community about file formats serializing multiple of these together like RecordIO - that, and my C skills are pretty rusty by now :)
Nope. Protos have repeated fields and can be hierarchical (that is, can contain other protos) and even recursive (that is, contain themselves, possibly as repeated fields). H2 is not going to work.
Don’t know about Arrow, but Parquet is a columnar format. Such formats can’t write record-by-record, they need a large number of records to shred into columns in order to realize their columnar benefits. In contrast, appending to RecordIO is little more than writing a binary string. The downside of RecordIO is that you can’t just read some fields in a message and not others. You have to deserialize the whole message. RecordIO is cheap to write and well suited for cases where reading the entire message is not that big a deal. Columnar formats are more suited for the cases where it’s ok to pay the relatively substantial up front encoding cost for vastly greater performance in analytical workloads. Advanced ones contain additional metadata (such as range and hash constraints, the former can be both per file and per block) which the analytical runtime will be able to take advantage of in order to avoid doing the work that doesn’t need to be done.
We built https://github.com/GyroscopeHQ/grpcat at my company, which takes text-format protos as input and sends them to a gRPC endpoint. Looking at Prototool I think I should just merge the functionality into Prototool. This is cool!
Says "Handle installation of protoc [...] behind the scenes in a platform-independent manner without any work on the part of the user", doesn't support Windows yet [0]. Granted, as pre-1.0 I should probably read the features as goals.
danby - grpc for the browser :: is looking for testers https://github.com/ericbets/danby
There are two upcoming features. The first one is streaming support. The second is a callback API template that mirrors the grpc node API exactly. Or you will have the choice to stick with the current promise API. It's not a priority for us at the moment but adding a simple load balancer that distributed traffic randomly across a set of servers would be a ~5 line patch.
this seems like it's only relating to people wishing to contribute to prototool. also, it's uber, so it's nice/expected that they would have this sort of thing.
also the code is basically about not being a jerk to other people. seems like a low bar to meet.
Why is a document describing the standards for social interaction that contributors pledge to live up to bad? Open source has been and continues to be rife with social interactions that are bad for individual contributors and for the project as a whole. CoCs strive to head that off and document a process for when people seem to violate their pledge.
> You don't need to tell people to be polite (and you shouldn't). If someone isn't everybody will notice it.
Unless it's done in private. If someone is a jerk to me in private (over email or Slack or whatever), what do I do?
If I'm contributing to a project without a CoC and someone is a jerk to me, I'm much more likely to just contribute to a different project, or start my own fork, or just stop contributing to open source entirely.
Projects with CoCs have a stated procedure in place for dealing with abusive contributors. That's what the whole thing is about. Of course you shouldn't have to tell people to be polite, but that's not how the real world works. There are jerks everywhere, and more often than not they're going to express that in private.
I always reply to offensive people. Someone who is offensive has no arguments. Saying him he is offensive is often enough. If there is a public (like here) people will understand who is insulting who.
You're looking for a way to be less implicated. It's because you're accustomed to live in systems based on laws and rules.
People aren't bad. Most are kind and they don't want to harm anonybody. The bad guys are very few (like 1% or 0.1%) and as they are few they should be handled case by case. Building a complicated system of rules for them will just bother the "not-bad" majority.
From the position of a contributor, I understand and agree with you. You shouldn't contribute to a project whose terms you do not agree with, and I think that the "CoC" movement helps surface the type of organizational policies that a potential first time contributor would want to know before committing to the project.
I am curious to understand how it contributes to your decision not to integrate the tool into your workflow. I don't typically choose products based on the politics of the company that create them, which is the closest analogy I can come up with.
Suggestion for a project: make a tool that, given a proto description and a file that contains concatenated proto messages stored as binary strings (sort of like RecordIO at Google) lets you run simple SQL queries on the data and extract a subset of the fields from messages matching a predicate, and maybe even do simple aggregations. That was pretty handy. I really wish Google would open source some or most of this stuff. It’s not like keeping it closed source creates any kind of insurmountable competitive advantage, especially compared to the advantages that would accrue from broader adoption of protobufs.