Letting ancient evil code run? Have we learned nothing from A Fire Upon the Deep?!

Link to the Prologue of Fire Upon the Deep: https://www.baen.com/Chapters/-0812515285/A_Fire_Upon_the_De...

It's very short and from one of my favorite books. Increasingly relevant.

I swear, I respect Vinge more and more based on how well he seems to understand human tendencies to plot some plausible trajectories for our civilization.

There's a little throwaway thing in the book (or maybe it was in the prequel) that I always liked, re understanding human tendencies. They're still using Unix time, starting in Jan 1st 1970, but given that their culture is so space-travel-focused they assume the early humans set it to coincide with man's first trip to the moon.

That's from the prequel, A Deepness in the Sky. (Which is also excellent.)

Deepness in the Sky is probably the first Sci Fi alien I read who didn't feel like a human wearing an alien suit.

Fantasy sometimes does this better but usually with specific tropes.

If you liked that and you haven't read it yet, give "Dragon's Egg" by Robert L. Forward a read.

I wish he could have seen the current state of GenAI. Several times in the book he talks about how the ship understands context clues and sarcasm, and that effective natural language translation requires near-sentience.

"It was really just humans playing with an old library. It should be safe, using their own automation, clean and benign.

This library wasn't a living creature, or even possessed of automation (which here might mean something more, far more, than human)."

＼(＾O＾)／ zones of thought mentioned ＼(＾O＾)／

Do you remember the part where they built a machine in the Transcend that had to work at the Bottom of the Beyond?

The other day I was using Claude for a task, but it occurred to me, what if Claude is unreachable.

So, I told it to "encode your wisdom into this script in case you are not available"

That was my own version of that

Legitimately listening to this book for the first time after a coworker recommended it. It's rapidly becoming one of my favorite books that balances the truly alien with the familiar just right.

Not so ironically, it came up when we were discussing "software archeology".

Learning from fiction? Let's learn from the Dune then and start Butlerian jihad already.

I've only just heard of it. But, I already knew to not run random scripts under a privileged account. And thank you for the book suggestion - I'm into those kinds of tales.

I love that book

Army of Darkness?

The Mummy?

Are you sure? Are you $150 million ARR sure? Are you $150 million ARR, you'd really like to keep your job, you're not going to accidentally leave a hole or blow up something else, sure?

I agree, mostly, but I'm also really glad I don't have to put out this fire. Cheering them on from the sidelines, though!

Honestly, since I'm never really in a position to see much of that money, at this point I'd be more concerned about my coworkers. And while that typically correlates with the amount of money you either have or receive, they're often out of balance one way or the other.

True but it does say something that such a script was able to lie dormant for so long.

Why would anyone test in production???!!!

Selecting the wrong environment in your test setup by mistake?

I refuse to believe that someone on the security team intentionally tested random user scripts in production on purpose.

Once you get big enough… there comes a point where you need to run some code and learn what happens when 100 million people hitting it at once looks like. At that scale, “1 in a million class bugs/race conditions” literally happen every day. You can’t do that on every PR, so you ship it and prepare to roll back if anything even starts to look fishy. Maybe even just roll it out gradually.

At least, that’s how it worked at literally every big company I worked at so far. The only reason to hold it back is during testing/review. Once enough humans look at it, you release and watch metrics like a hawk.

And yeah, many features were released this way, often gated behind feature flags to control roll out. When I refactored our email system that sent over a billion notifications a month, it was nerve wracking. You can’t unsend an email and it would likely be hundreds of millions sent before we noticed a problem at scale.

Yes this is a common release practice.

However this is a different situation as we’re talking about running arbitrarily found third-party scripts. I can’t imagine that was ever intended to be done in production.

Fun story, when I worked at Facebook in the earlier days someone accidentally made a change that effectively set the release flags for every single feature to be live on production. That was a day… we had to completely wipe out memcached to stop the broken features and then the database was hammered to all hell.

I would say you can get to this point far below 100 million people, especially on web. Some people are truly special and have some kind of setup you just can't easily reproduce. But I agree, you do really have to be confident in your ability to control rollout / blast radius, monitor and revert if needed.

> I refuse to believe that someone on the security team intentionally tested random user scripts in production on purpose.

Do I have a bridge to sell you, oh boy

I have never heard of this kind of insane behaviour before.

There are plenty of ways to safely test in production. For one thing you need to limit the scope of your changes.

"Everyone has a test environment. Some are lucky enough to have a separate production environment."

Or just restore from backup across the board. Assuming they do their backups well this shouldn't be too hard (especially since its currently in Read Only mode which means no new updates)