osquery is a cool project, with a lot of outstanding issues. It has a great deal of technical debt, including performance and security debts that don't receive adequate attention. It also has a huge user community around it, but only a handful of active recurring contributors and companies actually funding development on it (and, even then, the bulk of the development is feature work rather than debt burndown).
(Source: I worked on osquery's core and tables for 2-3 years for various clients, and my company employs several of the current maintainers.)
I don't know about production, but I use it on my MacOS laptop to list all listening sockets with the associated command line. Should I be concerned that this sort of thing has correctness bugs? (Can anyone give a competing non-osquery implementation on MacOS?)
sockets() {
(osqueryi --list --separator ',' | column -t -s ',') <<EOF
SELECT s.local_port, p.cmdline
FROM process_open_sockets AS s
INNER JOIN processes AS p
ON s.pid = p.pid
WHERE s.state = 'LISTEN'
ORDER BY p.cmdline;
EOF
}
There's at least one open data quality issue for `process_open_sockets` on macOS[1]. It's a few years old however and, if you aren't seeing that casting error, you probably aren't hitting it. But that's a good example of the kind of debt that's been built up over time.
(In terms of general purpose/flexible tooling, I'm not aware of a close replacement for osquery.)
Osquery: a privileged daemon written in an unsafe language that allows your junior sysadmins to take down all your machines at once or cause mystery performance blips that someone else has to diagnose for a year before they figure it out.
The senior sysadmins neglected to configure osquery’s client performance safeguards and apply properly scoped access (observer mode!) on the server.
Systems administration is the art of protecting everyone with less access from footguns while still and especially enabling their use of effective tools.
Feel your pain! This was the idea behind query performance "bubbles" in Fleet. (Uses the osquery performance metadata to indicate whether performance impact of a query on a host is "minimal", "considerable", or "excessive")
Next step is "clairvoyance", where we predict perf based on the SQL when you're in the query editor
OS state introspection can get expensive quick and often has a performance impact beyond the process doing the sampling, because you're locking tons of kernel structures to read from them.
Case in point, a simple "select * from processes" takes a solid 3 seconds of kernel time on my laptop.
Now you might say, "well that's clearly a dumb idea because osquery certainly relies on vtab's colUsed field to avoid querying all sorts of expensive stuff when it doesn't have to so you really should only query what you need" and that's of course 100% true. But it's also a senior developer thought. Easy to see how an inexperienced person might make mistakes like this with any one of the dozens or hundreds of tables offered by osquery and cause performance issues.
In terms of security, well it is clearly a kitchen sink project (there's a prometheus client in there, for example: https://osquery.io/schema/5.11.0/#prometheus_metrics), so there's a huge breadth of interfaces it talks to and files controlled by all sorts of people it parses, and the default does seem to be privileged usage, which is the general ballpark where AV engines and their highly dubious track record live.
Everyone who has experienced osquery in production when mandated from above and used by less qualified admins knew exactly what he was getting at and the simple truth of it.
Or they are a righteous developer who likes to trash people outside of development. It was a popular fad for a while, which worked both ways, such as, “Developers think they are experts in everything but couldn’t figure out how to plug in a computer.”
Definitely a tangent, but it's always interesting to see which of {a,an} people use for SQL and derivatives. It never occurred to me until today to pronounce "sqlite" as anything other than "sequelite".
When I have to say SQLite out loud, my brain decides in the moment between "ess-cue-lite" and "see-cue-lite." I have no idea what criteria/heuristics I use in that moment to decide which way to pronounce it, but for whatever reason that's how it works for me.
In my head I feel like I say "ess-cue-lite" pretty much every time.
huh, I get the "ess-cue-lite" but not (at all) the "see-cue-lite" -- for me, the "see" sound only makes sense in the context of "sequel" (see-kwell-ite), which is how I instinctively tend to pronounce it.
I think the former pronunciation is more correct, but...
I see the point of the later if you are prone to verbalize acronyms:
SQL => Verbalized Acronym: "See-Quell" vs Structured Query Language: "Ess-Queue-Ell"
So then See-Cue-lite is a concatenation of the verbalization with the "lite".
Whereas Ess-Cue-Lite is a concatenation of the acronym pronunciation with the "lite".
This probably solve the former question of why some people say both and don't understand why. If the conversation/person is saying "SeeQuel" and "Sass" or "less" etc. then "See-Cue-Lite" fits that dialect. In a dialog with a lot of acronyms e.g. SQS, VPN, AWS, etc. you are probably more likely to go the acronym route i.e. "Ess-Cue-Lite".
The term itself is a bastardization of an acronym and the "lite" term, I'd argue there's no correct pronunciation.
Can't it just be pronounced "sclite", rhyming with (e.g.) "sprite" ?
And when you apply FUSE (or FUSE-T or NFS or WebDAV) to its archive format "sqlar", getting a file system called "sqlarfs", then pronounce it as "sclarfs", rhyming with snarfs/scarfs/barfs.
Wouldn't that make SQL a sequel to Sequel? Sorry, I feel like I'm annoying on purpose, I shouldn't do this.
Reading up on the history parts in the Wikipedia article, the German language version presents it as if SQL was the Oracle (Relational Software Inc) implementation of IBM's Sequel, the English page says the name was shortened due to trademark issues, before mentioning any implementation outside IBM. That would make Sequel and SQL the same thing, just rebranded early on. It also lists a clever (b)acronym and two other reasons for why they called it sequel before shortening, so it seems like there's a decent amount of mystery around the name. Almost as if it happened before the invention of writing.
Interesting that is the case with an Oracle product. Even the name of the company is shrouded in myth. Apparently Larry worked on some government project called Oracle before founding the company, and liked the personalised acronym he could make of the word. But if that is considered by him to be an official acronym is (was) the subject of debate for quite some time.
But also, Microsoft's "SQL Server" is officially pronounced "Sequel Server", so a lot of devs who grew up working on Windows systems will pronounce "SQL" as "sequel". For them, it's not entirely wrong.
Huh, I thought osquery was a Facebook open source project. Surprised to see it under the Linux Foundation now. I wonder if Facebook still uses osquery in its current form or if they re-forked their own project to evolve separately in-house.
I don't know the exact dates, but Facebook turned osquery over to the community back in 2019 or 2020, after a period of uncertainly over (reasonable!) conflicting development interests[1]. It became a LF hosted project sometime after that.
(I don't know whether they still use it in-house.)
> Today we are excited to announce the creation of the osquery foundation. Please read the Linux Foundation's official announcement. We have created an osquery/foundation repository on GitHub to host the technical charter.
> The official repository has been renamed from facebook/osquery to osquery/osquery.
Not at all because the data sources are collected with scraping tech. I like your enthusiasm let me think of how to better scale the product. Is this something you’ve been interested in for a while?
Sqlite is embedded, so how do you actually build it into your app? Is there a C file somewhere, an amalgamation published? I haven't been able to find it, but surely it exists.
I'd like to build a Go binary with mattn/go-sqlite3 with osquery vtables included.
They did an amazing feat of engineering. Mike Arpaia's brainchild, co-created by my cofounder zwass at Fleet, and massively improved by seph (@directionless), ted reed, alessandro, sharvil, and many others: https://github.com/osquery/osquery/graphs/contributors
Have wondered for years whether Akamai's Query system (https://www.usenix.org/legacy/events/lisa10/tech/full_papers..., sadly never open sourced) had any influence on the creation of osquery. They are quite different architecturally but functionally similar.
Airwatch the Endpoint management system was working to incorporate this. It was pretty good. Unfortunately they were acquired by VMware which tried to maneuver the product into pretty unrelated VDI tech (Horizon). And of course VMware is completely in the shitcan now with everything being discontinued. We moved to Intune in the end. Which was not better by the way (nobody buys Microsoft because they're great, just because of the network effect). But it does have better long term outlook (no pun intended).
But it was a good product while it lasted. The osquery integration was really useful for custom scripts. AirWatch had linux support 4 years ago even though Intune is only starting with that (and it hardly works).
Deploy your own, stand-alone osquery instance - it’s open source.
Airwatch/Workspace One was a terrible product even before the acquisitions. You may want to try FleetDM, an MDM product that deeply integrates osquery (an intersecting set of people are responsible for the two).
I don't agree, I really liked airwatch and we managed tens of thousands of mobiles with it. It had its issues but every product does. The problem was that when we had to move to Intune some features we used in airwatch weren't even supported yet!
It really made me laugh when Gartner put Intune in their magical Quadrant but not airwatch as Intune wasn't even feature complete at that point for basic mobile uses. I'm sure those Gartner guys just talk to the sales suits but don't actually try the products.
But now we're stuck with Intune due to decisions made at top level.
Yeah the Mac side was the worst. Many features didn't work reliably or were not updated quickly enough to keep up with OS updates. It was all very beta unfortunately.
For Mac JAMF is pretty much the gold standard and I tried to get it but the leadership preferred a "single pane of glass" sadly.
For Windows we always just used SCCM and even now we're only in hybrid mode with most functions in traditional management.
I don't really know much about this area, but I did poke around Jamf one or twice and it seems absolutely awful. Not disagreeing that it might be the gold standard, but that doesn't really say much in the MDM industry.
For example: any single Jamf admin can type a bash script into a text box which will then be automatically executed on every machine as root.
As the other reply said, this is something you need to enable to only the most expert admins, and have processes to test it properly before you deploy it to the entire population.
FWIW our antimalware solution has the same capability.
But JAMF is the 'gold standard' because they are the only ones that really focus on Mac management. The used to be called Casper Suite, perhaps that rings more of a bell (I think JAMF is a pretty poor name but anyway)
There is a lot of innovation, with many viable options, in the Apple MDM market. Kandji, FleetDM, micromdm, etc. In my experience, Jamf is only the ‘gold standard’ when one has only cursory knowledge of the space.
I understand, I'm mainly talking from my enterprise background. From what I know the newer developments focus a lot on Apple specific implementations like DEP enrollment.
Unfortunately we can't use that because we can't use Apple federated accounts (they still require the UPN and email to be the same which we can't comply with) and because our IDP isn't supported.
This is the problem in enterprise, where you're often stuck with a relatively small amount of Macs (in our case less than half a percent) and thus the macs have to comply with the rest of the environment. Because the environment isn't going to be adapted to Apple's requirements. We're stuck with the requirement to limit user admin rights, to have security proxies (Zscaler), and many other things that don't work seamlessly on a Mac. Like the built-in password restrictions MDM profile, this is way too basic to fully encompass our security policy (it seems more like a mobile one shoehorned onto a Mac). So we need a lot of workarounds to meet our security policy.
For this JAMF works really well because it has lots of workarounds for enterprise setups that aren't done "the Apple way".
If you're free from legacy constraints the more modern solutions work great but in our environment we don't have that luxury, sadly. There's just too many strings attached from the security side.
You can sometimes script your way around the issues but every major macOS update will break things and JAMF is pretty good at figuring all this stuff out.
No approvals and no CI with clickops is indeed not a core tenet, but these are things you can implement on top of a solution like Jamf.
However, the market wasn’t always asking for it. Most Mac management is done by a sole individual at the org, usually not a large team where everyone can review everyone’s changes. This is steadily changing, but there are still tons of people who do clickops in Jamf because it’s what they understand and have the bandwidth to do.
We have almost a hundred thousand users and we don't have an approval process for macs either. Because we only have a few hundred :)
I think the windows guys have everything more proceduralised. But the Mac work is more of a one man show. I still don't think they have an approvals process though. They mainly still use SCCM and I don't think that has approvals built in.
And yeah things get tested in a separate environment before they're deployed in live. But as there's just one person doing both on Mac there's not much point to an approvals process. Which is also the person that gets to deal with any mistakes so there generally aren't any :)
And it's not a bad thing either, the Mac side is usually much quicker to adopt new features. Both because there's not that many and because Mac users are always interested in new OS versions whereas Windows users generally prefer to stay on what they know.
I recently implemented Terraform for a Jamf instance. In my experience macadmins are often much better at code driven workflows than Windows admins. MacOS is, after all, a real Unix. And Apple’s MDM protocol documentation is far superior (that’s why features are implemented quickly).
Jamf’s script feature is agent based, just like Airwatch’s, not an implementation of the MDM protocol.
> And Apple’s MDM protocol documentation is far superior (that’s why features are implemented quickly).
In terms of the overal tooling I have to disagree. SCCM is way more powerful than anythng Apple has to offer.
In terms of actual MDM "Modern Management" for Windows (Intune), yes that's only in its infancy but it's because most of their customers still use SCCM for most tasks. It's a bit chicken-and-egg.
But Apple's MDM is not great. The password profile is extremely simplified, not able to handle any complexity (example: In our AD passwords must contain special characters and numbers if they are shorter than 10 characters but don't need to when longer because we want to stimulate passphrases). Also, as far as I know (I don't work in this scope anymore) it still has no MDM profile to mandate the user installing updates in a timely manner. You can delay them but not force the user to install them. This stuff must be handled with scripting. The MDM app deployment is also very hit and miss which is why most MDMs do it through their own agent. It works fine when using the mac app store but most apps are not on there and usually there is a need for a customised package anyway.
And on the topic of customised packages, having to go through Apple's notarisation is really annoying. We should be able to just deploy our own signing keys to the machines that we own, and deploy to those machines whatever we want that's signed with our internal key without having to get Apple's OK on it. Sometimes the notarisation service refuses to work for some reason (happens especially with package installers combining code and signing keys from 2 different vendors) and I need to obfuscate the embedded packages to make it work.
So no, in terms of MDM I think Apple is not great for enterprise usecases. If you're a small all-apple shop and you can align everything with Apple's requirements then you may fare better but we don't. Less than a percent of our systems are macs.
> are often much better at code driven workflows than Windows admins.
Yes but Apple does shoot us in the foot sometimes by changing stuff around. I have to say that PowerShell is much more consistent in this manner.
I still prefer Mac but I have to say the enterprise management tooling is just way better on Windows. Apple doesn't really seem to care about enterprise users at all.
Another point is that terrible federated apple ID system that to this day still requires the UPN to be equal to the email address. In our environment this is different for a reason and there is no way it's going to get changed just to satisfy an Apple requirement.
I like the idea, but the queries seem to be longer than the shell commands, so the only advantage is not having to know the shell commands, and I'm not convinced you want people who don't know the system mucking about on the system.
This sounds like having a complete toolbox and saying "you know what would be cool? Using a drill to drive a nail into wood" and start banging the nail with it
(Source: I worked on osquery's core and tables for 2-3 years for various clients, and my company employs several of the current maintainers.)