nosh is, by far, the single most masterfully architected and carefully researched toolkit for init, process management, supervision (also with auxiliary utilities for user mode VTs and UCSPI) out there at present, trailing only with s6. The FreeBSD community will do itself a great favor by adopting it over the NextBSD experiment.
It is, indeed, daemontools++ with lots of optional compatibility layers, carefully planned module boundaries and a small chain loading language for composing execution state.
That said, compared to what is standard fare in DevOps, it will likely be found too esoteric. There is also a tad more to be explored in reexamining this controversial problem domain, which I hope to unveil in the coming months.
Active in this industry for several decades, and I've yet to hear a convincing argument why kitchen-sink init systems are somehow an improvement over rc.d.
There are race conditions in rc.d. Like most concurrency bugs, they are rare, but we shouldn't ignore them.
Consider what happens if you start a process, and then stop it before its pidfile gets written. Or, if you start a process that takes a while to initialize, and during that time you can't kill it normally. Now imagine combining these situations with auto-restarting daemons that crash, or shutting down a computer immediately after it boots.
The right thing to do in these situations is to block until the daemon gets into its normal run state, when its PID is known and you can kill it safely. rc.d init systems don't do this, they just fire off a signal to a PID and hope for the best. It doesn't always work and on top of that, it's easy to misconfigure something, like a pidfile path or permissions.
> There are race conditions in rc.d. Like most concurrency bugs, they are rare, but we shouldn't ignore them.
Doesn't systemd have roughly the same problem, with non-deterministic boot order and dependency heuristics resulting in opaque, hard-to-debug ordering issues that appear only intermittently?
Any parallel init system will have non-deterministic boot order. This is very different from race conditions. Non-deterministic boot order makes bugs in your dependency system very hard to track down.
Some parallel build tools have options to do a non-parallel build, but randomize the order of dependencies. This can help track down failures due to improper dependencies regardless of parallelism available.
I wish the various rc.d/* scripts can support real dependency, i.e. I can optionally designate S50 will not start running until S49 is completed, this is useful sometimes.
Actually I bootstrapped all of my init systems with OpenRC; it has tools for dependency calculation and all runscript based init scripts can be run with -D so that you can handle dependencies rather than it; I gradually replaced the existing init scripts with something a bit more amenable to the non-reparenting style I prefer, but it's very useful for getting started.
Thanks, but I don't follow this. Saving a few keystrokes here and there, and process monitoring (which can be provided for by plenty of other packages ) is sufficient cause to reinvent init?
Even if djb's /service claims makes sense, it still doesn't support the significantly expanded scope of, say, systemd. I mean, hijacking system logging? What could possibly justify doing that in an init system?
> I mean, hijacking system logging? What could possibly justify doing that in an init system?
DBJ's /service also hijacks logging. In fact, every decent init system hijacks logging -- you want your init to daemonize processes, and when you daemonize a processes you want its stdout to get logged properly.
Fifteen years ago there weren't plenty of other packages that offered process monitoring: There was init (inittab), /etc/ttys, and roll your own (init.d aka rc.d).
People didn't use inittab or /etc/ttys because telling programmers to edit these files was error prone and if they got it wrong, the system wouldn't boot up correctly.
init.d didn't have that problem, but programmers so frequently and regularly and consistently got "init.d" wrong in some other way (supervision, logging, etc) that some time in the last fifteen years most unix sysadmins stopped expecting programmers to daemonize and write pid files and simply started monitoring their services for them with tools like daemontools.
systemd is not really an init system project. Its a project to make GNU/Linux systems work better. Logging is one of the things I think they have tackled, along with service startup ("init").
I'd argue that logging is better and more useful with systemd as compared to with typical sysvinit (say in Debian/Ubuntu).
nosh does not solve every problem that launchd, mach, and libdispatch does, which is why launchd will still exist in the BSD world through Jordan, iXSystems, and NextBSD.
bringing in launchd provides many more features, but is also the stepping stone to:
* networkd
* configuration files that are never directly edited by humans
* Never having to HUP a process again
* The proper death of cron
* Processes never being stuck in the wrong state just because something
in the system changed, and being able to take intelligent action
based on state changes
nosh probably can get into FreeBSD faster, but really -- whoever can replace rc scripts first wins. nosh is far with the base system rc scripts, but ports... oh, ports....
I would argue the validity of some of these statements.
why does my init system need anything to do with my networking, they're completely separate systems in my mind.
Cron has a use, it's not like it's overused or underused, it has a niche and it fills it well, what is your argument that it's bad and needs to go into init?
Config files will always be hand edited, wether you facilitate that or punish that is up to you, most tools that exist for managing config files exist based on the premise that they can be changed by hand and that you "probably didn't want that" (a-la config management) but having hand-editable config files for testing things is a good thing.
HUP'ing a process hasn't been an "issue" for some time, initscripts tend to wrap HUP with a "reload" command, and I'd like people to keep signal handling inside their programs.. since that's well understood as a standard and is POSIX compliant.
Now, I won't argue that POSIX is the best thing ever, but unless there is a new better standard to supersede it comfortably, I am not in favour of adopting radical ideologies "because it seems like a good idea" right now, and, evidently, it doesn't anyway.
This is exactly the counter-argument that Jordan & co constantly have to battle.
"I've never had this problem!"
Networking: not what I said
Cron: Cron is terrible. Paul Vixie says so himself. Get over it. (everyone, not directed at you)
Config files: you're inferring the wrong thing
HUP: I shouldn't have to HUP a process to make it learn new things. It should be able to learn them automatically by listening for an event.
Please, take a listen of exactly the scenarios Jordan lays out in the latest BSDNow podcast. It makes sense. If we want to be able to build really cool reliablenon-kludgey things in the future we need these kinds of features.
If your Unix is a server that never moves or changes this is going to sound uninteresting and overkill. But I want the future to allow us to be able to do really awesome stuff. I envision being able to live migrate servers from my basement to Amazon to Vultr to Digital Ocean and not have a single process puke because of changes (network interfaces, IPs, hostnames, etc)
Parent was asking for (among other things) reasons why cron needs to die; saying Vixie ipse dixit is true, but not a technical argument. Notably, it doesn't help the reader understand what the replacement ought to look like.
Disk intensive daily jobs shouldn't run if the system is being crushed. I don't want to update my locate database during a backup or during a zpool resilver.
If I pause/suspend a {VM, server, laptop} and wake it, jobs are missed ... because it's not time to run them. What if I'm live migrating a VM and there's an important job that needs to run during the time it momentarily freezes the VM for migration? VM resumes on new cluster, job wasn't run. Sure if you're doing this once by hand you can be smart about making sure the timing of migration is safe and the job actually runs. But what if you're running thousands of servers? Ok, so now we throw a ton of code at the wall customized to our environment and hope we don't miss any edge cases. Or we can stop reinventing the wheel and use a job scheduling system that solves this problem.
Have you had the fun of cron jobs getting stuck and seeing them just endlessly build up, each one starting the job over from scratch, I/O grinding to a halt as the server gasps for breath under their weight? Alright, lesson learned, customers were angry, now we know we should wrap the job with a lockfile (and hope it's a proper BSD lockf(1) which guarantees atomicity) ... or you can have an intelligent scheduling system that knows not to do such a thing.
The possibilities are endless when you feed intelligence into a cron replacement.
there is a lead programmer where I work, a fiercely intelligent man.. the kind of person you want to be an architect of a project.. the kind of man that invents solid things _ALWAYS_ says.
"Do a thing, do it well, do only that thing; if you build things stupid and predictable they can be robust and reliable"
He's a C++ Developer working on windows. Not some GNU nut.
intelligence is fine, but it's hard to code, does unpredictable things and becomes a burden.
also, you can turn Cron off, it's not like it's a required system, it doesn't even come enabled by default for desktop distros (such as would be used on a laptop)
I mentioned it because I am a GNU nut, not intended as a smear. But people like to shut down your "argument" when you start saying things that sound like the unix philosophy.
> Disk intensive daily jobs shouldn't run if the system is being crushed.
ionice(1) works really well on Linux.
> Have you had the fun of cron jobs getting stuck and seeing them just endlessly build up...
It sounds like this would be fairly easy to fix. Either set a timeout for each cron job, or make a mark when each job completes and don't start a second copy of that job if the mark is not found.
Regardless, I don't see how these are reasons that cron needs to die. They are decent reasons for improving cron. ;)
Man, its functionality probably should be folded into nice(1) and renice(1). I'm kinda on the fence about whether or not ionice should also be folded into nice/renice.
The trick is determining what thing you're going to do.
I think it's perfectly reasonable for a job scheduler to have the ability to terminate jobs that run for too long and to ensure that no more than N copies of a given job are running simultaneously. This sort of thing also satisfies the DRY principle. :)
People really like to parrot this statement. Nobody is going to make it do "more" than a scheduling tool. It's not going to become a volume manager or start steering your clock...
Why aren't you angry about these redundancies:
* sort's -u flag when there's uniq ?
* grep's ability to search recursively when there's find ?
* grep's existence when there's ed ?
* cron's existence when there's at ?
* cat's existence when there's < ?
many of those things dont actually do the same thing.
at runs a scheduled job once and then terminates, cron runs scheduled jobs repeatedly.
ed/sed are powerful languages that are used for 'changing' data, not searching it.. it _can_ be used to search but it's very heavy for the task.
grep also searches content of files recursively, where as find does not, having find do it means forking the process and being very heavy also.
cat and < are different in that one is a shell builtin and is used to.. concatonate files.. which the shell builtins do not do... ofc many people "use it wrong" but that's neither here nor there in this discussion. https://en.wikipedia.org/wiki/Cat_(Unix)#Useless_use_of_cat
> at runs a scheduled job once and then terminates, cron runs scheduled jobs repeatedly.
Make recurring scheduled jobs reschedule themselves as their last action. This makes your job scheduler do one thing and do it well. :P
> [ed] _can_ be used to search but it's very heavy for the task.
So? ed was around before grep. Grep is therefore redundant.
> grep also searches content of files recursively, where as find does not.
The point there was that find can recursively create a list of files for grep to search, passing them along to grep. This means that grep's -r switch is redundant.
/me removes Devil's Advocate hat
> cat and < are different...
If I had more time today, I would figure out if one can abuse the hell out of shell redirection to replace cat. It sounds like a fun way to waste a half-hour.
It can be argued that Systemd does do just one thing, and well.
Unix has a process model which has historically been lax. Things built on top of it tend to be shell scripts. Cron is probably the worst example of "do one thing well", because it does that one thing very poorly. It can only start stuff. If you want to avoid dogpiling, enforce timeouts, enforce resources, prevent multiply-forking jobs from leaving orphan processes, preserve historical per-job output, pause or delay jobs, manually start jobs, balance jobs across multiple nodes, etc. — then you just have to cobble that together yourself. I certainly have.
What "cron" and "/etc/init.d/something" and "scripts that run on ifdown/ifup" and "scripts ran run on DHCP changes" and "scripts that run at specific times" have in common is that they are arbitrary processes whose lifetimes are bounded — by time (cron-like behaviour), dynamic events (such as networks going up or down), human input (manual start or stop) or other factors. There's no conceptual difference between a clock and a human action, for example. If job X is set to run at midnight, and I want to run it a little sooner today, couldn't I just trigger it manually? For Cron to have this, and for Cron to fix the other deficiencies I mentioned, it would have to replicate features to the point where it becomes... Systemd. Because Cron starting processes at specific times is just a special case of starting processes at boot (init) or starting processes when networking goes up (/etc/network/if-up.d or whatever).
Systemd is a process lifecycle supervisor. It tracks the lifetimes of processes. That's about it. Everything else is pragmatic problem solving. Just like grep's simplicity ("finds stuff by text") is clouded by the dozens of arcane flags it must support to real-world use cases (styles of regexps, reading patterns from a file, printing options), so must Systemd cater to many minor aspects.
It can be argued that Systemd could be less monolithic and more pluggable, but I feel that's another story.
> It can be argued that Systemd does do just one thing... Systemd is a process lifecycle supervisor. It tracks the lifetimes of processes. That's about it.
It's also a /dev node manager, and a syslogd replacement, and a cron replacement, and a dhcpcd replacement, and a mount/umount, insmod/modprobe reimplementation, [0] and a bootloader, [1] and...
The systemd project is huge, sprawling, and continues to grow.
> Cron is probably the worst example of "do one thing well", because it does that one thing very poorly. It can only start stuff. [If you want something sophisticated you inevitably have to implement systemd.]
It turns out that you don't! Check out fcron. Its crontab is described here: [2]. If you're impatient, search for "The options can be set either for every line".
> Things built on top of [Unix's process model] tend to be shell scripts.
The phrase "shell scripts" gets tossed around like it's a slur.
What are the essential differences between a Bash program, a Python program, a TCL program, an Erlang program, and a C program that all do the very same thing? What makes "shell scripts" so undesirable?
[0] Given the number of times things like recursive bind umounts, module loading/unloading, and more core stuff have broken on systemd-enabled systems and only such systems, it's almost impossible to believe that mount/umount, insmod/modprobe & etc. aren't being reimplemented in the systemd project. ;)
* Can I defer their scheduling programmatically without editing a file to comment it out?
* Can I add or remote new jobs programmatically and atomically?
* Can I ask it about the status of a job?
* If a job fails, is it retried, up to a limit?
* Can I give it a time budget after which the job is killed?
* Can I give it a resource budget (e.g. CPU usage)?
* Can it log the output tagged with an identifier that identifies the job uniquely across time?
* Can it notify me by some programmatic mechanism (so that I can pipe it to a webhook, say) if a job fails?
* Can I programmatically start jobs, edit jobs, etc. from a different host?
* Can I ensure that only a single instance of the job runs in a cluster, if multiple hosts have been configured with the same job?
* If the job runs so long that it overlaps with the next scheduled period, can it ensure that the next job doesn't start yet?
* If the job forks, can it ensure that the children are cleaned up?
These are some of the features that I want from a job scheduler. Yes, some of it can be cobbled together with wrapper scripts, manual cgroups, and so on. But I don't want to write all of that.
> What are the essential differences between a Bash program
Shell scripts are brittle. That's hardly debatable. You can write solid shell scripts, but people generally don't because it takes a lot of effort and arcane knowledge to do it successfully.
A lot of people don't know about classic safeguards such as "-e", "-o pipefail", exit traps and so on. Some of the safeguards you want are hard to do safely and atomically.
Unix has a nice pipe system, but shell scripts that need to drive a lot of things invariably mess it up because everything pipes to the same place. You get errors and information output intermingled and there's often no way to tell who emitted it without going step by step.
Throughout my 20+ years of using Unix variants, shell scripts and "shell-scripty interdependencies" have always been a source of problems, usually related to things like escaping, quoting, option parsing, alias expansion, variable expansion, bad toolchain detection (e.g. assuming your "ls" is GNU), ignoring exit codes, bad output parsing, etc.
"find | xargs" works just fine until the day you encounter a file name containing spaces. Then you do "find -print0 | xargs -0" and you're happy until the next edge case comes along that touches on space handling, of which the shell language family is rife. It gets worse when shell scripts try to be nice over SSH.
At least languages like Python and Erlang have first-class support for primitives, even if their dynamic typing leads to a lot of potential situations where the runtime parameters aren't what the author assumed, and things blow up because someone didn't sufficiently duck-type. I'm not a huge fan of Go, but Go is a surprisingly good fit for command-line tools, and its strictness and static typing helps avoid a whole class of errors.
I love shell scripts, and I write them nearly every day, but I would never build an OS on top of them.
Several of the items in your bulleted list make it seem like you either didn't read all of, or didn't thoroughly understand fcron's crontab manpage. I urge you to go back and re-read the documentation, or -at least- the Options section.
I'll address the items that aren't covered by the documentation that I linked to. :)
> * Can it let me start jobs manually?
> * Can I ask it about the status of a job?
Yep. [0]
> * Can I defer their scheduling programmatically without editing a file to comment it out?
I'm not sure what it means to "defer" scheduling. That sounds like changing the time a job starts. If I'm wrong about that, please do let me know. :)
Anyway. Every job scheduling system stores its non-volatile info on disk. fcron's full-featured interface to its job scheduling system is found in [1]. I don't know if that meets your requirement.
> * Can I programmatically start jobs, edit jobs, etc. from a different host?
Yep. [0] [1]
> * Can I add or [remove] new jobs programmatically and atomically?
AFAICT, yes. [1]
> * Can it log the output tagged with an identifier that identifies the job uniquely across time?
> * Can it notify me by some programmatic mechanism (so that I can pipe it to a webhook, say) if a job fails?
Yep. That's in the email that it sends. What your reporting infrastructure does with that email is up to you.
> * Can I ensure that only a single instance of the job runs in a cluster, if multiple hosts have been configured with the same job?
AFAIK, Systemd doesn't do this, so I don't know why you're asking for it. (Other than the fact that it would be rather nice to have.) :) But, no. AFAICT, fcron runs on a single host. I have heard about chronos, though. [2]
So, I need to preface the rest of this commentary with the following: Bash is clunky, kinda unwieldy, and very, very far from my favorite language. There are several reasons why I select Python for scripts that get much more complex than "rather simple". Please keep this fact in mind while reading the rest of my commentary. :)
> Shell scripts are brittle. ... You can write solid shell scripts, but people generally don't because it takes a lot of effort and arcane knowledge to do it successfully.
But C programs written by people who can't be arsed are also brittle and -even worse- the language itself is filled with hidden and subtle pitfalls!
> Unix has a nice pipe system, but shell scripts that need to drive a lot of things invariably mess it up because everything pipes to the same place [and this makes unwinding intermingled output difficult sometimes].
So, why not religiously do something like using logger(1) with appropriate tags and ids? I mean, you actually have to work a little to get reasonable logging in (almost?) every language. Hell, -IIRC- C doesn't ship with anything substantially more sophisticated than printf(3) and friends. ;)
> Then you do "find -print0 | xargs -0" and you're happy until the next edge case comes along that touches on space handling...
How does
IFS=$'\n'
and/or the
"${VAR}"
pattern not fix all space handling problems when using bash? (Bash is one of my weaker languages, so if you know, I'm genuinely interested in hearing about it.)
However, if your answer is something like "Some other script you use might screw things up!", then my reply is "Bash and sh programs are not the only languages in which we might find an erroneous program on which we depend.". :)
> It gets worse when shell scripts try to be nice over SSH.
How's that? I'm seriously asking, for my own edification.
> I love shell scripts, and I write them nearly every day, but I would never build an OS on top of them.
I don't think that I (or anyone else who has an informed opinion on the topic of sysvrc replacements) has ever seriously suggested that one built an entire OS on top of shell scripts. Frankly, they're insufficiently powerful (not to mention [comparatively speaking] too damn slow) to reasonably accomplish the task.
However, OpenRC -and so many other RC and init systems- demonstrate that shell scripts can provide rather powerful, relatively foolproof methods for a packager or service author to control services on their system.
> At least ... Erlang [has] first-class support for primitives, even if their dynamic typing leads to a lot of potential situations where the runtime parameters aren't what the author assumed, and things blow up because someone didn't sufficiently duck-type.
I am totally, seriously, not (!!) getting on your case here, but how much work have you done with Erlang? I'm a novice-to-middling Erlang programmer, and the situation you described seems... difficult to run into unless you're unusually careless. (Python, OTOH...)
> I'm not a huge fan of Go...
Me either! It's so opinionated. Does it still consider unused imports to be compile-stopping errors? :(
I could talk about this all day, but I don't have the time, so I'll just address some of your more important points.
Fcron has some features that I like that I didn't find when I looked at it (the documentation is very poor: "dialog dyn-amically with a running fcron daemon" didn't really invite me to think that it had a job scheduling UI), but it's not sufficiently built out that I will be able to trade my current toolset for it. For example, it seems that it lacks any process isolation. Email-only reporting is not acceptable. And so on.
I agree that Fcron is a better Cron. But I also think that it's a very poor Systemd. Like Systemd, it controls the lifetime of processes; it has an overlapping feature set (job files, backing store, control UI) and if one made it complete, it would look a lot like Systemd. I'd rather have one tool do process orchestration, rather than two.
> But C programs written by people who can't be arsed are also brittle
Certainly. But it's hard to write C programs that fail for the same half-assed reasons that shell scripts do. C programs suffer from other problems. I'd much rather have a C program segfault on bad input right away than have a shell script happily continue to run because it ran some text file through tr, got a different result than it expected, stored it in a variable, passed it badly-quoted to some tool 27 lines down in the script, and triggered an impossibly obscure error message unrelated to the original cause.
But the real alternative isn't C, of course. It's one of the languages you mentioned. Ruby, Python, Erlang, Go, Nim, etc. all improve on the historical sloppiness of Unix.
> How does ... not fix all space handling problems when using bash?
There's much more to it than that, I believe.
For example, preserving quoted arguments is a nightmare. Let's say you read some file that contains command-line options. It contains something like:
-x -y --string="hello world"
You want to read that, preserving quoting, into an environment variable, which you then want to modify and pass on to another script. Turns out this is hard. Normally, quoting "$foo" works; but the moment you fiddle with it, the quoting is lost. There's a neat bash trick few people know about which exploits its array support:
Throughout this script, $ARGS' internal token structure is preserved, even when it's manipulated. There are other ways to accomplish this, but none of them are pleasant, and this way is quite pleasant, once you get past the weirdness. But I would bet that most developers don't even know that bash has first-class array support. (If you know a better solution,
This is just one horrible corner case where a naive implementation would blow up. As I said originally, it's possible to write safe, resilient, mostly well-behaved shell scripts. But it's hard; you have to know about every possible pitfall [1] and divergent behaviour on GNU and BSD. Shell scripts implement most of their functionality through external commands, not functions, so you cannot assume much about the interface of commands. This takes time and effort. But shell scripts are easy to write, which invites sloppiness. Few people write unit tests for shell scripts. Few people use something like autoconf to sniff the runtime environment (e.g., GNU vs BSD).
As for "entire OS built on shell scripts", every Unix distro out there is built on tons of shell scripts. Even Upstart and Systemd can hardly avoid using a bit of scripting.
As for SSH, you're in a pretty good spot if you send your script using <<'EOF and disable the tty. Otherwise you run into quoting issues fast, and scripts that expect interactivity will hang. There might be some other corner cases, I forget. Controlling SSH from scripts has always been painful and brittle for me.
> [Erlang] ... difficult to run into unless you're unusually careless
Sure. But it's there. Erlang's duck typing means it's susceptible to sloppy inputs just like any other duck-typed language. It's a lot better than the shell script situation, to be sure.
> (the documentation is very poor: "dialog dyn-amically with a running fcron daemon" didn't really invite me to think that it had a job scheduling UI)
I... uh.... read the documentation -rather than the second headings- and immediately understood that it had a job scheduling UI. :) I understand that we're all busy, but it does pay to take fifteen minutes or an hour every couple of weeks to sharpen the axe.
> it's not sufficiently built out that I will be able to trade my current toolset for it.
I never was claiming that you should. I was addressing your initial claim, mentioned a little bit later in this comment.
> For example, it seems that it lacks any process isolation.
Linux does that.
> Email-only reporting is not acceptable.
Consuming email is trivial. There are something like 349024823904 libraries and like a handful of good ones to do so in every popular language. :)
> I agree that Fcron is a better Cron. But I also think that it's a very poor Systemd.
Your initial assertion was that the only way to get a good cron was to implement systemd. I demonstrate that that's not true, and you counter with "Well, that cron's not systemd!"!
sigh.
> I'd much rather have a C program segfault on bad input right away...
You and I both know that it's trivial to write a C program that does exactly the same thing that your hypothetical poorly-written bash program does. :) Bash didn't invent Heisenbugs.
> It's one of the languages you mentioned. Ruby, Python, Erlang, Go, Nim, etc. all improve on the historical sloppiness of Unix.
/s/Unix/Bash or sh/ and I agree with your statement. I also note that Perl (shudder) is conspicuously absent from your list. :)
> There's much more to it than that, I believe. For example, preserving quoted arguments is a nightmare.
Thanks for the examples! I'll go over them over the next couple of days and see what I can learn from them. :D
> But I would bet that most developers don't even know that bash has first-class array support. (If you know a better solution,
Did you accidentally a sentence or two? Also, the recurring theme that I've been spotting here has been "Programmers who don't know their language often write bad or broken programs in that language.". Is this not one of the larger themes of your statements? :)
> Controlling SSH from scripts has always been painful and brittle for me.
Sorry, this is something that I've not yet had call to do. What exactly do you mean by "controlling SSH from shell scripts"? Something like
embedded in your script? (Or the equivalent here-document?)
> As for "entire OS built on shell scripts", every Unix distro out there is built on tons of shell scripts.
That hardly means that the entire OS is built on shell scripts. I argue that the vast majority of the shell scripts out there are to handle process startup. While this is quite important, it's important to remember that the stuff that actually gets work done in the OS is very rarely written in Bash.
> Erlang's duck typing means it's susceptible to sloppy inputs just like any other duck-typed language.
I see what you're saying (and agree with your final sentence), but one could just as justifiably say
"C and C++'s ability to permit the programmer to override the specified type of a given variable means that such programs are susceptible to errors of sloppy logic. We should really be using Haskell to write our system management programming." ;)
Anyway. Thanks for the ongoing conversation. This has been quite informative.
Actually, you pointed me to Fcron, to which I countered that for Fcron to be an satisfactory scheduler (that is, satisfactory to me) it would have to become Systemd. I think I was pretty clear on that, actually.
No, I really did mean "historical sloppiness of Unix." It's a fine foundation, circa 1985, but it's time to make some progress. For example (just a minor example), the fact that Cron's error reporting is email is ridiculous. This means that the only way to collect reports is to either use a mail client, or to set up a special rule (in Postfix or whatever) that pipes incoming mail to a special directory or a process. Both options are insane, not to mention horribly brittle. Surely Cron violates the Unix principle (the "do one thing well" one) here; the right thing would be to not support email at all, but to mandate that every error report is piped through a per-job command. So PIPETO rather than MAILTO. If you really want email, according to the Unix principle, you just pipe to sendmail, after all. But a whole lot of the traditional Unix toolset is weirdly crufty this way.
As for shell scripting, my original point was precisely that the shell language family is very hard to get right, and so it's a poor foundation to build complicated, resilient moving parts on top of, emphasis on "moving". The situation is much worse than C; the class of possible bugs is completely different, and much more damaging. Things like process management should not be done using shell scripting. Lots of things suffer from bad scripting; autoconf, for example.
By controlling SSH: Yes, I mean interacting with SSH commands. You have to cover a bunch of pitfalls: disabling tty, disabling the control master multiplexing (has never been stable), always using heredocs, etc.
> Vixie ipse dixit is true, but not a technical argument.
It's also rather presuming that Vixie cron is the only cron, something that hasn't been true for ... ahem! ... a little while now.
Back at the turn of the century, a bloke named Paul Jarc looked into how one could do cron the daemontools way. What he came up with was called runwhen. You can still find his WWW page on the subject.
I incorporated some of the ideas into the (32-bit) OS/2 Command Line Utilities, albeit heavily modified to address an operating system that most definitely did not do chain loading.
There are several things to learn from this. First, the world is not as rigid as some would have you believe. There's been a lot of work on service and system management, cron alternatives, and other stuff over the years. For a less radical cron than runwhen, witness Bruce Guenter's bcron. (The nosh package has a service bundle for bcron, by the way.)
Second, what might be true about old Vixie cron specifically isn't gospel for the entire world. It's interesting to observe, as I have mentioned before, how much influence the daemontools family has had over the past decade and a half. Looking at cronie, to pick an example, one can see it nowadays is yet another member of the club of programs sporting a "don't do all that bogus daemonization stuff" option, -n in this case. GNU cron (a.k.a. Mellor cron) actually does not "daemonize" by default, and one has to explicitly choose that with a -d option. To look at old Vixie cron, one might come to the erroneous conclusion that cron always and unavoidably "daemonizes", something that was indeed a problem for daemontools users in the late 1990s. But this isn't true by a long chalk today.
> Cron is terrible. Paul Vixie says so himself. Get over it.
"cron" does not[1] necessarily imply vixie-cron. A variety of implementations are available that have different feature sets. What exactly do you not like about cron the concept, and how are you running periodic events without
> HUP ... listening for an event
HUP is an "event". What is this "event" you want to listen for that requires junking HUP support? A new event type that tells the daemon the same information is just as kludgey as sending SIGHUP while adding unnecessary complexity.
You can listen for an inotify event (or similar) on relevant file handles if you want, but that isn't applicable in all cases. If your complaint is that some daemon doesn't support something like inotify, file a feature request or send patches.
HUP has the advantage that it is well-known, and you probably probably still want a way to force a daemon to re-read it's configuration even if you are already detecting changes with inotify/whatever.
The nosh package gained a service bundle for fcron in version 1.20, back in September. As you can see, it uses the --foreground and --nosyslog options. Thank you for those M. Godouet.
> nosh is far with the base system rc scripts, but ports... oh, ports...
I recommend reading the nosh Guide. It has a whole chapter on creating service bundles, and lays out in detail the three-pronged approach to ports that nosh enables. This is just a précis.
* run scripts are short and simple. One can just write one for a port. It can be a shell script, a perl script, a python program, an execline script, a nosh script, or something else.
* You'll find that ports are growing systemd unit files. Indeed, they not only are but have been. The writing has been on the wall about this development for some years, now, and it's well into actually happening. The nosh toolset includes a mechanism that can take the majority of such unit files and generate a native service bundle from them.
* One can just take the stuff that's already out there and that's already supplied. Recently the 157 rc.d scripts part of the project has hit the Russian news. That's only part of the story, though. Read the nosh Guide, and you'll find pointers to four prominent collections of daemontools/runit scripts that have existed for years. Look into the package itself, and you'll find almost four hundred service bundles pre-made and supplied in the box. Which of course feeds into the first approach: One can take an existing service bundle for a similar program and adapt it.
nosh is by far the worst system I've used in the daemontools line. I experiment with init scripts on a vm I have and have run several variations of daemontools on it. nosh was the only one I couldn't get working, and so I looked at the code to try and figure out what was going on...
It appears that they have sacrificed readability of the code in exchange for never having to call fork(). After being unable to fix my problem, it is, in my opinion, not a good tradeoff.
Your question is cognitively devoid of meaning without giving a frame of reference, methodology, heuristic or other a priori anchor for me to be able to give you a reasonable comparison between two software architectures.
Nonetheless, I can give you examples of where launchd is deficient in its own problem domain, and particularly for a BSD. It has non-generic socket preopening (unlike UCSPI), its service classes are daemons, agents, XPC and Mach services. The latter two are OS X-specific and bound to IPC mechanisms no less, which is an odd taxonomy. Furthermore, agents (per-user/session daemons) are subdivided into four session types: pre-login, per-user, GUI session and non-GUI session. GUI and pre-login session types are specific to OS X's Aqua and loginwindow, and an impedance mismatch anywhere else. Further, non-GUI session is related to how launchd starts sshd from a plist, which again is a rather odd taxonomy-related coupling. Further still, the process types (four of them) aren't used as information for what readiness protocol or startup discipline to use, but are tightly coupled to scheduling policies and resource limits! These four process types make sense only within the context of a desktop system where you want priority scheduling and resource accounting to keep system responsiveness as events queue up, but are meaningless in the generic case. Then, launchd strictly limits you to a lazy loading startup discipline and does not have any formal ordering or proper service relationships (other than KeepAlive, all options of which are either prone to race conditions or too narrowly defined) and instead relies on daemons to resolve relationships cooperatively via IPC with all the problems of cooperative/uncoordinated protocol negotiations. Fine for a cathedral like OS X, unworkable anywhere else. This on top of system and service state being intertwined in the same critical paths of PID 1. The whole thing is a mess of layering violations. Let's not ignore how the NextBSD developers had to break Mach thread and task semantics to port the API (as a kernel module, no less!) over to a monolithic Unix with which Mach does not properly overlap. Finally, a more minor cosmetic gripe is LaunchOnlyOnce being a key instead of a job type.
It's an absolute catastrophe through and through. Awful design.
EDIT: Odd that I'm being downvoted for specific technical points.
> nosh is, by far, the single most masterfully architected and carefully researched toolkit for init, process management, supervision (also with auxiliary utilities for user mode VTs and UCSPI) out there at present, trailing only with s6. The FreeBSD community will do itself a great favor by adopting it over the NextBSD experiment.
Isn't your statement cognitively devoid of meaning without giving a frame of reference, methodology, heuristic or other a priori anchor to give a reasonable comparison between multiple software architectures?
If it isn't, shouldn't basic human courtesy and common sense lead you to assume that 2trill2spill's question should be interpreted in whatever frame of reference etc. you were using when making your initial claim of nosh's superiority over all other options?
I can't believe people are not downvoting you. Everyone should be downvoting you for the uncharitable, rude, and hypocritical way you are approaching this discussion, regardless of any technical points.
I'm sorry my question made you so upset, but I have seen several comments by you asserting nosh is superior to launchd, and that the NextBSD project is misguided, yet I haven't really seen much specifics, on those assertions. So I thought that because you would wrote...
> nosh is, by far, the single most masterfully architected and carefully researched toolkit for init, process management, supervision (also with auxiliary utilities for user mode VTs and UCSPI) out there at present, trailing only with s6. The FreeBSD community will do itself a great favor by adopting it over the NextBSD experiment.
You would have specific examples of nosh being a better init system than launchd.
I would also add that you didn't really address my question, you explained some deficiencies of launchd, but didn't explain how nosh address's those issues or how it avoids them.
Finally If you really wan't to advocate for nosh over launchd your best off not writing sentences like...
> Your question is cognitively devoid of meaning without giving a frame of reference, methodology, heuristic or other a priori anchor for me to be able to give you a reasonable comparison between two software architectures.
If nosh really is an improvement on launchd great! But petty verbal attacks are no way to get the FreeBSD community to adopt nosh.
I would also add that you didn't really address my question, you explained some deficiencies of launchd, but didn't explain how nosh address's those issues or how it avoids them.
Those deficiencies are all intrinsic to launchd's architecture specifically, and not intrinsic to the problem domain. As such, simply not being launchd is sufficient to avoid them.
If nosh really is an improvement on launchd great! But petty verbal attacks are no way to get the FreeBSD community to adopt nosh.
There are no petty verbal attacks anywhere. That your question was noncognitivistic is I think not of dispute, but even if it is, it's still not a "petty verbal attack" that was leveled, but a methodological criticism. Instead, you're ignoring all of the flaws I listed in favor of a red herring dismissal by focusing on my criticism of your question. Not everything revolves around your feelings.
> Instead, you're ignoring all of the flaws I listed in favor of a red herring dismissal by focusing on my criticism of your question. Not everything revolves around your feelings.
I'm not ignoring the issue you have pointed out, rather It would be nice to hear about nosh. There's significantly more information about launchd than nosh, for example launchd has two videos, explaining the reasoning and methodology for bringing in launchd and the other apple libraries.
A quick search on google and their are no videos about nosh. The nosh website doesn't contain much information about the design of nosh. Looking around the source of nosh the various html files look promising and i'm going through those. Unfortunately my c++ skills are inadequate to judge the code quality of nosh effectively. Which on a side note I think will hurt nosh chances of being adopted, because the open source os world seems biased to using C. I't just harder to find stuff out about nosh and you seemed knowledgeable on the subject, hence the original question. So is there any good documentation on nosh besides the source it's self?
> So is there any good documentation on nosh besides the source it's self?
Yes. Taking a page from upstart (with its Cookbook), FreeBSD, and even Microsoft, nosh not only comes with reference manuals but comes with the nosh Guide. (One thing that Microsoft doco such as the MSDN Library consistently does is have both "about" and "using", i.e. reference and guide, doco.) It's first on both of the binary package download pages, separately downloadable and installable without need for installing any of the programs, and recommended by the timorous admin's how-to right at the start.
The very first thing, in the introduction chapter, is a section on the design fundamentals.
Strictly speaking, this wasn't actually page from either upstart's or FreeBSD's books, as the habit pre-dates upstart's very existence by well over a decade. nosh is not the first project to have both reference doco and user guide (and indeed announcement blurb separate from both) by any means. Here's another:
Odd that I'm being downvoted for specific technical points.
No, I presume you are being downvoted for your apparent rudeness despite your excellent technical points. For me, the two cancelled, and I simply chose not to upvote an otherwise excellent post. I like your wit, but sometimes it's a bit sharp. Others apparently felt the rudeness was the dominant factor, and (as is their right) downvoted.
Largely, the author doesn't believe that launchd will ever successfully come to BSD. It's had a number of attempts, all of which either failed or fizzled out.
I wouldn't call that a very compelling argument about why nosh is better than launchd. So what if previous attempts to port launchd failed when this port(NextBSD) appears to be working. So far I've seen several people on hacker news say that nosh is a better alternative to launchd, but I haven't seen any specifics, right now it just seems like politics.
Yea not saying I buy that argument either, the compatibility/conversion layers that exist for nosh could probably be built for launchd, so I wouldn't consider those significant advantages. I also wouldn't consider launchd being deployed on OSX that much of an advantage either given that it seems large parts had to be rebuilt to use the BSD kernel interfaces rather than the darwin ones. Maybe that it's already got configurations for a bunch of services for things from Homebrew/MacPorts/Fink, but they would still need to be reviewed for any broken paths and things from OSX that aren't on BSD.
Interestingly, FreeBSD is well along in resolving remaining issues such that nosh can supplant the init/rc.d tools in use for ~15 years. See [0].
Haven't yet learned about nosh in detail. It does appear to be well thought out, and obviously it's been around for quite a while reaching some state of maturity. While I've not noticed particular difficulties with the current init system in FBSD, my system is relatively straightforward. I can see how dependency complications can arise in complex situations.
Given the notorious controversy over systemd, it's curious that at least some Linux distributions haven't adopted nosh as an alternative to systemd. Perhaps such distributions exist, though if so not well publicized.
nosh is cool, but the "systemd compatibility" is for the systemctl command, socket activation, and automatic unit file conversion; it doesn't offer implementations of the systemd APIs, which is where "systemd compatibility" is really needed.
I've just read that it uses a lot of Linux-specific things, but are there features that inherently can't be ported to BSD? And are those features absolutely critical to running it at all?
What I've read sounds more like "we're focusing on Linux, and won't be porting it" rather than "this software is inherently Linux-only, and can't be ported".
I daresay coming up with a shimmy layer to port it to BSD is probably less effort than completely rewriting it.
I like systemd, but I do think it'd be far easier to write a compatible system for FreeBSD (or enhance the FreeBSD kernel to provide the necessary features) than to port it. For instance, systemd fundamentally depends on cgroups; without an equivalent mechanism, it won't work at all. The same goes for dozens of other features. The necessary functionality doesn't exist in POSIX; rather than attempting to write a half-functional "portable" implementation, it would make more sense to embrace the FreeBSD kernel, use its features as appropriate, and build a full-featured init system specifically for FreeBSD. In an ideal world, that system would provide all the features of systemd.
(Note, for instance, that I said "FreeBSD", not "BSD"; portability is not worth trying to enhance 3-4 kernels to support the features you need, or sinking to the lowest common denominator.)
> I daresay coming up with a shimmy layer to port [systemd] to BSD is probably less effort than completely rewriting it.
The self-styled "uselessd guy" is right here on this very page. Xe attempted making systemd portable. The project is now dead. The experiences of the "uselessd guy" and of Ian Sutton, who is the bloke who did the work trying to make a systemd-logind for OpenBSD, are worth hearing.
They've expressly said that they won't accept any patches that will make it portable -- "We do not consider it feasible to port systemd to other Unixes (let alone non-Unix operating systems) and will not accept patches for systemd implementing any such portability" [1]. A lot of the issues come down to how Linux does namespacing, and how there's a lot of code that's hard-wired to require it. So any port would be a fork by near definition.
The systemd developers do no care about BSD, full stop. In fact they believe that portability is a negative attribute that leads to compromises and that by coupling everything together systemd can make Linux a better user experience and more like the popular proprietary OSes.
On top of that, systemd is LGPL. That may not be a 100% showstopper for inclusion in a BSD-licensed system, but it is fairly close to it, especially for such a core component.
Given the other replies in this discussion thread, I am inclined to believe that they are /Good Stewards/ of the design goals they have.
I believe a more proper examination of 'open-souce' or even Free Libre citizens might be found in examining the license selected for their works.
They chose the GNU LGPL 2.1+ license. Given their other stated beliefs, I read that they believe in allowing others to build off of this work, but would prefer those developers fork it and to begin a sister work that is more optimally coupled to another operating system.
That would be fine if they would standardise their APIs. But they only do that at the boundary; as more and more functionality moves into systemd that means more and more with no documented, supported API. Any port would have to implement systemd piecemeal, like GNU did with unix - but all the systemd interfaces are nonstandard and unsupported, making that impossible.
The internal ones, as I said. E.g. you can't write a drop-in replacement for logd, or even a reader of its files. IIRC systemd doesn't support non-kdbus dbus - or indeed older kernels at all.
systemd supports non-kdbus DBus, that is essentially what every deployed systemd system is using. Including this laptop, which is on latest systemd (227) with upstream kernel (4.2.5, kdbus not included).
Why is it important to be able to replace journald (assuming that is what you meant by logd)?
Being able to replace individual pieces is the heart of open source - the only reason the GNU project was able to get off the ground is because it could swap out individual parts of unix without having to write the whole thing from the ground up. More generally, forking only works if you can do it on a granular level. So if I have an idea for improving journald and for whatever reason the official maintainers won't accept my patches, I should be able to release my own fork of just journald and if people like it then it will become more popular and maybe be mainlined or maybe not. If I have to fork the whole of systemd to do that then it's too much overhead.
journald is a separate module in the systemd tree, so this is very likely possible. You have both the code implementing the daemon and the one using it available to study, so the interface.
The issue is that systemd might change that interface at any time, even in a minor or patch release. Because systemd's released as a single monolithic project, they don't have to commit to any standardization, deprecation cycle or the like for the interface between components.
Yep, no stable interface for journald. Someone is gonna have to prove to them it is needed, by implementing something useful that utilizes said interface.
but as documented previously in thread, for plenty of their components they have committed to stable, documented interfaces.
systemd supports 'older kernels', just not old ones.
systemd 227, released last month, requires Linux 3.7+. That version was release December 2012, or nearly 3 years ago.
I'd love to see a side-by-side comparison of systemd vs SMF. (I just searched for one but could not find it.) I wonder if those Linux folks who are so anti-systemd would feel the same way about SMF? When SMF came out, I don't remember any of my Solaris sysadmin friends complaining about it, they all seemed pretty excited.
I'm in the "Loves SMF but not a huge fan of systemd" camp
SMF took over and replaced things that made sense for it to replace. It didn't attempt to replace everything everywhere, which as best I can tell, seems to be the overall summary of the systemd roadmap.
I would love SMF on Linux, but unfortunately it would be quite difficult to port over - it makes extensive use of contracts and doors.
It would be interesting to know how large this demographic is: one of the interesting possibilities we have with LX-branded zones on SmartOS[1] is that "SMF for Linux" is a distinct possibility, albeit by bringing Linux to SMF rather than the other way around.
I'll go on the record and say that, as someone who uses SmartOS in production, I've thought about spending the time to try to coerce SMF to manage things running in LX-branded zones.
If nothing else, it would make building reliable services a little more accessible to some of the systems administrators that I support who aren't quite in love with the idea of using runit and monit over SysV init and cron.
As an originally Linux admin, and then a UNIX admin, and now back to Linux..
I liked SMF.. Fuck SystemD with a rake in the rain on a Monday.
SMF had a lot of improvements also after opensolaris died off and we were left with openindiana.. the thing is, it didn't fly in the face of the current system, logs were referred to not 'journaled' away, and it didn't try to gobble up the networking stack at all, but it did allow the system to verify networking ability.
it was very nice, and I regret not understanding it more.
Although I didn't like that it was XML, yaml,JSON or even INI would be preferable in my mind.
It sounds like you've had specific bad experiences with systemd, would you mind sharing what they were? A lot of the criticisms I've seen of it were philosophical and many hadn't actually ever used it, so I'm curious to hear some practical complaints. Personally, I've been using it for a while and it's been pretty good to me, so it is interesting to hear from people who have been less fortunate.
I've also used SMF a bit and thought it was pretty decent.
mostly in the logging, it's hard to get anything truly meaningful when shit hits the fan, I even studied the man pages fairly religiously but because the native application had it's own logging mechanism which was removed by the distro maintainer/packager (fedora) it took a while to realise and reenable. this is the first listed not because it's a big deal but it cost me a day of work, which is hard to explain to my boss.
There are multiple issues in how it handles networking, notably that 802.3a/d bonding is kinda broken and nobody seems to give a crap, which means using it in production is somewhat dangerous for me, the same way that loading modules breaks, gets fixed then rebreaks in odd ways (lsmod/modprobe)
the fact that nobody seems to give a crap about the previous bug, and that bugs get introduced and the developers hostile attitude also rubs me wrong.
FirewallD is a shitshow, and yes it can be disabled, but that doesn't mean that it's -not- going to be the officially supported method of interfacing with netfilter in the kernel.
I've had systems become unbootable because systemd introduced a bug which caused it to segfault and I couldn't "revert" the change like I would with a bad kernel, but this is the cloud era and I should really be adopting ephemeral builds- but if it ever happened on a stageful machine I'd be angry, I can HA all I want but I want to be able to recover and more robustness is significantly better than less.
My problems are fewer because I never wanted to touch systemd, I've been skeptical and rewarded somewhat for my skepticism when i actually started using it out of curiosity or because I really hate being on the previous version of a thing.
Actually, what I really hate most is not the issues I've had, new systems often have warts and we should encourage discussion to get them fixed.. not shut them down when problems are shown because there _are_ problems.. my issue is how under-represented other init systems are, I didn't like systemd and was basically told (by developers, mostly) to "suck it up" and to stop "getting in the way of progress", and while I've never tried to defend sysVinit, I was always keen to have a discussion about a _well engineered_ solution to this would be.. somehow SystemD has taken over and it's more windowsy than we imagined and growing daily.. it even has a bootloader now.
Nosh is a cleaner implemented init system and so is SMF, they fix a problem and do so well..
systemd works great on my laptop though! (arch linux)
So, this isn't exactly a tech support forum, but...
Does your Arch Linux system consider nr_inodes=0 to be a valid mount parameter for tmpfs? If it does, does it use systemd 225?
I upgraded to Kubuntu 15.10. On first reboot, I found that my boot failed because the system claims that tmpfs does not accept a value of 0 for the nr_inodes parameter.
It has been documented for a very long time that "0" for nr_inodes, nr_blocks, or size means "unlimited". My Gentoo Linux systems running kernels from 3.x through 4.2.x have always happily accepted this value, so I know this isn't a mainline kernel change. Kubuntu 15.04 also accepted this value.
Are you suggesting BSD will get systemd? I don't understand. All this is is an init replacement that takes some cues for systemd and launchd and SMF, but it's certainly not systemd.
It is, indeed, daemontools++ with lots of optional compatibility layers, carefully planned module boundaries and a small chain loading language for composing execution state.
That said, compared to what is standard fare in DevOps, it will likely be found too esoteric. There is also a tad more to be explored in reexamining this controversial problem domain, which I hope to unveil in the coming months.