I've worked as a sysadmin/devops for a double digit number of years.
Across companies, jobs, and hobbyist collaborators - I've never met someone who didn't at least like systemd, if not sing its praises. It made so many incredibly painful things about Linux administration disappear.
E.g. logging, now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
Boot and shutdown has previously been deterministic, now things just randomly hang. You do the "windows solution", reboot again, now it magically works through the power of race conditions that are inherent in systemd's mode of operations.
Network device naming is also a fun one. Remember how systemd was supposed to do away with the inconsistency and mess that is "eth0, eth1, eth2, eth..."? Well, now you get either eth0, or ens123, or enp17s7f9, or enx8b220b34. Each distro does it differently, so put in a live-CD to debug something and your network devices are suddenly weird. Reinstall to another version, suddenly weird again. Btw, the issue that names could change depending on boot order was already fixed by udev before, consistent-net.rules nailed it to a mac address on first boot. So systemd took a situation that was already fixed and better than before, and made it far far worse.
Oh, and all the fun that is systemd-dbus, policykit, systemd-logind and permissions on those. Because you always wanted a Javascript interpreter and libxml to tell who could be root on your system... And tons of confused deputy exploits on that cloud of confused systemd-subdaemons.
I'll grant that service files were a nice idea. But beyond that, it has been a very mixed bag, and in places a desaster.
The grand-parent post was about the general consensus: sysadmins generally like systemd. So I'm not sure answering individual concerns is useful, but I'll take the bait...
> now that there is journald, you have to pay attention to another hop in your logging chain and take care of journald in addition to rsyslogd.
No, you don't have to, because a syslog daemon is not required any more. BTW, rsyslogd is not the only syslog service, I've had to work with alternatives like rsyslog-ng.
And the daily usage is now much simpler: `journalctl -b -1 -p err` or `journalctl -u mariadb --since "24 hour ago"` were painful queries with (rotated and compressed) syslog files.
> Boot and shutdown has previously been deterministic, now things just randomly hang.
That was not my experience. IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either.
> Network device naming is also a fun one.
The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel. I think it's still possible to configure the kernel, udev and systemd to work with the old naming, but if it's just because "shorter is better", I don't think it's worth it.
> No, you don't have to, because a syslog daemon is not required any more.
That is "not required" for the single-computer-sysadmin. If all you know is your laptop, then journald might be fine. If your computers are cattle and have numbers, you don't want to read your logs on each individual computer, if possible you don't want to read logs at all, you want some automation to do it for you, aggregate it, warn you about things, etc. journald just makes that harder, and going through some syslog protocol implementation (of which there are a few, you are right there) is the only relevant game in town.
> IIRC, with sysv-init boot dependencies were declared with comments inside shell scripts and parsed by insserv. That was a mess, and race conditions did occur. I have no experience with other boot systems, but I remember that upstart was not immune either.
Dependencies and insserv were a later addition. Basically you just defined a boot order by numbering your /etc/rc[0-6].d/S* links in ascending order, either manually or in a dependency-based fashion through insserv. This means that you can have a suboptimal ordering if those dependencies are wrong or you choose your ascending order poorly. But it also means that once you have a working boot order, it can be repeated, and it will be repeated. That is what I mean by determinism. Once you have a working boot, it will keep working. Solaris SMF did something more modern, you would define dependencies in service definitions, and it would compute a static deterministic ordering from that, without those /etc/rc[0-6].d/S* symlinks. Deterministic, but dependency-based. If it has worked once, it will work again. And they even had a rollback: if after a change, the next boot wouldn't work, you could go back to the previous boot order and fix things from there. And neither had race conditions, because you usually wouldn't do things in parallel, one service would wait for the previous one in the boot order (parallel was possible but rarely used, because of the inherent fragility and possible race conditions).
Systemd does it dynamically on every boot, no precomputation, just looking at all the files and starting everything that could possibly be started at the same time, in parallel. The reason for this is that this makes Lennarts laptop boot faster, which is true in the ideal case. But it is neither deterministic, nor debuggable. And other than the previous methods, it can lead to race conditions.
Note that there is one more race condition at boot, which is races against kernel devices and systemd-udevd. That neither got worse nor better with systemd, it just lead to the kernel people giving up and handling more things like firmware loading in kernel.
> The naming policy is vaguely relevant to systemd. `net.ifnames=1` is the default configuration of the Linux kernel.
The Linux kernel does not have a net.ifnames parameter, and the kernel interface naming scheme is still eth[0-9]+. When you stick that parameter in the kernel commandline, the kernel ignores it and only systemd and systemd-udevd interpret it later on.
Please, if you want to sing the praises of software or deflect blame from broken crap, make sure that you know what you are talking about...
It's a little weird to reply to someone complaining about administrative problems caused by systemd with "generally sysadmins seem to like it."
That kind of response is exactly why everyone gets emotional in these discussions. Technical problems are one thing but when you gaslight people experiencing and reporting legitimate problems you should not be surprised when they get mad and rip your solution out.
GP is being far kinder to the systemd devs than they deserve, I gave up on the project years ago for exactly this reason.
> consistent-net.rules nailed it to a mac address on first boot
fails even harder for
> put in a live-CD to debug something and your network devices are suddenly weird
I don't know that there is really any good answer here. Just methods that break in ways you're used to, and methods that break in ways you aren't. I appreciate the attempt at static naming based on hardware connection, even if the realities of how these things are presented by the bios make the results an inscrutable mess.
Maybe pure enx would have been better? But that looks like crap for even the simple cases, and fails if you swap out a piece of hardware or start doing weird things with MAC addresses.
The only consistent naming scheme I've ever seen working was Solaris on Sun hardware. But that only worked because they controlled both the hardware and the software side of things.
And yes, consistent-net.rules doesn't solve LiveCDs and reinstallations. But as I've said, systemd doesn't solve those either, and in most cases actively makes them worse.
My experience has not been that the systemd device names are "actively worse". The biggest problem I've had is changing around VMs where subtle shifts in PCIe IDs (specifically subdevices) end up changing changing names. Meanwhile, consistent-net.rules seems like a hack, always in the back of your mind as an arbitrary stateful part of the system. Why should I think of my primary network device as eth2 if there is no reason for it to be eth2 other than merely being the order on the first install?
But if you prefer consistent-net.rules, then more power to you. I'm guessing most of your frustrations could have been addressed by systemd having some shame about changing functionality out from under users, and providing an up-front option to switch between the different methods.
I used to be fine with it, until I started writing my own software tools, then looking at how to actually do stuff with systemd and I'm like, nope nope nope nope. At some point when I have space for a new project I'm going to migrate from Arch to Obarun, which uses a modern update on the excellent daemontools to provide init. When systemd works it's mostly fine, when it breaks, path of least resistance is often to reinstall the OS. Which is fine if you're just working on it. Painful if it's your personal machine, and those are what's most likely to break systemd.
> when it breaks, path of least resistance is often to reinstall the OS
Can you elaborate how do you even break this?
Adding a systemd unit cannot really break much on systemd itself. Unless you’re messing with internal systemd stuff, I am not sure what is even there to break.
At the moment, on this machine I'm using, audio just doesn't work. Dunno why. Tried a few different things to fix it, no dice. Trying to troubleshoot is made enormously more difficult by the complexity of systemd. If I had a simpler system I could run it all the way back to the source, but really, I can't be bothered, I don't really need audio on this machine atm.
i have the same sentiment, and for me systemd was getting so slow and hard to tshoot (the tools to analyze boot graphs and sequence showed service sequence breakage, some times there is no problem, sometimes sequence changed and problem happened) and i just changed for another distro without systemd and boom, startup in seconds no problem at all.
This doesn't match my experience at all. For the life of me, I cannot imagine what you are talking about, what any of your issues are. Did you file any tickets, or report any issues?
What about "actually doing stuff with systems" was "nope nope nope nope"? What situations did systemd break in? What did you try before giving up to reinstall the OS?
You probably use a desktop environment like GNOME or KDE. Maybe do some light coding in an IDE. I do not. I run window manager only, i3, with a really stripped down toolset. I like the idea of knowing my system in and out. My current project is writing my own shell. It can switch between command mode and ruby REPL with ctrl-z.
You mention filing tickets / reporting issues, sorry, that means contributing to a project. I'm not wholly opposed to doing that, but I have to like the project first. And there's nothing to like about systemd.
You go do that if you like it so much. My guess is you simply won't, and will just keep using other people's software without a care in the world, thinking this kind of online warriorship is actually helping anything.
Bro, I've been running wmii since before i3 forked off from it. Don't come at me with this dripping disdainful attitude, pretending like you are some high and mighty elite system whisperer.
It's so unfortunate that there is such a widespread persistence of hater-dom. It's so toxic and awful, and so rarely does it have even the faintest shred of anything to stand up on. I struggle so much to see these attractors into the negative as so compelling, so ensnaring of the human souls as they have been lately.
Systemd is amazing, provides such a rich world versus piecemeal poorly managed systems of yore. Yes you could understand init scripts with very little effort but what a sad virtue, to dare to go no further than what an afternoon of exploration might get you.
I don't fucking hate systemd. I'm marginally glad it exists. I just don't want to live under it's despotic regime. You might like it because you think trains running on time is more important than freedom.
You're the one coming at me with fucking attitude, like a 16 year old who just discovered IntelliJ. I have done the exploration, then looked online for why everything is so damn complicated now. It's because systemd colonized the world.
It's not just an init system, it's everything below the userland. And a lot of those components are very poorly designed. And it's this big ball of mush where if you use one component, thy must use them all. It's so bad Arch does not support replacing systemd, like at all. You're totally on your own.
Go ahead, try replacing systemd components in an Arch system. I'll wait. And lose the fucking attitude.
> It's not just an init system, it's everything below the userland. And a lot of those components are very poorly designed. And it's this big ball of mush where if you use one component, thy must use them all. It's so bad Arch does not support replacing systemd, like at all. You're totally on your own.
It's true, it's not only an init system anymore, it's also a service manager, a network manager, a DNS cache and resolver, a proper logging system (as in, with metadata, all in one place) and more.
Despite being opposed to it in the beginning, probably because it was immature at the time, I grew to love it even on Gentoo - my desktop and server OS of choice - which gives you OpenRC as a fully supported alternative:
- on my desktop, laptop and work VMs, it just works, with varying adoption of its components and services started, lazily activated if unneeded
- on my server, it simply makes sense for service auto-restart: OpenRC still requires you to manually enable support for service auto-restart ( https://wiki.gentoo.org/wiki/OpenRC/supervise-daemon#General... ), a feature that comes built-in in systemd and without extra complexity or components, and with predictable outcomes given the unit file
- almost all modern DEs rely on systemd, because it generally improved their session handling, sandboxing, brought user (rootless) services and other features
And generally, I think the biggest win is creating a system service management platform that can be used by all distros in the same way, without having to know 5+ init systems, having different scripts for the same service for each distro. I can basically work on my own devices, on work devices, on cloud VMs, on IoT devices, and have the same behavior, across different distros and hardware.
> OpenRC still requires you to manually enable support for service auto-restart ( https://wiki.gentoo.org/wiki/OpenRC/supervise-daemon#General... ), a feature that comes built-in in systemd and without extra complexity or components, and with predictable outcomes given the unit file
fwiw it's being worked on to have supervision as the default in openrc as well
It's an improvement, I'll give you that much. But it's moved in the opposite direction and seeks to reduce freedom, not give more of it. I should be able to run my own DNS cache, network manager, service manager if I want to. Systemd makes that all but impossible.
I semi-respectfully disagree that I've given you undue attitude.
One of the things I really enjoyed about the Flix programming language submission today was running across their principles section. Right off the top of the list:
> Simple is not easy
> We believe in Rich Hickey's creed: simple is not easy. We prefer a language that gets things right to one that makes things easy. Such a language might take longer to learn in the short run, but its simplicity pays off in the long run.
In my view, yes, there are warts to systemd, but it is an operable understandable system which has a hard fought simplicity to it. I can go to any systemd computer and get a good idea of what's running, understand how it's wired. I use common practices and configs to change or drop in config.
My feeling is that a lot of people want easy. That systemd doesn't pass the easy test.
The past? The past was enormously complicated. It was complicated because every shop has its own way of configuring things. Every shop has their own tools to assess system state, their own way to configure subsystems like logins, dns, networking, logging, time sync, booting. There were countless countless easy answers, a path littered with the bodies of dead easy answers to this or that concern. And you needed an easy answer for each concern; easy didn't go hard to integrate the different factors.
That past was a horrible mess of easy. Systemd feels complex, but compared to the past, there is a persistent understandable simplicity that is a raw joy to use. Things make sense. When you see one thing, you sort of learn how the next works. The back to the woods rejectionism that springs up today is a bizarre weird thing; folks have not seen how wildly different and bedazzling system management used to be, within and across orgs. You might not appreciate why but there's very very good reasons, and I'm fact, the world is inherently complex, there's aot to do, and systemd has created a wonderful pervasive cross-sectional (via it's many opt-in components) "easy" that harnesses and enables the complexity that is, in a sensible glorious way. ("This has made a lot of people very angry and been widely regarded as a bad move.")
Systemd is anything but simple. And the past definitely wasn't easy. You seem to have the terms backward. For the end user, yes, it's a polished experience that you mostly don't have to mess with, and the parts of it you're most likely to interact with are, well, fine. Not great, but fine.
> operable understandable system
Every system is understandable, it's just a question of how much work you have to put into understanding it. Good design reduces the amount of this effort needed. Many many systemd components simply aren't well-designed. But they all interop with each other so there's no way to slowly wean a system off of systemd. Trust me I tried.
I don't want to understand systemd. It feels like being at work. Good on you for putting in the effort. I should be able to use software I like and want to use. Maybe even hand-write some replacements, like I'm doing with my shell. Systemd doesn't get in my way there, thankfully, it's been awhile so I don't have specifics off the top of my head for the last time I tried removing a few systemd components only to find hard dependencies on each other.
This is not how Linux needs to be. I agree the old ways sucked. I don't want to go back to them either. But neither do I want to be locked into the big ugly mess that systemd turned into. I want clean, modern, well-designed components that are independent of each other.
Across companies, jobs, and hobbyist collaborators - I've never met someone who didn't at least like systemd, if not sing its praises. It made so many incredibly painful things about Linux administration disappear.