It sometimes blows my mind how questions which essentially boil down to "How do I best manipulate you for personal gain?" can be asked in such an unabashed fashion.
> They mostly seem to be into their own thing. How can I energize engagement?
You jest, but the best way is to give the nerds only a feed and don't allow any other organizing principles for navigating the content. Lured by the promise of something interesting sometimes, people who are trying to follow their curiosity for math/archaeology will stumble into threads about corporations and products, throw ideas out in public for free, get sucked into politics, etc etc
Introducing a separate charge specifically targeting those of your customers who choose to self-host your hilariously fragile infrastructure is certainly a choice.. And one I assume is in no way tied to adoption/usage-based KPIs.
Of course, if you can just fence in your competition and charge admission, it'd be silly to invest time in building a superior product.
We've self-hosted github actions in the past, and self-hosting it doesn't help all that much with the fragile part. For github it is just as much triggering the actions as it is running them. ;) I hope the product gets some investment, because it has been unstable for such a long time, that on the inside it must be just the usual right now. GitHub has by far the worst uptime of any SaaS tools we use at the moment, and it isn't even close.
> Actions is down again, call Brent so he can fix it again...
We self host the runners in our infrastructure and the builds are over 10x faster than relying on their cloud runners. It's crazy the performance you get from running runners on your own hardware instead of their shared CPU. Literally from ~8m build time on Gradle + Docker down to mere 15s of Gradle + Docker on self hosted CPUs.
This! We went from 20!! minutes and 1.2k monthly spend on very, very brittle action runs to a full CI run in 4 minutes, always passing, by just by going to Hetzner's server auction page and bid on a 100 euro Ryzen machine.
After self hosting our builds ended up so fast, that we were actually waiting for was GitHub scheduling our agents, rather than it being the job running. It sucked a bit, because we'd optimized it so much, but we on 90th percentile saw that it took 20-30 seconds for github to schedule the jobs as they should. Measured from when the commit hit the branch, to the webhook begin sent.
My company uses GitHub, GitLab, and Jenkins. We'll soon™ be migrating off of GitLab in favor of GitHub because it's a Microsoft shop and we get some kind of discount on GitHub for spending so much other money with Microsoft.
Scheduling jobs, actually getting them running, is virtually instant with GitLab but it's slow AF for GitHub for no discernable reason.
lmao I just realized on this forum writing this way might sound like I own something. To be clear, I don't own shit. I typically write "my employer", and should have here.
I don't know, maybe it's a compliment. Wood glue can form bonds stronger than the material it's bonding. So, the wood glue in this case is better than the service it's holding together :)
I tend to just rely on the platform installers, then write my own process scripts to handle the work beyond the runners. Lets me exercise most of the process without having to (re)run the ci/cd processes over and over, which can be cumbersome, and a pain when they do break.
The only self-hosted runners I've used have been for internalized deployments separate from the build or (pre)test processes.
Aside: I've come to rely on Deno heavily for a lot of my scripting needs since it lets me reference repository modules directly and not require a build/install step head of time... just write TypeScript and run.
We choose github actions because it was tied directly to github providing the best pull-request experience etc. We actually didn't really use github actions templating as we'd got our own stuff for that, so the only thing github actions actually had to do was start, run a few light jobs as the CI was technically run elsewhere and then report the final status.
When you've got many 100s of services managing these in actions yaml itself is no bueno. As you mentioned having the option to actually be able to run the CI/CD yourself is a must. Having to wait 5 minutes plus many commits just to test an action drains you very fast.
Granted we did end up making the CI so fast (~ 1 minute with dependency cache, ~4 minutes without), that we saw devs running their setup less and less on their personal workstations for development. Except when github actions went down... ;) We used Jenkins self-hosted before and it was far more stable, but a pain to maintain and understand.
In 2023 I quoted a customer some 30 hours to deploy a Kubernetes cluster to Hetzner specifically to run self-hosted GitHub Actions Runners.
After 10-ish hours the cluster was operational. The remaining 18 (plus 30-something unbillable to satisfy my conscience) were spent trying and failing to diagnose an issue which is still unsolved to this day[1].
Am I right in assuming it’s not the amount of payment but the transition from $0 to paying a bill at all?
I’m definitely sure it’s saving me more than $140 a month to have CI/CD running and I’m also sure I’d never break even on the opportunity cost of having someone write or set one up internally if someone else’s works - and this is the key - just as well.
But investment in CI/CD is investing in future velocity. The hours invested are paid for by hours saved. So if the outcome is brittle and requires oversight that savings drops or disappears.
I use them minimally and haven't stared at enough failures yet to see the patterns. Generally speaking my MO is to remove at least half of the moving parts of any CI/CD system I encounter and I've gone a multiple of that several times.
When CI and CD stop being flat and straightforward, they lose their power to make devs clean up their own messes. And that's one of the most important qualities of CI.
Most of your build should be under version control and I don't mean checked in yaml files to drive a CI tool.
The only company I’ve held a grudge against longer than MS is McDonalds and they are sort of cut from the same cloth.
I’m also someone who paid for JetBrains when everyone still thought it wasn’t worth money to pay for a code editor. Though I guess that’s again now. And everyone is using an MS product instead.
I run a local Valhalla build cluster to power the https://sidecar.clutch.engineering routing engine. The cluster runs daily and takes a significant amount of wall-clock time to build the entire planet. That's about 50% of my CI time; the other 50% is presubmits + App Store builds for Sidecar + CANStudio / ELMCheck.
Using GitHub actions to coordinate the Valhalla builds was a nice-to-have, but this is a deal-breaker for my pull request workflows.
I found that implementing a local cache on the runners has been helpful. Ingress/egress on local network is hella slow, especially when each build has ~10-20GB of artifacts to manage.
ZeroFS looks really good. I know a bit about this design space but hadn't run across ZeroFS yet. Do you do testing of the error recovery behavior (connectivity etc)?
This has been mostly manual testing for now.
ZeroFS currently lacks automatic fault injection and proper crash tests, and it’s an area I plan to focus on.
SlateDB, the lower layer, already does DST as well as fault injection though.
GH actions templates don’t build all branches by default. I guess it’s due to them not wanting the free tier to use to much resources. But I consider it an anti-pattern to not build everything at each push.
They already charge for this separately (at least storage). Some compute cost may be justified but you'd wish that this change would come with some commitment of fixing bugs (many open for years) in their CI platform -- as opposed to investing all their resources in a (mostly inferior) LLM agent (copilot).
- github copilot PR reviews are subpar compared to what I've seen from other services: at least for our PRs they tend to be mostly an (expensive) grammar/spell-check
- given that it's github native you'd wish for a good integration with the platform but then when your org is behind a (github) IP whitelist things seem to break often
- network firewall for the agent doesn't seem to work properly
raised tickets for all these but given how well it works when it does, I might as well just migrate to another service
I don't work for Microsoft (in fact, I work for a competitor), and I think it's totally reasonable to charge for workflow executions. It's not like they're free to build, operate, and maintain.
You don't trust devs to run things, to have git hooks installed, to have a clean environment, to not have uncommitted changes, to not have a diverging environment on their laptop.
Actions let you test things in multiple environments, to test them with credentials against resources devs don't have access to, to do additional things like deploys, managing version numbers, on and on
With CI, especially pull requests, you can leave longer running tests for github to take care of verifying. You can run periodic tests once a day like an hour long smoke test.
CI is guard rails against common failure modes which turn requiring everyone to follow an evolving script into something automatic nobody needs to think about much
> You don't trust devs to run things, to have git hooks installed, to have a clean environment, to not have uncommitted changes, to not have a diverging environment on their laptop.
... Is nobody in charge on the team?
Or is it not enough that devs adhere to a coding standard, work to APIs etc. but you expect them to follow a common process to get there (as opposed to what makes them individually most productive)?
> You can run periodic tests once a day like an hour long smoke test.
Which is great if you have multiple people expected to contribute on any given day. Quite a bit of development on GitHub, and in general, is not so... corporate.
I don't deny there are use cases for this sort of thing. But people on HN talking about "hosting things locally" seem to describe a culture utterly foreign to me. I don't, for example, use multiple computers throughout the house that I want to "sync". (I don't even use a smartphone.) I really feel strongly that most people in tech would be better served by questioning the existing complexity of their lives (and setups), than by questioning what they're missing out on.
It seems like you may not have much experience working in groups of people.
>... Is nobody in charge on the team?
This isn't how things work. You save your "you MUST do these things" for special rare instructions. A complex series of checks for code format/lint/various tests... well that can all be automated away.
And you just don't get large groups of people all following the same series of steps several times a day, particularly when the steps change over time. It doesn't matter how "in charge" anybody is, neither the workplace nor an open source project are army boot camp. You won't get compliance and trying to enforce it will make everybody hate you and turn you into an asshole.
Automation makes our lives simpler and higher quality, particularly CI checks. They're such an easy win.
The runner software they provide is solid and I’ve never had an issue with it after administering self-hosted GitHub actions runners for 4 years. 100s of thousands of runners have taken jobs, done the work, destroyed themselves, and been replaced with clean runners, all without a single issue with the runners themselves.
Workflows on the other hand, they have problems. The whole design is a bit silly
it's not the runners, it's the orchestration service that's the problem
been working to move all our workflows to self hosted, on demand ephemeral runners. was severely delayed to find out how slipshod the Actions Runner Service was, and had to redesign to handle out-of-order or plain missing webhook events. jobs would start running before a workflow_job event would be delivered
we've got it now that we can detect a GitHub Actions outage and let them know by opening a support ticket, before the status page updates
That’s not hard, the status page is updated manually, and they wait for support tickets to confirm an issue before they update the status page. (Users are a far better monitoring service than any automated product.)
Webhook deliveries do suffer sometimes, which sucks, but that’s not the fault of the Actions orchestration.
I'm seeing wonky webhook deliveries for Actions service events, like dropping them completely, while other webhooks work just fine. I struggle to see what else could be responsible for that behaviour. it has to be the case that the Actions service emits events that trigger webhook deliveries & sometimes it messes them up.
You can always upload your own, it's pretty simple doing so in a reproducible manner using something like Packer, but even without it you can just boot a VM into a rescue system, write your OS of choice to the VM disk and reboot.
Technically there are two clients: The camera and whatever device is used to access the feed.
I can absolutely imagine an architecture where video can be streamed in an encrypted manner, or stored in encrypted time-stamped blobs, allowing the server to provide rough searching, and then the client can perform fine-grained scanning.
This obviously doesn't enable any kind of processing of the video data on the server side, and doing it on the receiving client would require the feed to be active This means that any kind of processing would almost necessarily have to happen on the sending device, which would probably increase the power and compute requirements by a lot.
Yeah, the entire point of this seems to be "we'll watch your baby monitor and provide alerts if something happens". That requires either processing on a server (as they do), processing on the uploading client (the camera), or having a receiving client which is constantly receiving that data and analyzing it to provide alerts.
The third option is unreliable because if that "client" (a desktop app, a phone app, etc.) dies, then the process stops working completely. The second option is unreliable because if you increase the cost of the camera then most users will buy the other camera because everyone is financially constrained these days.
That basically just leaves the first option as the only practical one at an appealing price point.
Doing a self-audit like this is actually an amazing idea. I consider and re-consider my choices every once in a while, but sitting down and doing an end-to-end write-up would put me a lot more at ease.
Like you, I also considered the implications of mixing TOTP into KeePass, but eventually landed on going all-in on the one database. It does mean raising the bar for keeping it secure, but it was already very high to begin with.
One thing I have considered is combining this all-in-one approach with an additional keyfile, which I could then share OOB to devices, effectively adding a second factor. I like the idea of using Yubikey or similar, but the fear of locking myself out is too great.
My team is worried about that too. We've been a java and spring shop for years. We're looking at micronaut, it's similar enough.
When I had someone from another team take a look at broadcom and what they could do to spring, they said the licenses are permissive, it will be fine. Likely not that simple.
- Shorter support windows, with longer support available for purchase (VMWare actually introduced this, but Broadcom can weaponize it)
- Then Enterprise Spring, which has additional features
- Then some other license shenaningans.
Hazelcast recently made the move where CVE security updates are only released into the OSS ecosystem quarterly - whereas the enterprise model gets them as soon as they're ready. In OSS, you have to rebuild and patch yourself.
That's a special kind of evil, which has Broadcom DNA all over it.
They're still technically Avago Technologies, just wearing the name of Broadcom after the acquisition in 2015-2016. Not sure if there's much of Broadcom left, beyond the name and what IP they had at the time which was not sold off, like they did with the IoT related IPs.
Taking a bunch of projects and making containers and flexible helm charts for them is kind of an interesting model. It’s what Redhat and Canonical do with raw Linux packages; they charge for premium support and even patches or extended support.
I was going through one of my clusters, I have two bitnami uses and they are both ‘building blocks’ I use Trino, which uses a metastore which uses postgresql and then some other package uses redis. It seems like both postgresql and redis could/would have containers and charts to install their stuff, where it breaks is the postgresql guys probably want to support “current” and not 4 major releases back, which is kind of normal to see in the wild.
It is kind of an interesting model, I’d love it if rancher or openshift or someone started to seriously compete. Shipping a Kubernetes in a box is nice but if they started packaging up the building blocks, that’s huge too.
Bitnami started out (2010? definitely since 2014) distributing virtual machine images (e.g. preconfigured LAMP stack server) and somehow inherited the official kubernetes helm repo several years ago, which even then, I think we all saw the writing on the wall.
Broadcom has always been about pure evil (cough capitalism cough), you just haven't been affected by it before. Ask anyone who's worked with their hardware...
So
Others have already provided good answers. I wouldn't classify it as evil if all they did was to stop maintaining the images & charts, I recognise how much time, effort and money that takes. Companies and open source developers alike are free to say "We can no longer work on this".
The evil part is in outright breaking people's systems, in violation of the implicit agreement established by having something be public in the first place.
I know Broadcom inherited Bitnami as part of an acquisition and legally have no obligation to do anything, but ethically (which is why they are evil, not necessarily criminal) they absolutely have a duty to minimise the damage, which is 100% within their power & budget as others have pointed out.
And this is before you even consider all the work unpaid contributors have put into Bitnami over the years (myself included).
It's also entirely fine if they delete these images to me. But not with a week of time frame, as originally intended.
And sure, we can go ahead and discuss how this being free incurs no SLAs or guarantees. That's correct, but does not mean that such a short time frame is both rude and not a high quality of offering a service. If I look at how long it would take us to cancel a customer contract and off-board those...
And apparently it costs $9 to host this for another month? Sheesh.
I agree. We do have mirrors setup, we do have observability into the images we use across the infrastructure. This has concluded we only have a minor issue with this move, wonderfully.
But, just butting users with "Just do this good practice" or "Just do this NOW" still is an uphill battle and will usually not cause the best effect with users. We're currently doing this while moving our 2-3 artifactories into one. If we just shut this stuff off because "You should have more control with your builds", there'd be a riot.
And sure, some people will still fail the migration no matter what. But such time frames are still punishing any but the most professional companies.
That's all in all the work I consider a good operations team to do. Make a stink, provide a migration path, be noticeable, and push people off of the old stuff. Just breaking things because "you should have" is burning trust by the mile.
This is not an accurate characterization of what's generating the outrage
The Path to Outrage is actually:
1. Launch HN with MPL licensing, "we <3 Open Source!11"
2. (a few moments later) Onoz, engineers cost money and that sweet VC juice dries up
3. echo BuSL > LICENSE; git commit -am"License update"; blog about "safeguarding our customers" or whatever
4. shocked pikachu face when users, who starting using the open source product, and maybe even contributed bug fixes or suggestions on how to make the community product better, lose their minds about having the rug pulled out from under them
Contrast this with:
1. Launch HN, asking for money to pay for the product
2. Potential customers will evaluate if the product adds enough value for the money requested
There is no step 3 containing outrage because everyone was on the same page from the beginning
> 2. (a few moments later) Onoz, engineers cost money and that sweet VC juice dries up
>
> 3. echo BuSL > LICENSE; git commit -am"License update"; blog about "safeguarding our customers" or whatever
In this case, it's a lot more nefarious. My boss has a list of companies Broadcom has literally sucked dry for money regardless if the company will make it 2 more years. Pretty much everything maintained by VMWare Tanzu and VMWare has to be considered a business risk at this point.
And I maintain, I'm not even mad that the free images go away. I'm saying it's unprofessional and rude how they are being removed. Which however isn't surprising with Broadcom per the last point.
And sure, the answer is to do it all in-house. But the industry has saved a lot of manpower and made tremendous progress by not doing that and sharing effort for a long time.
> The evil part is in outright breaking people's systems, in violation of the implicit agreement established by having something be public in the first place.
Something, something, sticking your hand in a lawnmower and expecting it not to be cut off.
that's an assumption, but Broadcom is most likely using open source software in all of their apps. So they do have a moral to also give something back. So just saying it's fair that they don't want to provide something for free anymore isn't really all that fair.
Oh don't get me wrong, my claim is that they are not even clearing the absolute lowest bar when it comes to their stewardship of the Bitnami repositories: Do no harm.
The images are currently in Docker Hub. If $9/month (or $15, not 100% sure if $9 includes organizations) to keep those images available is too much for Bitnami I'm sure there are many organizations who wouldn't mind paying that bill for them (possibly even Docker Hub itself).
Broadcom is deciding to host it on their own registry and bear the associated cost of doing so. Not sure what this has to do with sponsoring network egress
I suspect this might be helpful for minor integration challenges or library upgrades like others have mentioned, but in my experience, the vast majority of Rust compilation issues fall into one of two buckets:
1. Typos, oversights (like when adding new enum variants), etc. All things which in most cases are solved with a single keystroke using non-LLM LSPs.
2. Wrong assumptions (on my part) about lifetimes, ownership, or overall architecture. All problems which I very much doubt an LLM will be able to reason about, because the problems usually lie in my understanding or modelling of the problem domain, not anything to do with the code itself.
I tried the same about a month ago and was unable to get anywhere, probably due to being very new to the custom keyboard scene and having only a very vague understanding of how qmk/rmk/vial/keymaps fit together. Thanks for writing this up, I might give it another go now!
The custom keyboard bubble is really deep and filled with many, many different kind of setups which makes research not really easy.
To get you started:
- QMK is probably the most mature firmware for non-wireless devices and widely used. Probably the best starting point to get in touch with all the different concepts of keymap layers, one-shot modifiers, hold-tap-mechanics, ...
- Via/Vial is based on QMK and simply tries to provide a GUI for instant and live configuration of the keyboard, instead of doing it by code.
- ZMK is also very stable and designed for wireless usage (battery consumption) but also REALLY complex because of the toolchain (at least in my opinion)
- RMK, KMK, ... are all different kind of firmwares trying to do something different or better or just in a different language like rust or python. Most of the time these kind of firmware expect some kind of "basic knowledge". So I would start with the classics QMK or ZMK.
Thanks for writing all this down! I've written a lot of Rust, which was why I was drawn to RMK initially, but ran into a lot of the same headaches as the author, but without the perseverance to resolve any of them.
I did manage to get QMK working eventually, but without VIA/L the prospect of having to flash the two parts of my Ferris Sweep every single time I wanted to fine tune it, and having already spent a lot of time just getting that far, completely burned me out on the project.
I think impact can have a big influence on mechanical hard drive longevity, so it could be that the way the ServerPartDeals drives were sourced, handled or shipped compromised them.
Postgres upgrades were actually annoying the last time I did, where I had to explicitly import data from a previous version into the new one, instead of the software just automatically detecting that the data was a version behind and doing whatever to upgrade the format.
This would probably not have been as big of a headache, if it wasn't because it was running in a container, and was deployed as part of a separate project, meaning CloudNativePG (which probably handles this for you) was not an option.
reply