Hey! I'm one of the RWX cofounders. We don't have a self-hosted option yet, but we're interested in opening it up in the future. Our infrastructure is more complex to run than normal self-hosted runners infrastructure, due to extensive caching and provisioning optimizations.
There's a difference between small scale CI and large scale CI.
Small scale: a project is almost small enough to run the build and tests locally, but you still want to have a consistent environment and avoid "works on my machine" problems.
Large scale: a project is so large that you need to leverage remote, distributed computing to run everything with a reasonable feedback loop, ideally under 10 minutes.
The opposite ends of the spectrum warrant different solutions. For small scale, actually being able to run the whole CI stack locally is ideal. For large scale, it's not feasible.
> A CI system that’s a joy to use, that sounds like a fantasy. What would it even be like? What would make using a CI system joyful to you?
I spent the past few years building RWX[1] to make a CI system joyful to use for large scale projects.
- Local CLI to read the workflow definitions locally and then run remotely. That way can you test changes to workflow definitions without having to commit and push.
- Remote breakpoints to pause execution at any point and connect via ssh, which is necessary when running on remote infrastructure.
- Automatic content-based caching with sandboxed executions, so that you can skip all of the duplicative steps that large scale CI otherwise would. Sandboxing ensures that the cache never produces false positives.
- Graph-based task definitions, rather than the 1 job : 1 VM model. This results in automatic and maximum parallelization, with no redundancy in setup for each job.
- The graph based model also provides an improved retry experience, and more flexibility in resource allocation. For example, one task in the DAG can crank up the CPU and memory without having to run more resources for downstream tasks (steps in other platforms).
We've made dozens of other improvements to the UX for projects with large build and test workflows. Big engineering teams love the experience.
Sounds good, but it's still YAML and shell scripts. It's not even close to ideal.
A custom lazy, typed functional language that doesn't differentiate between expressions and "builds" would be much better. Even better if you add "contexts", aka implicit tags under values for automatic dependency inference. Also do "serializable bytecode", and closing over dependencies of thunks efficiently like Unison does for great distrubuted builds.
And it would be pretty easy to add a debugger to this system, same logic as doing "import"
Nix gets somewhat close, but it misses the mark by separating the eval and build phases. It having terrible documentation, 1332432 ways to do the same thing, not properly separating the nixpkgs/nix divide, and nixpkgs being horribly, but still insufficiently abstracted also doesn't help.
Also, I'm not sure why you posted this comment here, as there is nothing that prevents you from writing a Radicle CI adapter that can handle huge repositories. You can reference the bare git repo stored in the Radicle home, so you just need to be able to store the repo itself.
Every time there's yaml, you can use dhall and compile to json instead. You get typing and strictness that way - regardless of whether the service allows it internally.
We're a CI/CD platform with an entirely differentiated execution model. Rather than parallelizing jobs on VMs, we provide a serverless interface using a graph of tasks. We then have automatic, content-based caching, which eliminates the large amount of redundant executions that CI platforms have today. You can test changes from a local CLI so that you don't have to commit and push to run. We've made substantial improvements to every aspect of the developer experience.
We're looking for backend-focused engineers who excel at building developer tools. Our engineers are effectively product managers as well and work highly autonomously.
GitHub Actions should use a lockfile for dependencies. Without it, compromised Actions propagate instantly. While it'd still be an issue even with locking, it would slow down the rollout and reduce the impact.
Semver notation rather than branches or tags is a great solution to this problem. Specify the version that want, let the package manager resolve it, and then periodically update all of your packages. It would also improve build stability.
Also don't het GH actions to do anything other than build and upload artifacts somewhere. Ideally a write only role. Network level security too no open internet.
Use a seperate system for deployments. That system must be hygienic.
This isn't foolproof but would make secrets dumping not too useful. Obviously an attack could still inject crap into your artefact. But you have more time and they need to target you. A general purpose exploit probably won't hurt as much.
A signed commit [0] might be good for internal devops stuff (e.g. "yes, we really do want this version in production") but unfortunately that's not gonna work for pulling in third-party tooling, since most won't use it.
The version numbers aren't immutable, so an attacker can just update the versions to point to the compromised code, which is what happened here. Commit hashes are a great idea, but you still need to be careful: lots of people use bots like Renovate to update your pinned hashes whenever a new version is published, which runs into the same problem.
Yes, you're right. I wasn't able to double-check as the repo was deleted at the time. That said, AIUI making the tags read-only would still often be vulnerable to semantic-version exploitation.
Since they edited old tags here … maybe GitHub should have some kind of security setting a repo owner can make that locks-down things like old tags so after a certain time they can't be changed.
Checksums are a good use case, where you’re not worried about any sort of forged collisions and you want to calculate the checksum as fast as possible. Although cryptographic hashes are fast enough that there likely aren’t too many use cases where the difference in speed is meaningful.
If you don't care about forged collisions, what makes a secure-ish checksum bette than a faster insecure checksum? Both are likely to result in a different value for bit-flipped input, which is the point.
Note that you said “likely” and not “equally likely”. Therein lies your answer.
There are applications where hashes are used as identifiers, where it would be impossible to use the original data to resolve possible collisions. One example would be in RTTI (Run Time Type Information), when you want to check if two objects (of unknown-at-compile-time type) are instances of the same type. The only performant way to achieve this is if the compiler emits each type’s RTTI with a unique integer key to efficiently compare against, and this key must be unique across all object files and shared objects. In practice, this is done using hashing. If there is a collision, then the program behavior is undefined, so it is ideal to minimize probability of collision. You can see this issue in the Rust compiler project where there is ongoing discussion about how to handle this hashing, trying to balance compiler speed and collision likeliness:
Another scenario would be like what Backtrace does: a hash is used to identify certain classes of errors in application code. The cost of analyzing each error is quite high (you have to load objects into memory, source maps (if available), perform symbolification, etc), so comparing any two errors every time you want to perform any metrics or retrieve associated data (like issue tracker identifiers) would reduce the performance to a point that any such application would be too slow to be useful. So a lot of thought was given to producing fast, non-cryptographic hashes (these hashes aren’t shared across tenants, and no customer is going to go out of their way to work out initialization parameters so they can be their own adversary) with a known likely hood of collision. You can read about the hash (UMASH) here:
Well, also notice that I said "bit-flipped input." I.e., the use case I believed to be talking about and responding to was data corruption detection (which is also associated with "checksum" as a term of art, as opposed to a generic "hash"). I agree that hashing for identity is a different use case than checksumming for data integrity.
FWIW I think commonly used checksums are basically ideal ("equally likely" to come to a different value) on single-bit flipped inputs, but don't quote me on that.
That’s a good question. A cryptographic hash must not leak any information other than “yes” or “no”. A checksum-type hash might have an additional property where data that only has a single bit flipped might hash to something close to the original hash, allowing you to refine your collision attack by knowing if you’re hot or cold.
For a checksum, this isnt a bad thing and actually could be good.
A new CI/CD platform. Local CLI (run without git push), remote debugger, automatic content-based caching, DAG-based definition, and dynamic tasks. It’s taken a lot of work to build it, but I’m really excited about how well it’s working. https://rwx.com/mint
I haven't tried this yet, but I wanted to say I think there's a lot of potential in this space. There's so much friction with the current popular solutions...and yet it's so hard to justify trying some of the newer and less popular ones.
I wish you luck because there are a lot of good ideas in here. Running locally and remote debugger are the most exciting to me.
I like the idea, but I have used Azure Pipelines, and the key differentiating it from GitLab.ci is its "service connection" that enables seamless integration with third-party services and a wide range of tasks (PowerShell, pscore, az, docker, etc).
I wish your dependency semantics yaml could be converted to Azure pipeline task, job, and stage similar to how typescript converts to javascript.
I am saying this because there are very complicated infrastructural problems that hundreds of people work on at big companies when creating a ci/cd I see you are not focusing on them yet but that DAG logic could be applied to existing azure pipeline yaml.
This seems really neat! I'm a huge proponent of incremental CI for any project that gets big enough to care about test speed.
I'd be pretty worried about an abstraction ceiling on the caching here relative to something like Bazel, though. Like would Mint get mad if I generated 4000 dynamic tasks?
Out of interest: CircleCI also has flaky test detection, rerun failed tests etc. Or did you mean that the CI that you were moving towards could benefit from Captains features?
I didn't make myself clear I think. You mention that you love Captain, and wish you had access to it when migrating away from CircleCI. I'm interested in understanding the specific reason(s) for your love for Captain's features, that seem to also be available in CircleCI?
I don’t understand why any of that indicates that a 70 year old couldn’t have created this, unless you think 70 year old people aren’t capable of doing that.
The age isn't really the inconsistency, it's the "only just learning to code" aspect. My first ever webpage was certainly not hosted behind Cloudflare. (I'd also expect beginner resources to point to something like GitHub Pages instead)
- local CLI instead of git push to run
- graph-based task definitions with automatic distributed execution, instead of the job/step abstraction
- automatic content-based caching to skip unnecessary executions (happens a lot in CI pipelines)
- container-based runtime (instead of proprietary base images) without using docker directly (too slow)
There are a lot of other ways to improve the developer experience. Happy to chat with anybody interested, I'm dan@rwx.com