Yes, you are correct - SourceFS also caches and replays build steps in a generic way.
It works surprisingly well, to the point where it’s hard to believe until you actually see it in action (here is a short demo video, but it probably isn't the best way to showcase it: https://youtu.be/NwBGY9ZhuWc?t=76 ).
We intentionally kept the blog post light on implementation details - partly to make it accessible to a broader audience, and partly because we will be posting gradually some more details. Sounds like build caching/replay is high on the desired blogpost list - ack :-).
The build-system integration used here was a one-line change in the Android build tree. That said, you’re right - deeper integration with the build system could push the numbers even further, and that’s something we’re actively exploring.
We hear you on the “we want more technical blogs” part - they’ll be coming once we get a breather. We kept this first post high-level to reach a broader audience. Thanks for reading!
Obviously this is the best-case, hyper-optimized scenario and we were careful not to inflate the numbers.
The machine running SourceFS was a c4d-standard-16, and if I remember correctly, the results were very similar on an equivalent 8-vCPU setup.
As mentioned in the blog post, the results were 51 seconds for a full Android 16 checkout (repo init + repo sync) and ~15 minutes for a clean build (make) of the same codebase. Note that this run was mostly replay - over 99 % of the build steps were served from cache.
Do you have any technical blog post how the filesystem is intercepting and caching build steps? This seems like a non-obvious development. The blog alludes to a sandbox step which I’m assuming is for establishing the graph somehow but it’s not obvious to understand where the pitfalls are (eg what if I install some system library - does this interception recognize when system libraries or tools have changed, what if the build description changes slightly, how does the invalidation work etc). Basically, it’s a bold claim to be able to deliver Blaze-like features without requiring any changes to the build system.
Please fill in this form: https://www.source.dev/demo . We’re prioritizing cloud deployments but are keen to hear about your use case and see what we can do.
You’re absolutely right - SrcFS and EdenFS were inspirations for SourceFS.
The challenge with those systems is that they’re tightly coupled with the tools, infrastructure, and even developer distros used internally at Google and Meta, which makes them hard to generalize. SourceFS aims to bring that “Piper-like” experience to teams outside Google - but in a way that works with plain Git, Repo, and standard Linux environments.
Also, if I’m not mistaken, neither SrcFS nor EdenFS directly accelerate builds - most of that speed comes from the build systems themselves (Blaze/Buck). SourceFS goes a step further by neatly and simply integrating with the build system and caching/replay pretty much any build step.
The Android example we’ve shown is just one application - it’s a domain we know well and one where the pain is obvious - but we built SourceFS in a way where we can easily integrate with a new build system and speed up other big codebases.
Also you’re spot on that this problem mostly affects big organizations with complex codebases. Here without the infrastructure and SRE support the magic does not work (e.g. think the Redis CVE 10.0 of last week or the AWS downtime of this week) - and hence the “talk to us”.
We plan to gradually share more interesting details about how SourceFS works. If there’s something specific you’d like us to cover - let us know - and help us crowd source our blogpost pipeline :-).
It's a shame that AI is ruining certain phrases, the "You’re absolutely right" was appropriate but I've been trained reading so many AI responses to roll my eyes at that.
Hey everyone. I’m Serban, co-founder of Source.dev. Thanks for the upvotes and thoughtful discussion. I’ll reply to as many comments as I can. Nothing means more to an early-stage team than seeing we’re building something people truly value - thanks from all of us at Source.dev!
While I’m sure it’s much more advanced, out of interest is this similar to the Python tool ‘fabricate’, which would use strace to track all files a program read, and wrote?
source.dev | Senior Full-stack and Android Platform Engineers | UK, EU, IN | ONSITE or REMOTE
source.dev is transforming software development and updates for the device ecosystem.
We are building an AI-native DevOpsSec platform that’s vertically integrated with device codebases, to accelerate development, reduce maintenance costs, and deliver frequent, bug-free software updates.
Our mission is to simplify the complexities of developing, deploying, and maintaining software for Android and Linux-based devices — serving everyone from System on Chip (SoC) companies to OEMs. We empower our partners to launch devices faster and maintain them for longer.
We are founded by a team of former Googlers with deep expertise in Android and Linux, and backed by leading venture capital. If you’re passionate about making a real world impact we’d love to chat to you!
We're seeking:
- Senior Full-stack Engineer strong at Python/FastAPI and front-end frameworks like React, Next.js. You will be responsible for building and optimizing the user-facing side of our platform, working closely with our DevOps and Platform engineers to explore possibilities and enhance existing workflows.
- Android Platform Engineer experienced with at least some of AOSP, Android frameworks, BSPs and HALs, as well as modern programming languages like Rust. You will collaborate closely with former Google Android engineers to expand Android OS tooling, from builds to testing and emulation, while integrating it with our DevOpsSec platform.
These positions offer a unique opportunity to work on cutting-edge technologies and be part of one of the world’s leading Android teams, providing immense potential for learning and growth.
To apply or ask questions, email us at careers at source dot dev.
We intentionally kept the blog post light on implementation details - partly to make it accessible to a broader audience, and partly because we will be posting gradually some more details. Sounds like build caching/replay is high on the desired blogpost list - ack :-).
The build-system integration used here was a one-line change in the Android build tree. That said, you’re right - deeper integration with the build system could push the numbers even further, and that’s something we’re actively exploring.