The more I get problems with different machinery (chainsaw, raider lawnmower, waterpump, powerwasher) I've really started thinking more and more how every piece of equipment should automatically come with a service manual included which would also have all the bits and pieces clearly stated and catalogued. I've seen some of these manuals not being a detailed as I hoped. For context, I'm in Europe.
For an example I needed new lifting chain for the raider lawnmower and it was an ordeal finding the correct chain and making it correct length without such manual. With manual I'd expect it to take 5 minutes to find the correct piece, it's size, lenght and required stress endurance level.
It's 2023, doesn't have to be physical (or could for some extra buck), but at least have one available on your website.
I'm thinking about starting to ask for it for everything I buy... Not really hoping to get it every time but still...
But it's not that simple, for an example I have older personal projects as well which I'm unable to run without doing considerable changes to the code after moving to M1 chip.
Just because I'm unable to run older versions of node and thus unable to run some of the dependencies and thus need to update the dependencies and then the code etcetc.
Yes, I could use docker and containers...
Yes, I could use Rosetta...
But to answer your questions:
1. It took around 1-2 months of experienced DevOps works (1-2 devops), this included testing and the actual switchover to new AWS infra.
2. We actually used outside contractors (actual k8s gurus) to do the work. But the decision was made in-house by myself. I'm no k8s guru, hell, I'm not even devops, but I have enough experience with the "big guys" so the choice was calculated from technical perspective, considering our growth needs and technical needs for now and for the future.
3. I haven't had any reasons to regret the decision, but I knew that before. I knew AWS offered the things we need and how to use them and what it's roughly going to cost, so this wasn't really a surprise. I did my due diligence.
4. Heroku bills were around 5$k/mon... AWS ones are in 5 digit, but it was expected because we started to use so much more services and resources than we did (or could) in Heroku. We also expanded into two new countries (one of the main reasons the switchover was done) so the costs are sadly not directly comparable. I can say that the new costs are in expected ranges and I'm happy with what we get for the money.
5. We run a eCommerce site with monthly GMV of 1m€+ and around 700k monthly unique visitors. Our stack is (per country) 3 nodejs services, PostgreSQL, redis and some frontend services. Main cost factors are the database(s) and CloudFront.
6. These decision have to made per use-case, per project and taking into consideration your actual infrastructure needs and budget limits. I don't think there isn't anything AWS can't do, but is the cheapest? Of course not. Hell, there have been plenty of topics on this very page about moving your infrastructure to your own bare metal and saving hundreds of thousands per month.
Think through what are the issues you are currently having (ie, for us it was the Herokus stack limitations, couldn't get http2, couldn't do custom monitoring / alerting, no access to LBs, no scalable DB hosting, the need to easily roll out new countries without having to do too much of manual work, weirdly high cost of some services, like redis for an example). If your problems are in the wallet, you need to consider this as your first priority and find a provider which will meet your technical needs with best price. All the big guys also have cost calculators available which will allow you to get an estimation on what you would be paying. Take time, this isn't an easy decision and it will affect you a lot in the future as well, so you don't want to get it wrong to be stuck with another set of issues.
But the whole point is that the UX of the website is perfect. It has nothing to do with what the webapage is built on.
It could be built on PHP, it could be Java, it could be node, it could ASP.NET but if it's fast and with perfect UX like Mcmaster.com, what's the difference?
It is definitely possible to build such a website using all the mentioned languages, they have happened to have chosen ASP.NET and have done it well, but I wouldn't blindly throw everything else under the bus with such a comment.
And they aren't clear of "fancy web shenanigans" as well, they use socket.io and their website is unusable without Javascript so there's a lot of going on and this isn't your typical static website anyway.
So I think your sarcasm is a bit out of place in this instance.
And yes, I'm biased as I've been node.JS developer for a long time but I don't agree that the languages are the ones building shitty UX webpages that are slow... it's the developers that do that.
We left Heroku last spring with a project consuming around 5k$/mo worth of Heroku services.
Main reason for the move was pretty simple, we just needed more control over our own infrastructure and Heroku wasn't able to offer that. Things like HTTP2 support, access to load balancer configurations, timeout settings, different auto-scaling options and better monitoring are first things that come to mind.
We are now using AWS directly (EKS, RDS, Lambda etc) and even though the move itself did cost a bit, I wouldn't say monthly costs went up too much (but it's bit hard to compare as we're using more services at AWS and scaled up right after migration).
Basically, we just grew out of Heroku.
And personally I wouldn't choose them again even if opportunity appeared.
Not sure how much use it would be for Tallinn anyway, you have Wifi in your pocket anyway so that really isn't a deciding factor usually (unless you are traveling through and don't want to get a temp SIM for mobile internet).
If I'd have to guess, people pick cafes to work from here based on the location, atmosphere and food.
Could literally work under a tree in a forest here if you have enough battery.
If we exclude smaller scripts then one thing that stands out is a small CLI application (written in nodeJS) which takes all emails from Gmail marked with a special label and uploads their attachments to special Dropbox folder for my company accountant.
So forwarding invoices to my accountant is as easy as throwing a special label onto the email and boom, done.
Was totally worth automating, been using it for like two years now.
Moved a project with around 600k monthly users from Heroku / Google (split setup) to full AWS setup.
Whole process took around 3 months, that was from start of creating AWS account to finish when all production environments were running on AWS and Heroku was "shut down". There was some planning ahead of this as well, so actual time varies.
Heroku was heavily limiting platform (for example, they didn't and still don't support http2) and we needed more control over our infrastructure to support further growth without paying enormous costs (for example, redis prices in Heroku are just mind-blowing).
Also as we were about to open few new markets, Heroku would have required a lot of manual work to get everything working, something which is really, really simple with Kubernetes.
Our monthly costs did go up vs what we had at Heroku at that time, but we're getting a lot more control and bang out of the buck.
Regarding convincing stakeholders, you really need to have good reasons to do it. These kind of switches are not cheap nor easy and come with bunch of risks. The easiest thing to sell is always pricing, but in that case you have to show calculations (big guys like AWS and Google have pretty decent calculators you could use) which show the switch is worth it.
As I was moving from small player (Heroku) to a big player (AWS) I also had other good reasons (better CI, better logging, better performance overview, more control in general). So it really improved a lot of things for the developers, devops and users.
What are you using for compute EKS and Fargate? We just helped someone switch from Heroku to AWS and they dropped their cloud costs by 67%. This is using ECS + EC2. Fargate is typically 2x more expensive.
I probably should have clarified that the extra cost was expected as we did a lot more in AWS than we could in Heroku, we used the switch to start using bunch of stuff AWS offers like Lambda functions, CloudFront, RDS etc. Stuff that we just didn't (and couldn't) use on Heroku, thus didn't pay for it.
As the purpose of the switch was to get more control, features and out of the Herokus "black box", higher costs were expected and perfectly normal.
As other have asked, it would be nice to get some clarification a bit on what exactly do you mean by "homesteading?". Did you buy a full blown farm and are now keeping cows? Is there a specific reason you didn't want to move to countryside but still keep your tech-job remotely?
I can share some of my own experience, as I'm a second year bee-keeper and CTO with around 16 years of experience. So I've chosen to share my life between a excellent tech job & a city apartment and a country side house (2 hour drive from my apartment).
We decided to start bee-keeping with my wife two years ago because we both felt the need to be in the nature. And bee-keeping has really been a great choice because it doesn't need your attention 24/7/365. During the ~4 winter months we have here in Estonia, I barely go there, maybe 2-3 times. So I still have the freedom to travel or pursue other hobbies if I'd like to. And during the sprint - summer - autumn, I only need to be there once a week which also isn't a fixed day, so I can play around my work schedule nicely. Sometimes I even go there during workdays and just do some remote work from there, in the middle of the woods where the only sounds you can hear is wind blowing and birds chirping. I'm obviously there a lot more than that, but this is the bare minium, this is the "I HAVE TO" and if the weather sucks or I'm busy otherwise, I don't have to go more than that.
And even though it's sometimes physically hard and tiring, it has had a enormous positive effect on my life. Doing something with your own hands that you can see and touch is a very different kind of happiness than the one my work offers me.
But all this comes with a "but". I mostly grew up on the country side, even more so, I'm actually keeping my bees at my grandmothers house (where she no longer lives due to her age) where I spent most of my summers during childhood. So I knew what was waiting for me regarding up-keeping the house & land.
What I'd suggest is to do the things that make you happy not the things that others are doing. If you don't like the physical work, don't do it. If you'd just like to enjoy nature, just get a cabin the in woods where you can go from time to time.
I'm in a lot better place with my good paying tech job and half-time bee-keeping because I also have the funds to invest into the house & land and make it more livable for the future. If I would have just quit my job and moved there full time, I'd be struggling to make the ends meet, let alone actually build houses there and develop the place for the better.
Tools, resources and materials (wood, metal, sand, gravel etc) and help from professionals all cost money. It's a lot easier to earn that money in tech than with farming.
But in case the society as we know it today collapses, I still have the skills, tools and land to live off, but it hasn't yet, so no point in going full off-grid yet.
As some of the commenters here have already voiced, I also struggle a bit to understand the issue with using relational databases if the data is actually relational. Which, for probably like 95% projects it is.
The argument that "it will slow us down" seems so vague, I can't really see how defining the schemas and relationships slow you down. You need to know what data you are dealing with anyway even when prototyping, how else would you create views for the users? Tables? Are profile fields for users also open ended? I can add any number of fields with random values?
And "Your database doesn't do much for you?" So what exactly does NoSQL database do for me that PostgreSQL (or MySQL) won't? I'm really curious, I'm not attacking anyone, I just want to know, maybe I've been using the wrong database my whole life.
But I think that database as anything else in your project should be selected by best fit not by mere "it's a cool buzzword, lets use that". Most of the projects out there have relational data and should be using relational database because it fits the core data model.
I don't have anything against NoSQL databases, they obviously have their usage and place but I have against choosing them because they are "cool" and "fast to prototype on" if the project itself has 100% relational data.
Also the tools fastest to prototype with are the ones you know best, it doesn't mean that tool is best fit for the project.
I'm not sure OC is asking for NoSQL - just a better experience with SQL migrations.
The relational model always requires a judgement call about how normalized to go. If I have a deep nested json object of data, you could argue to define all the relations and normalize. But then to query the data you now have an 6-way join and the query planner starts to struggle a bit. And then if you decide to change that structure, you now have a very tricky migration script to run.
Whereas if you just kept it as a json object, you have avoided a lot of unnecessary pain, and you can just write some quick re-mapping code in the application layer to handle the older version of data, and you can actually start storing data in the new schema immediately. Then you can scan over the collection and modify the existing data of the old schema, and remove your re-mapping layer.
So yes, your data has been relational the whole time, but the query complexity and migration complexity increased dramatically. It became very difficult to change the model.
The relational model is essentially a constraint on how you need to structure your data in order to allow relational algebra to be used to help optimize query plans. When you release these constraints, you find things like Datalog that provide purer ways to represent your data and relationships, in the sense that they can map closer to the real-world - but then this comes at the cost of automated query optimization.
So the fact that relational data models are usually less representative of the real-world because of adherence to the relational model, usually denormalized to optimize the underlying database engine, and difficult to migrate, create complexity and slow down shipping some features (although they can also speed up many others greatly - like analytics).
That being said, SQL dbs are probably the best dbs we have today for solving business problems, and I think the migration and modeling problems could be solved with better tooling.
> You need to know what data you are dealing with anyway
Exactly. At some point, somewhere, you need to understand your data and how different attributes or objects relate to one another. You can do this in a variety of ways, but if you don't understand this you don't understand what you are building.
For an example I needed new lifting chain for the raider lawnmower and it was an ordeal finding the correct chain and making it correct length without such manual. With manual I'd expect it to take 5 minutes to find the correct piece, it's size, lenght and required stress endurance level.
It's 2023, doesn't have to be physical (or could for some extra buck), but at least have one available on your website.
I'm thinking about starting to ask for it for everything I buy... Not really hoping to get it every time but still...