Reading the article I got the impression the big challenge is doing "personalzation" of the content at scale.
If it were "just" static pages, served the same to everyone, then it's pretty straightforward to implement even at the >300m users scale Netflix operates at. If you need to serve >300m _different_ pages, each built in real-time with a high-bar p95 SLO then I can see it getting complicated pretty quickly in a way that could conceivably justify this level of "over engineering".
To be honest though, I know very little about this problem beyond my intuition, so someone could tell me the above is super easy!
Whats the point of building & caching static pages if every single user gets their own static page... The number of users who will request each page is 1?
I dont think that's what they're doing. They seemingly cache some common templates and then fill in the dynamic placeholders per-request. So, their new architecture that has some sort of distributed in-memory cache allows for doing so much more efficiently than, presumably, doing the same queries over the network. The way I understand it, its essentially just a fancy SSR.
Not necessarily - SSI/ESI do http requests for each placeholder and fill it with an HTML fragment. They may just be doing some templating of whatever sort, from the kv store.
Yup. Content that may be read-only from a user's perspective might get updated by internal services. When those updates don't need to be reflected immediately, a CDN or similar works fine for delivering the content to a user.
When changes to data are important, a library like Hollow can be pretty magical. Your internal data model is always up to date across all your compute instances, and scaling up horizontally doesn't require additional data/infrastructure work.
We were processing a lot of data with different requirements: big data processed by a pipeline - NoSQL, financial/audit/transactional - relational, changes every 24hrs or so but has to be delivered to the browse fast - CDN, low latency - Redis, no latency - Hollow.
Of course there are tradeoffs between keeping a centralized database in memory (Redis) and distributing the data in memory on each instance (Hollow). There could be cases where Hollow hasn't sync'd yet, so the data could be different across compute instances. In our case, it didn't matter for the data we kept in Hollow.
Personalization could include showing the same cached content to many people. The personalization here could refer to which content is served from the CDN to which users.
I can’t imagine the number of editors is that large? Are we talking about hundreds or thousands? Taking a look at the site there can’t be that many users that need this low latency preview capability.
In a previous role we used their (first party) physical transfer appliances to upload ~600PB of video into S3. It was a complex logistical exercise to take it from the physical SANs, but the AWS specialists we worked with were great and it went without a hitch.
I don't want to be too negative, I generally agree with and am aligned with the content of the article, but this struck me as a really bad take:
> In my experience, “show me the data” is often a tactic employed by weak managers who don’t know how to hang as part of a design process.
I really don't understand how asking to see data and talk in facts, rather than opinions, is a bad thing? This take seems to be implying the "design process" is just a giant, strictly qualitative "appeal to authority" fallacy and anyone who doesn't "get it" is some kind of naive rube?
> I really don't understand how asking to see data and talk in facts, rather than opinions, is a bad thing? This take seems to be implying the "design process" is just a giant, strictly qualitative "appeal to authority" fallacy and anyone who doesn't "get it" is some kind of naive rube?
I don't think you're quite understanding the context when this happens and why it's a sign of a weak manager.
The team is discussing a new design. A disagreement comes up: what should the text on this button say? Both sides seem to have good arguments, but the manager doesn't know how to pick between them. What to do?
"Do we have any data on this? Maybe we should run an A/B test?"
But no A/B test is needed: this is text on a button that doesn't have a meaningful conversion rate, it's just about clarity. The data we gather will not be useful. The problem is the manager didn't want to be seen as picking sides or playing favorites, or just doesn't understand what's happening, so they moved their own responsibility to an "objective, external measurement" to avoid making a decision.
Generally, "data-driven" is a buzzword that means "we manipulate statistics to do what we wanted to do anyway".
Not having data is not the equivalent of only talking in opinions. I will tell you rhat gathering data can and usually is tedious in many things. If I have to measure every single thing to make any argument, I'm quitting. Looking and thinking in data isn't a bad idea, but if it's just used as a crutch in an argument, then what I think the OP is arguing is that their manager is using it as a tool to make decisions but likely only the decisions they disagree with. As an engineer, I know many times that changes I make are going to work better, sans some fancy chart. If they're not, I think they'll show up on a graph one way or another and I think it's also healthy to be told my assumptions are wrong sometimes and I will happily revert my code. But I've never seen any org make purely data driven decisions as it is truly difficult to do, and whoever is saying things like that probably is selectively using that argument to get their way.
The odds of "the data" being gathered in a rigorous enough manner to have any truth value is exceedingly low. It's mostly a tactic to avoid accountability, much like consultants and committees.
Although I do design things, I'm not a manager so maybe this is just outside my job description, but how do you "hang" during a design process? What does it mean? Leave it for bit and see? Honest question.
In my experience, a lot of the motivating factors for large enterprises mandating encryption at rest aren't about specific security controls. They will often hand wave in that direction, but as as the OP says in their post, without being able to describe a coherent threat model.
Instead a lot of motivating factors I've seen are about preventing various paths for "legitimate" data disclosure to third parties. For example, when data at rest is combined with additional requirements like "bring your own key" it means a subpoena or NSL needs to be served on the _first party who owns the data_ (as they need to provide the keys) and can't be served on just the cloud provider without the first party having at least visibility of it.
The "template" use case was a very common one from iframes/framesets in the early 2000s, and the markup often looked a lot like your desired example.
My memory's hazy, but I remember it becoming more and more complex to maintain as the web's security and performance model evolved, as you needed to manage and secure all of these disparate, discrete DOM trees in the same page context.
Wouldn't what you suggested just been "reinventing" frames/framesets if you added the ability to arbitrarily pull in templates via the network?
"Hard to write but easy for customers to deploy" is my guess. There are a bunch of very high-performance computing use-cases in finance (quant, HFT) that Java gets used for pretty routinely. That's a very attractive market to build primitives like databases for, they have very deep pockets, but you need to play in their ecosystem.
I've seen this "GC-less" Java in those use-cases quite a bit. From a conceptual design POV it's likely not the best approach, but there's a lot of sunk cost in that eco-system and a lot of trust and expertise where "Choosing a better language" is often several orders of magnitude more expensive.
> in my experience you provide the debt collections agency all supporting contracts and account documents for each loan
This might be true for first party lender to first party debt buyer, but it often falls apart after that first step.
The article mentioned the chain of debt ownership and that, in my experience, is where contracts and supporting docs are lost. Once you're buying debt 3 or 4 steps removed from the primary lender the documentation becomes sparser and sparser, which is why as the article rightly said, most of these "last mile" agencies rely on people's ignorance and use a "spray and pray" strategy for debt collection.
My experience, from a large primary card issuer in the US, was that most of the major, reputable debt purchasing agencies would only buy primary debt portfolios that had _contractual guarantees_ about the accuracy of the supporting contracts and docs. They'd spend a lot of time and effort reviewing them, but they would in turn outright _refuse_ any such provisions when THEY sold their debt to smaller agencies!
I'm interested in this line "Many media outlets are increasingly reliant on money from betting companies" as I'd be interested to know how reliant the Guardian was on this revenue. Is stopping this type of advertising a big impact on their top line?
I applaud the move either way, but stopping doing something that's 0.1% of your revenue vs something that is 25% are two very different things.
I doubt Guardian is as reliant as say Sky is. Betting is pretty much the primary driver of revenue for their sports coverage, and indeed football as a whole. It seems half of premier teams have some form of betting company on their shirt -- often aimed at those in the far east
Judging people and companies on motivations rather than actions is kind of a losing game. You can never really know for sure, and it’s easy to speculate in every direction.
It seems inevitable since even when actions are the same, intent really does matter. An adult inviting a child into their home is an action that happens every day, but intent is pretty important.
We should speculate in every direction while considering the motivations of a person or company, and we should weigh the likelihood and potential costs of being right or wrong about our suspicions when choosing how to react to events. Blinding ourselves to motivations (actual or possible) is a losing game which will only make us easier to manipulate.
> We should speculate in every direction while considering the motivations of a person or company, and we should weigh the likelihood and potential costs of being right or wrong about our suspicions when choosing how to react to events. Blinding ourselves to motivations (actual or possible) is a losing game which will only make us easier to manipulate.
Totally disagree, especially for companies. Companies are usually thousands of people. Any action was probably hotly debated, and some agreed and some opposed. It's just silly to treat a company like an individual person who has good motivations and bad motivations.
You do you, but I don't think making up motivations and then acting like they're true makes anyone less easy to manipulate.
If you run a successful blog or similar, you will start getting offers from gambling companies wanting to purchase advertising and articles. These are legitimate offers and they pay well - even if your content is very local or narrow, such as citizen journalism. Gambling is an extremely profitable business, and they want as many customers as possible. Expect their advertising to only grow.
They won't tell you, or give an exact figure, other than this nonsense:
> Guardian Media Group increasingly relies on contributions directly from readers, rather than advertising, for its income.
Which is no wonder they 'seem' to be continuously begging their readers to pay. Now after this, I should expect them to be begging even more aggressively, unless someone is funding them grants or what not, since they are still not telling us something or giving an exact figure.
Either way, one-sided publications like the Guardian is hardly worth paying for regardless.
The Australian operation of the Guardian has said recently that donations from readers are now more than 50% of the revenue. And they’re trying to increase that percentage further. Not sure about their UK or US operations.
That is quite phenomenal really - I don't think any other widely-used service (other than actual charities) operates on that basis. Though I don't really see my monthly payment (a whole $10, mind you) as a donation. But their spiel that convinces readers to do so is well-written, and hooked me in. Well that, and just a look at how bad the quality of most other online news sources was, seemingly particularly the ones operated by profitable companies.
I visited Chittagong a decade or so ago (as part of a non-profit working with child workers) and I can only emphasise what a visceral and horrifying thing this kind of ship breaking is to witness. 10,000's of people (as young as 6/7) working 24/7 in conditions that bear absolutely no respect for their health or safety.
One of the most morally repugnant parts of the Western legal system is the creation of the labyrinthine corporate structures used to to protect the ship owners for any liability as part of their ships being broken.
> ‘There’s also the Basel Convention, which is on the trans-boundary movements of hazardous wastes, such as end-of-life vessels.’ The problem is that a vessel only counts as hazardous waste once the company announces its intention to dispose of it. ‘It’s very easy for ship owners to simply claim that they’re on an operational voyage, sail all the way to India, Bangladesh and Pakistan, and only then declare that the vessel is actually heading for scrap. It makes it impossible to enforce that piece of law.
Any long-term commitments can be potentially worked around via bankrupcy. As long as we allow companies to take the full responsibility for consequences of their actions, and completely absolve their owners, the system will not work that great.
I'm sure you could just contrive an accidental sinking, or even better have the ship taken by "pirates" who happen to leave some cash in the lifeboats with the dispossessed.
Seems like you'd have to just outright ban selling it to foreign owners?
Otherwise someone will make a "legitimate" business in a country that doesn't ban selling to (non-compliant) scrappers, whose job in life is to purchase old ships and sell them to scrappers. Maybe hanging on to them for a bit first to make it slightly less blatant...
Is it more horrifying than them starving to death though or doing some worse job? I agree the conditions sound horrible, but it's still bringing jobs and money. Is making it illegal or uneconomical going to result in a better quality of life for the people that work there and their families?
My dad is from near Chittagong. I think it’s a thousand percent better for Bangladesh to do this dirty work to build a real economy of its own, instead of depending on European countries for hand outs. It’s not like america didn’t have kids working in coal mines when it was in an early stage of development.
To me this is first order thinking. If paying to keep workers and the environment safe means developing countries are denied work and resources, is that actually better? The simplistic "dangerous work is bad" analysis doesn't account for the upside compared with not bringing this work to the subcontinent. And the solutions seem to ignore the downside of shutting down or rendering economically unviable the shipbreaking yards. There's certainly going to be a gradual pathway to improving working conditions, but I think it has to be gradual and deliberate, or it's just going to shut the current workers out.
More regulation always favors big companies. A bad end point would be where Indian and Bangladeshi shipyards are regulated out of existence and established companies that can do lots of paperwork get to benefit from a regulatory regime that makes them mandatory. That's where is is likely to head when the problem is considered in a shallow way
This looks deep at one side, and fully ignores the other. Think about the consequences of not improving safety - personally a lot of poor people will be hurt, either resulting in them having to be supported by their family, or more likely simply dying, and structurally you're not encouraging any change in these conditions. By allowing the workers to be made to work in unsafe conditions you'll reduce cost and prices, but you won't have an incentive to develop safer ways to process everything, since the unsafe way is allowed and way cheaper!
> More regulation always favors big companies.
Is there actual research into this? I see this religiously repeated by a lot of people, but I somehow never see actual data to support it.
Why not just break the ships up in Western countries then? For Western countries to impose their safety and environmental regulations on other countries is denying them their sovereignty. The whole system of growth of nations and their economies is built on the fact that developing countries don't have to play by the same rules until they want to, or they have something to lose (in a reciprocal fashion, like with IP rights). In a trade regime, they're supposed to be punished when they engage in mercantilism, not self-sacrifice.
They aren't but that's really semantics. It's labor being used by these western corporations. You are explicitly responsible for the labor you directly source but you are also implicitly responsible for the labor you source through the services/products you purchase.
Outsourcing labor intensive work to lower income regions is almost universally a good thing but that doesn't mean that western corporations shouldn't be held liable for that outsourced labor. What needs to change is that corporations need to be required to perform due diligence that the work they contract out is meeting at least some established international labor standard (likely a lower standard than the country the western corp is operating out of has).
In the case of ship breaking, what needs to change is that any western corporation that owns and/or operates a ship during the last few years of its life needs to be held directly responsible for that ship until after it has been scrapped. They can contract the work out and all that but they should not be able to sell the ship off to a scrapper in the last few months of its life and wash their hands of their responsibility.
How long that last few years should be probably needs to be class dependent and you should probably be able to push off that "last few years" time window with a re-certification process (i.e. showing the ship still has at least x years of life on it).
If western countries actually held their corporations liable and stopped them from blindly pawning off work, they could drag up labor standards in other countries without depriving those countries of the outsourced work that their economies are heavily supplemented by.
Shouldn't it be a Bangladeshi government responsibility to make sure that their companies aren't abusing their workers? And, if the government doesn't care, why should foreign governments intervene? That sounds like a return of patronising colonialism.
It should be the business of other countries too, because their workers have to compete with Bangladeshi workers. It's harder for a German shipyard worker to tell the boss to pound sand when asked to do something needlessly dangerous without the right gear if their job can easily be moved somewhere where worker safety is just not a consideration at all.
Therefore German workers should use their government to stop this practice using whatever levers they have, which mainly means going after German companies.
Not to mention that forcing companies to internalize the full cost of breaking up ships safely will put pressure on shipbuilding firms to design ships that are easier to break up, probably increasing full-lifecycle efficiency without anyone needing to risk their life for a couple tons of scrap metal.
It's about protecting German workers' safety, not just "German jobs". They're inseparable goals. One abused worker is a threat to all workers.
Anyway yeah, realistically, the reason rich-world workers should care about poor-world worker safety is because it's in their material interests: They will be forced to do unsafe things at work, or else face being fired/laid off and the company will move the work to locales where (expensive) worker safety isn't done.
Right? Like those people in China who were so happy to break, split, sort and burn the plastic waste we used to send them back in the day. Why don’t they want it anymore? They should have been grateful. At least they had jobs. Otherwise who knows what they would have been doing! /s
This is not what the grandparent is saying. The typical situation is not that people are forced to do the work. We're not talking about slavery* here.
They are doing it because they see it as their "best option". By stopping it, the result is their "best option" is taken away from them, and they have to go do the next "best option". You could keep saying "this is inhuman, let's stop it from happening", but the effect is just making it worse and worse. Instead, better options should be provided. That's the real solution, but it's a whole lot harder to implement than the knee jerk response of stopping an activity.
edit to preempt ridiculous interpretations of my words: This is assuming slavery is not the case. If slavery is involved, it is a different situation to what i am talking about.
> This is not what the grandparent is saying. The typical situation is not that people are forced to do the work. We're not talking about slavery* here.
> They are doing it because they see it as their "best option". By stopping it, the result is their "best option" is taken away from them, and they have to go do the next "best option".
Exactly. I know what the gp said. It was the same with those in China. The West didn’t force anybody in China using slavery to take that plastic. They were happily buying it. It was the “best option under given circumstances” for those who were doing it. Whole families built businesses on it.
And it would be really easy to regulate this issue away. Pass a law in the EU or US that stipulates that ship owners AND the shipping companies that contract them have to perform due diligence on the end of life value chain of their vessels. AND make it possible for foreign nationals to sue these companies in U.S./EU courts if this due diligence hasn’t been done for damages due to injury or environmental damage.
Hmm, so how would that work if the ship changes hands multiple times. And might end up in it multi-decade service life in hands of some bankrupted entity? Original buyer being responsible of also decommissioning? What if they get restructured or cease to exist?
If at each transaction the prior owner would have to do serious due diligence on the likelihood of the next owner to dispose of the ship sustainably and would be liable for the method of the due diligence, it would preclude unreliable owners from ever buying the ship. And in well regulated economies, going bankrupt is not actually a license to ignore your corporate responsibilities. Owners and executives retain personal liability until the entity is actually dissolved, including assets like ships.
This is actually how we do a lot of regulation quite successfully, when we care about the outcome. There is not actually a government regulator sitting on every assembly line of every factory. Companies mostly self certify that they adhere to regulation and face steep liability if they don’t.
If it were "just" static pages, served the same to everyone, then it's pretty straightforward to implement even at the >300m users scale Netflix operates at. If you need to serve >300m _different_ pages, each built in real-time with a high-bar p95 SLO then I can see it getting complicated pretty quickly in a way that could conceivably justify this level of "over engineering".
To be honest though, I know very little about this problem beyond my intuition, so someone could tell me the above is super easy!