What an unimaginable horror! You can't change a single line of code in the product without breaking 1000s of existing tests. Generations of programmers have worked on that code under difficult deadlines and filled the code with all kinds of crap.
Very complex pieces of logic, memory management, context switching, etc. are all held together with thousands of flags. The whole code is ridden with mysterious macros that one cannot decipher without picking a notebook and expanding relevant pats of the macros by hand. It can take a day to two days to really understand what a macro does.
Sometimes one needs to understand the values and the effects of 20 different flag to predict how the code would behave in different situations. Sometimes 100s too! I am not exaggerating.
The only reason why this product is still surviving and still works is due to literally millions of tests!
Here is how the life of an Oracle Database developer is:
- Start working on a new bug.
- Spend two weeks trying to understand the 20 different flags that interact in mysterious ways to cause this bag.
- Add one more flag to handle the new special scenario. Add a few more lines of code that checks this flag and works around the problematic situation and avoids the bug.
- Submit the changes to a test farm consisting of about 100 to 200 servers that would compile the code, build a new Oracle DB, and run the millions of tests in a distributed fashion.
- Go home. Come the next day and work on something else. The tests can take 20 hours to 30 hours to complete.
- Go home. Come the next day and check your farm test results. On a good day, there would be about 100 failing tests. On a bad day, there would be about 1000 failing tests. Pick some of these tests randomly and try to understand what went wrong with your assumptions. Maybe there are some 10 more flags to consider to truly understand the nature of the bug.
- Add a few more flags in an attempt to fix the issue. Submit the changes again for testing. Wait another 20 to 30 hours.
- Rinse and repeat for another two weeks until you get the mysterious incantation of the combination of flags right.
- Finally one fine day you would succeed with 0 tests failing.
- Add a hundred more tests for your new change to ensure that the next developer who has the misfortune of touching this new piece of code never ends up breaking your fix.
- Submit the work for one final round of testing. Then submit it for review. The review itself may take another 2 weeks to 2 months. So now move on to the next bug to work on.
- After 2 weeks to 2 months, when everything is complete, the code would be finally merged into the main branch.
The above is a non-exaggerated description of the life of a programmer in Oracle fixing a bug. Now imagine what horror it is going to be to develop a new feature. It takes 6 months to a year (sometimes two years!) to develop a single small feature (say something like adding a new mode of authentication like support for AD authentication).
The fact that this product even works is nothing short of a miracle!
I don't work for Oracle anymore. Will never work for Oracle again!
I stayed at my second job for nine years from 1999-2008. By 2008, I became an expert beginner. I asked myself for years why I stayed there so long. The current me, would have left April 2003 when I closed on the house I was having built.
If I do a retrospective, it is kind of clear:
I worked with most of the same people all nine years. My coworkers became my friends and they were the only constant in my life.
My managers were shady as hell though.
I was going through a horrible marriage (2002-2006) a horrible financial mess because by 2006 I had $500K worth of mortgages making $70K a year (because you could back then).
My skillset was outdated. I was maintaining VB6 code in 2008 - 7 years after it had been discontinued (and some old school C++/MFC/COM).
As bonuses got cut and raises were meager, I only made $8K more in 2008 than I did in 2001.
My head wouldn’t have been in the game enough for another job.
I learned my lesson with my next six jobs. I do resume driven development. I only worked at companies that were using in demand technologies. I jumped ship after two meager raises or when the bullshit/pay ratio got too high.
I kept my network strong.
But on the other hand, I’m going to “stay put” at my current job at $BigTech until at least my initial 4 year initial vest (and an interim refresher) is done and see what else is out there.
I recently made the transition from full-time software engineer at bigco to independent consultant, so maybe I can help explain.
There's a fairly large mentality shift between salary work and contract work. When you're salaried, you're supposed to work eight hours yes - on paper. Really it's generally accepted people work 4-6 hours a day, with the rest being filled with lunch break or internet browsing or doing online chores or whatever. There is a tacit understanding between employer and employee about this, so any steps by the employer to monitor daytime computer usage is understandably seen as breaching this understanding in a misguided attempt to get employees to work more hours - or exposing the employee to arbitrary punishment if management wants to selectively undermine them for breaking rules that everyone breaks.
It's different when you're contracted per hour. When you carve out an hour for a client, it's game time. No faffing around on Facebook, no checking financial statements. You do the work. Does this mean you have to work a full 8 hours? Absolutely not. Just do part-time contracts to fill up as many hours as you want to work, then charge more per hour. Your hourly contract rate will be much more (2x or more) your hourly salaried rate anyway. So, since you're actually factually working during the contracted hours, I don't really mind whatever desktop screenshot system they want to use.
On another note: actually working 40 hours a week basically sucks. Let's acknowledge how good a deal we have as full-time tech employees and show solidarity to our brothers & sisters who do full-time shift work. Pushing full-time work down to 32 hours a week or less is a very worthy goal.
"""
For commonly used 1024-bit keys, it would take about a year and cost a "few hundred million dollars" to crack just one of the extremely large prime numbers that form the starting point of a Diffie-Hellman negotiation. But it turns out that only a few primes are commonly used, putting the price well within the NSA's $11 billion-per-year budget dedicated to "groundbreaking cryptanalytic capabilities."
"""
Most people leave their bosses and not their jobs. The inverse is also true. My last job as a programmer was not so great but I stayed there for many years because my boss was quite great and supportive.
Now, I run a tech recruitment consultancy and if programmers come to me about changing jobs, in most cases the reason is the boss.
Most programmers invest a lot of time in coding and related activities but neglect most other aspects of their career. This Ask HN is a proof of this. (Shameless plug: I am writing a guide on how to be more efficient in the workplace and get better compensated, pre-order "Coderfit: Make more money as a programmer" here: https://gumroad.com/l/cdrft)
I didn't know this format was a thing and am so very excited to discover it. I hope you folks enjoy reading horror stories.
I got a job as a Software Engineer in my current company 4.5 years ago; friend-of-a-friend sort of thing. The company had an apparently disastrous piece of software that was their main LOB. They had gone through pretty much every local consulting agency - at least once, on a few occasions they had gone back to one they had already used. It was about 10 years old and consisted of a mix of VB6(!), VB.NET, C#, F# and somehow now Node. At the time tackling a disaster like that sounded fun and I was miserable at a consulting gig. It was a 20k bump but no benefits (health or retirement), but as a single guy 6 months away from paying off his college debt I wasn't worried. I figured I'd dump a few years in then move on.
Three months in, I'm absolutely baffled at what the company does. I was told they handle insurance claims, basically acting as a TPA. (Important detail: I had no idea what a TPA was at the time. It's gonna matter later.) The software does handle claims, but they also have 10 other projects that cover a bunch of random business use cases. Apparently the CEO is a self-described "idea man" and would task the previous developer to 'prototype' his ideas from time to time. The problem was his idea of a prototype was a fully-functional application that he could sell to investors and clients - until he got bored with it and shelved it. This ended up with the company having around a half-dozen actively used products in a half-dozen markets. In addition to the TPA side of the company that was about 50% of revenue, the other half was split over 1) check cashing software, 2) HR/onboarding software, 3) some sort if discount medical visit scam, 4) some sort of MLM scam that the CEO's brother-in-law co-opted him into, 5) a random cannabis and self-help website run by some yoga guru type dude the CEO knew and finally 6) a piece of software that let helped churches organize events and donations that took about 50% of any transaction that was run through it as "fees" for our company. Now I could talk about any of those monstrosities at length, but this is already shaping up to be a wall so I'll skip that.
1.5 years later. I've wrangled the mix of VB6, VB.NET, C#, F#, PHP4, PHP5, PERL, ASP.NET WebForms and MVC, SQL Server, Postgres, MySQL still using MyISAM, god knows what other horrors I've forgotten. All of this without version control - just folders copy-pasted over and over on a 10 year old server in the closet that has no redundancy, two failing disks and one PSU out of order. The last guy had started some positive changes: moving everything over to Azure, porting everything related to the claims business into a more modern MVC app. I finished his work. I squashed about a dozen Wordpress instances into a single, multi-tenant host. Squashed out all the other languages and databases into just C#, ASP.NET, SQL Server. Ended up reducing the Azure spend by about $2000 a month. Felt good! CEO loved me. COO (my direct manager) loved me. CFO was pleased. All throughout this, I had convinced the COO to cut out all the shady, near-illegal, morally bankrupt garbage we did. No more check cashing (awful, awful industry), no more MLM of any sort, no more stealing money from churches (we kept that going, just changed our fees to a nominal amount). All the work I had done lead to a decrease in onboarding time from 2-3 days to 10 minutes and the TPA side of things was now about 85% of our revenue. Happy ending, right? Just you wait...
Somehow, I had not encountered a single brilliant "CEO Idea" for 1.5 years. He decided to fix that on one delightful summer day in the mid-west by announcing that we would be acquiring a healthcare startup that a buddy of his ran. Now this pissed most of the folks at the company off and is probably a good point to talk a little about the structure of said company. As mentioned, we had a CEO, COO, CFO, and "Chief of Sales" (never heard of a COS myself, but who knows). We didn't call ourselves a startup and had none of that Bay-style of startupness; we were just a small business with some investors. After the C's we had myself as the lone engineer, two sales guys, three admin-types and six or so customer service folks. None of which had healthcare or retirement benefits, mind you. So there was a bit of rancor when Mr. CEO started talking about dropping $5 mil to acquire this fancy new healthcare company. Somehow me, Mr. Software Engineer, ended up being the guy that needed to take this head-on (well, to be fair, the COO and I had great relationship). That's a tale in and of itself, but at the end of the day we ended up getting a 6% matching 401k and $500/$1000 single/family monthly reimbursement for health insurance, stopped 3-4 people from quitting, got me a whole lot of respect in the office and a fancy new title of "Chief Technical Officer" (not related to the benefits; CEO was just happy at how efficient I'd made everything) and 20k base salary increase. CTO at a company with 1 engineer. Neat. Happy ending, right? Just you wait...
We also got a brand new healthcare startup for about $2.5 mil in cash, $2.5 mil in stock. We got sheisted and it was our fault. While I'm no MBA, I know what due diligence is, and I intended to do it from the technical angle while our CFO handled it from the financial. Before we bought the company I made every effort to actually review what their software looked like, but was single-handled blocked by my own CEO. "We're never going to do that, Throwaway," he would say, "Other CEO is my friend! I've known him for twenty years and if he says his software is solid, it is! Just trust me." Diligence took about three months and despite dozens of arguments, I was denied any access to anything technical. All I ever got was: "Our software is in Node using MongoDB and is hosted in the cloud." Great. I was never even allowed to meet or speak to their development team (apparently 5 engineers, all of which were phenomenal). The only human being I ever spoke to at this company was the CEO. So I tried other angles, the big one being: what the hell does your software actually do? Their big claim to fame was 'modernizing concierge medicine using AI'. If you're like me and have no idea what concierge medicine is, it basically means your doctor comes to you because you're a rich yuppie and can't be bothered to leave your beach house to visit him. How do you enhance that using AI? I had no idea. Still don't. And so we bought the company with zero diligence done, though the CFO did say their books looked good, whatever that means. So the nightmare begins...
2 years in. We start onboarding people, I start onboarding the project itself. I am finally given direct developer contacts, which are a bunch of emails that don't end in the same domain as the company we just bought? Pardon? They're all @BobsRandomConsultingCompany. I reach out, explaining who I am, that we just acquired Project X and I need access to the code, environment, engineers - the whole nine. I get a very lovely, professional response from a Project Manager over at Bob's who lets me know that they will be sending over a contract so we can get started right away, along with their rate sheet! I'm baffled! I thought Project X had 5 internal engineers, Mr. Other CEO?! At this point I promptly aged 6 months in 6 minutes and I felt the first twinge of an ulcer growing.
Contract arrives, I sit down with COO and CFO and explain that we have been duped. COO is angry; CFO is not concerned until I show him the contract that Bob's sent over. The contract ye olde healthcare startup signed apparently agrees to pay for 5 fixed resources (at $200/hr!) for 40 hours of work each, per week, for a period of a year. Now I'm not unfamiliar with being outsourced as a resource, from a consulting company, for a fixed amount per week - but never have I seen a contract that binds you for a year, especially for 5 resources, with not one deliverable mentioned anywhere. Maybe my five years of consulting wasn't enough, but that blew my mind. Additionally, they sent us the server bills (AWS) and informed us we paid directly for utilization in addition to a "HIPAA Monitoring and Compliance Fee" of $3000/mo. As I had not a year ago lowered our own cloud costs to about $800/mo, this number struck me as staggering. $3000/mo base + around $2000 for the servers currently running. Also, "what the fuck is HIPAA" I said aloud, the only answer being the two confused shaking heads of my COO and CFO. Uh-oh...
Segway. The actual Project Manager of the acquired company (not the one from Bob's Hair Care IT Consulting Nail and Tire Salon) has moved in and I've finally got a victim to victimize with my many, many questions. She already looks harrowed before I begin my interrogation. Are people actually using this? How much do we make per visit? Visits per month? I forget the answers to these, but the end takeaway was: we bring in about $10k/mo net right now. I'm no accountant, but I'm fairly confident you can't pay the expenses of a company + a half dozen employees on $10k/mo. PM agrees - they've burnt through about $7 mil of investor cash over their 6 years of existence. No path to profitability is in sight.
Around the same time I've got the Project X repository (whew, at least they used source control) moved over into my world and have started reviewing the actual source. I'm no Node wizard, but I'm immediately confused as I see both Express and Hapi (two server frameworks, generally considered competition to one another) used in the same project. That's...odd. Investigation intensifies: it's a simple CRUD project that takes a form submission from a registered user, saves it in Mongo and slaps it into a queue for delivery to the given doctors email. That's really it. There's some back-end admin that allows the doctor to write some notes about their visit. Like a little baby EMR (though I had no idea what an EMR was at that time). Amusingly, it's got an Angular front-end (1.x, because why not spread salt on my wounds) that hits an Express endpoint that then proxies the call to a Hapi endpoint. For no reason. I can't find a single comment or piece of documentation explaining why. Icing on the cake? Their is in fact authentication used from Angular -> Express. The Hapi endpoints, however, are wide open - but surely not from the ELB, right? Certainly it's just an idiotic architectural decision that isn't actually exposed to the public? Nope. There's a rule in the ELB. Sweet Baby Ray's someone help me, there is a publicly accessible, completely open API that anyone could discover that gives away patient and doctor information. Huh, I wonder if the US has any sort of regulation on that kind of stuff? I should really take some time to investigate that HIPAA thing I found earlier, maybe that's got something to do with it...
Employment duration: unknown. My ulcer has had a baby. I think I may have had a psychotic break. I Googled HIPAA. I simultaneously shat and pissed myself, which I didn't think was possible during a panic attack, but the human body is an amazing thing. I took Thursday and Monday off from work to read through a PDF I found of this most enlightening "HIPAA" legislation. It says "SAMPLE" or "UNOFFICIAL" or some such on it, so I'm not sure how accurate it is, but whatever - I need to educate myself somehow. I spent a thrilling four days reading, re-reading, and summarizing what I understood of the several hundred page document - printed in three-column layout because why not make it more abysmal. It doesn't seem completely dire; it looks like there is some stuff we need to do if we are storing this mythical PHI, but it isn't terribly complex (at least technically!). I had already been planning encrypting everything we own, and all of our sites are already behind SSL, so this should be cake. Phew! Calm down, baby-ulcer, don't think about grand-kids quite yet. Also I found a few great summaries of the Act which I could share with my COO - but really, we need to sit down with Legal and have them explain why this was never brought up. And let's be honest, I'm not a lawyer - the professionals can handle this!
Legal has never heard of HIPAA. That's not good. I convince COO to ask Legal to reach out to a different Legal who specializes in healthcare. We sit down with them a few days later and our new Legal turns white after I lay out everything we do, our concerns, and the simple question: "Do we need to do any of this stuff I read about?" Turns out, having your CTO read a complex, many-hundred-pages legal document is not the best way to get accurate legal advice. We're fucked. We're a TPA filing insurance claims - we absolutely, 100% must comply with this Act. Oh and guess what? The Act has a delightful addition called an Omnibus, passed back in '13, that makes any possible defense we might have had to not comply...completely null and void. We're in what is called 'Breach'! We have fucked up. Royally and legally. Icing? We're all personally liable, at least to the letter of the law. But don't worry - we didn't know we fucked up, so the fees are an order of magnitude less. They'll only bankrupt the company 5 times over, instead of 10! Hurray!
Here's an idea to help "viralize" tree planting, especially among younger folks.
How about a tree social network? A website/app that incentivices the person to plant a tree and adopt it as a pet. The person plants a tree, names it and takes a few photos. A profile is then created.
The person can find nearby trees to socialize. Some cute artwork may help engaging people. Snapchat-like filters could be used to "dress up" the tree, further increasing cuteness (and the engagement).
People keep score by how many trees they've planted, which gamifies the process.
This may sound crazy (and possibly it is) but hey, it may just work.
If you (or anyone else) like the idea, I'd be more than happy to help build it, with the condition that it should be open source. Email is on profile.
Cross platform mobile games when you have C++ skills: Use C++ with OpenGL and share most of the code.
Cross platform mobile games when you need to ship faster and don't want to deal with C++: Unity
Cross platform traditional apps when you don't have a ton of developer resources: React Native (or better yet, Re-natal with ClojureScript)
Cross platform traditional apps when you have ample developer resources: A native app for each platform
Need to ship something quickly that will perform poorly, have weird UI quirks, janky scrolling, misbehaving touch events, and hard-to-debug canvas bugs that only occur on specific devices? Use one of those JS mobile frameworks.
+1 for Index investing - also called passive investing, couch potato investing etc. It is so simple and elegant I can put it in code:
// Decide asset allocation based on risk tolerance. E.g. 75% Stocks / 25% Fixed income
set_asset_allocation(risk_tolerance)
// Determine which index you want to track. E.g. 25% Canada (TSX)
// 25% US (S&P500), 25% Developed World (MCSI EAFE)
select_funds(list_of_funds)
// Set up automatic purchase plan so your account automatically purchases the funds bi-weekly or monthly.
enable_auto_purchase(name_of_financial_institution)
while (alive && working)
// Live and not worry about investments. Instead focus on coding, knitting, cow tipping, etc.
live(1 year)
// Buy/Sell more units of your funds to return to desired asset allocation
rebalance()
done
retire()
I've done lots of therapy. Really! It's very sweet how worried everyone in this thread is about me, but maybe I've painted a dire picture.
Time heals all wounds. I'm only 25, it's only been 6.5 years since I was last homeless. Even though it hasn't been too much time, I already feel quite a bit better:
* I used to have anxiety attacks and heart palpitations nearly nightly; now, it happens maybe 2x a year.
* "Money isn't everything - unless you don't have it." Without financial stress (yay corporate programmer salary!), I am much calmer and happier. Most people on HN are men so I don't know if you will understand this, but women (at least in the US) are often pressured to go into a "do-gooder" career. People look at me like I'm a snail when I say that I care first and foremost about being financially independent, but I've learned to ignore it.
* I've completely cut out toxic family members and learned to ignore any asshole that says "you will regret not having a relationship with ____ when he/she is gone"/"the bible says 'honor thy mother and thy father'" etc. If you say these things to someone coming from a family of abuse, you are very tone deaf. No one cuts family out of their life on a whim.
* Everyone talks about "not being a victim", but for me it was a very healing thing to admit that there were a bunch of bad situations where I was surrounded by bad people and did nothing wrong. Maybe it was pride that kept me from admitting this. Maybe shame. Maybe misplaced affection. Whatever the reason, once I shook this off I was able to move on.
In summary: it's okay to admit you've been a victim. It's okay to cut toxic people out of your life. It's okay to be selfish and "materialistic" sometimes. It's okay to ignore people judging you for things that they cannot understand.
Once I learned these life lessons, I left behind a lot of my issues. The more "physical" ones, such as startling easily and having nightmares, are relatively lightweight compared to the more insidious psychological damage caused by years of living with addicts and being treated like my needs were secondary to those of everyone around me.
I don't know if anyone reading this could be in the same situation as my teenage self, but if you're out there: please tell everyone trying to use you to go kick rocks. Know that you can do a lot of very awesome things, as long as you keep your mind open. And you are not "broken"! I won't lie and say Nietzsche was right about "what doesn't kill you" -- but you're never broken, not until you give up. Don't do that.
It is plain and simple, Kids. I'm 52 - been programming professionally since the 70's when I started writing C code and getting paid for it in 5th grade. Our "professional" is writing glue code, and how it is done and what hoops are jumped through simply do not matter: all that matters is the final shipping product, widget, or logical dodad works for the immediate marketing moment.
I speak from enviable experience: game studio owner at 17, member of original 3D graphics research community during 80's, operating system team for 3DO and original PlayStation, team or lead on 36+ entertainment software titles (games), digital artist, developer & analyst for 9 VFX heavy major release feature films, developer/owner of the neural net driven 3D Avatar Store, and currently working in machine intelligence and facial recognition.
Our profession is purposefully amateur night every single day, as practically no one does their computational homework to know the landscape of computational solutions to whatever they are trying to solve today. Calling us "computer scientists" is a painful joke. "Code Monkeys" is much more accurate. The profession is building stuff, and that stuff is disposable crap 99% of the time. That does not make it any less valuable, but it does render it quite ridiculous the mental attitude of 90% of our profession.
Drop the attitude, write code freely knowing that it is disposable crap, and write tons of it. You'll get lazy and before you know it, you'll have boiled down whatever logic you do into a nice compact swiss army knife.
And the best part? Becuause you'd stepped off the hype train, you'll have more confidence and you'll land that job anyway. If they insist or require you to learn and know some new framework: so what? you're getting paid to do the same simply crap over again, just more slowly with their required dodad. Get paid. Go home and do what you enjoy. This is all a huge joke anyway.
My retired optometrist friend recommended that I wear reading glasses with a very low power at all times while using the computer, and I do that now. They're .75x or 1x, I can't remember. I don't need them--nothing is blurry without them--but they help your lenses to relax by doing some of their work at close distances.
The condition he wants me to avoid is called nearpoint stress, but sounds very similar to me (a layperson). It culminates in your eyes lenses becoming "stuck," unable to fully relax to focus properly on things at a distance. I guess the reason is that your cilliary muscles (that alter the shape of the lens) become overworked to the point of a spasm.
One symptom of that is looking up from your computer and objects being blurry for a few seconds, but then coming into focus.
My own startup is using Django, so I'm certainly aware that there are new sites being launched with it every day. I was one of them last week. :-)
What I mean by obsolete is that it is no longer the hot new thing on the bleeding edge of technology. I chose Django because it wasn't the hot new thing; I'm already familiar with it, I've written a half dozen sites using it, and a general rule when founding a startup is that you shouldn't compound market risk with technical risk. Right now, my biggest problem is building a product that users want; I want the shortest path toward getting the big things right, not the sexiest little things.
But Django is definitely showing its age, and the environment has changed in ways that make you work around the framework instead of with the framework. Users are getting accustomed to logging in with Facebook or Google instead of having a separate registration system for each site; this is available with django-allauth, but then what's the point of having a built-in authentication system? Users are increasingly getting accustomed to direct-manipulation user interfaces instead of forms; Django has no support for that, and newer alternatives like Meteor beat the pants off it in that regards. Users are shifting to mobile; Django mobile support is spotty, largely provided by a set of sparsely-maintained third-party addons.
And then there are new technologies that promise to throw a huge monkey wrench into the web development ecosystem. Polymer & webcomponents are coming down the pipe, quickly; they introduce the idea of an application as a client-side set of individual downloadable components, which means that a lot of Django's routing, form, and templating infrastructure is no longer relevant. The increasing use of storage options other than RDBMS (Redis, Mongo, RethinkDB, protobufs or JSON on disk) makes the ORM less useful. JS-heavy apps shift the focus from building HTML quickly to building bundles of related JS functionality quickly, and Django has no built-in features for that. The most common solution, Bower, is pretty heavily tied to the Node.js ecosystem.
It is close to 25 million lines of C code.
What an unimaginable horror! You can't change a single line of code in the product without breaking 1000s of existing tests. Generations of programmers have worked on that code under difficult deadlines and filled the code with all kinds of crap.
Very complex pieces of logic, memory management, context switching, etc. are all held together with thousands of flags. The whole code is ridden with mysterious macros that one cannot decipher without picking a notebook and expanding relevant pats of the macros by hand. It can take a day to two days to really understand what a macro does.
Sometimes one needs to understand the values and the effects of 20 different flag to predict how the code would behave in different situations. Sometimes 100s too! I am not exaggerating.
The only reason why this product is still surviving and still works is due to literally millions of tests!
Here is how the life of an Oracle Database developer is:
- Start working on a new bug.
- Spend two weeks trying to understand the 20 different flags that interact in mysterious ways to cause this bag.
- Add one more flag to handle the new special scenario. Add a few more lines of code that checks this flag and works around the problematic situation and avoids the bug.
- Submit the changes to a test farm consisting of about 100 to 200 servers that would compile the code, build a new Oracle DB, and run the millions of tests in a distributed fashion.
- Go home. Come the next day and work on something else. The tests can take 20 hours to 30 hours to complete.
- Go home. Come the next day and check your farm test results. On a good day, there would be about 100 failing tests. On a bad day, there would be about 1000 failing tests. Pick some of these tests randomly and try to understand what went wrong with your assumptions. Maybe there are some 10 more flags to consider to truly understand the nature of the bug.
- Add a few more flags in an attempt to fix the issue. Submit the changes again for testing. Wait another 20 to 30 hours.
- Rinse and repeat for another two weeks until you get the mysterious incantation of the combination of flags right.
- Finally one fine day you would succeed with 0 tests failing.
- Add a hundred more tests for your new change to ensure that the next developer who has the misfortune of touching this new piece of code never ends up breaking your fix.
- Submit the work for one final round of testing. Then submit it for review. The review itself may take another 2 weeks to 2 months. So now move on to the next bug to work on.
- After 2 weeks to 2 months, when everything is complete, the code would be finally merged into the main branch.
The above is a non-exaggerated description of the life of a programmer in Oracle fixing a bug. Now imagine what horror it is going to be to develop a new feature. It takes 6 months to a year (sometimes two years!) to develop a single small feature (say something like adding a new mode of authentication like support for AD authentication).
The fact that this product even works is nothing short of a miracle!
I don't work for Oracle anymore. Will never work for Oracle again!