> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.
This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling failures)", but rather "this complex system collapses (rather than degrade gracefully) when demand exceeds capacity".
When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.
Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections: https://sre.google/sre-book/handling-overload/ Most other large internet services have since copied them, afaik.
My last three employers refused to take advantage of Kubernetes PriorityClasses and agree to schedule work to agree (as a cluster-wide resource that affected many teams) on what our PriorityClasses should be and to migrate workloads to have priorities. And this is something relatively easy to implement - no developer work required, and practically no YAML to write.
Why not? Because sadly, fundamentally, most workplaces are not run by people who care about day-2 operations or long-term health. Product or Sales pushes customer-visible work into the pipeline, and you dare not say no. "Day-2" work is not considered to be something that moves the needle. Even now, with GitHub facing these severe outages, it's not like they're facing some massive exodus; their load seems to be getting worse over time, not better.
I'd be very surprised if there weren't any employees at GitHub who had read the SRE book. I'd expect that they're just not listened to.
What TP is talking about has nothing to do with workload preemption and is more of a variation of loadshedding (e.g overload management in Envoy). When I was part of the team that ran Google's clusters we had very few priority classes - basically just one for system and majority of serving workload ran on another priority and the rest was for batch. Pretty sure SRE book recommends just that.
Workload pre-emption is a form of load-shedding - you shed the load of lower-priority workloads (by evicting their Pods) to free up capacity to schedule more Pods of higher-priority workloads that were added by the Horizontal Pod Autoscaler.
> basically just one for system and majority of serving workload ran on another priority and the rest was for batch
RCA blames in-house load-balancing services (HAProxy) that reached capacity limits. Even if autoscaling is not working correctly because it didn't take Istio into account - why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
Yes, just absolutely crazy way of doing load shedding.
> why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?
The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario.
> Application backends?
I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only workloads continue to succeed.
First of all, web/api being down is not the worst case scenario - i didn't even notice this till late in the day. Secondly, even 1M qps at the edge is tiny amount of compute, what you have here is not a scaling problem it's a misconfiguration problem. Don't make it worse by adding more load on the system with your preemption churn and maybe taking out more important services causing more severe/longer outage - just fix it in the edge.
are these separate services? Are you going to integrate how services preempt into overall system design?
My take is that you want to move these decisions into the application tier as much as possible as the first line of defense, both because you can make more precise decisions in the application and you can respond much more quickly. You want things like preemption to about a slower moving loop where you are applying much coarser logic to what gets squeezed.
> you want to move these decisions into the application tier as much as possible
I actually agree, but this is a luxury that most large companies cannot politically prioritize (it is not Product/Sales-driven, see earlier comment). Especially when the company is large, and there are dozens if not hundreds of developer teams in a polyglot microservice environment, pushing application-level handling of these concerns is virtually impossible without executive support, and because it doesn't move the bottom line in an easily measurable way, you won't get executive support.
Companies much prefer infrastructure-based solutions to these problems, even if they're coarser, because the relatively small number of people who need to be involved makes it politically feasible. Easy example off the top of my head - mutual TLS encrypting east-west traffic has been implementable at the application layer for decades, but it was a pipe dream until service meshes made it easy to deploy (it's still a pipe dream for many orgs that refuse to schedule any infra work not Product/Sales-driven though).
This is an excellent book and its lessons saved my bacon many times!
As it happens I have a hardcopy of this book (along with "Seeking SRE" and the "SRE Workbook") that I am giving away (because of a move). If you want a hardcopy then email me your UK address I will be happy to post them to your for free.
I tried putting them on the street in a little box but surprisingly none of my neighbours grabbed any of my software books. :)
actually, I think this time it is the other way around, Azure is more the solution than the problem.
Github in the past ran on their own Hardware. That is fine, if your load is predictable and nto changing rapidly. However, the evolution of the past few months/years has shown, that the previous assumptions about growth are now outdated and scaling that capacity on your own metal is not that easy. Hardware has lead times of many weeks, especially in the current situation, datacenter capacity is even longer and more difficult, especially right now.
Choosing not to deal with scaling the hardware is a valid choice in this situation. Yes, Azure is a bunch of servers held together with glue, duct tape and a lot of hope, but I think, the github hardware is not much better at the moment.
Agreed on not hosting things yourself, but, I am not arguing against cloud hosting at all - just Azure. If they just came out and admitted it is a disaster and swallowed their pride and moved their services towards anything else at all - GCP, AWS or whatever else - I think their uptime would significantly improve. Of course, the fundamental problem here is the culture of the company itself. That's harder to fix.
It's wild to me that when I think MS I think this article, Vista, Windows Phone, Zune, Metro, ads in the start menu, the pleated pants meme and so-on, but I have coworkers that are total evangelists for MS technologies and platforms. Clearly a lot of devs get value from the ecosystem.
A lot of their platforms (not Azure) used to just work. Visual Studio and .NET were solid. The Windows NT line was pretty good for servers too (but different from Linux, and not as flexible by virtue of not being a bazaar). It was at Windows 8 era they turned it into shit, first by designing a UI for tablets (fine) and then putting that UI on desktops (wtf) and silently upgrading Windows 7 users without their consent.
I remember running IIS and Apache side-by-side many years ago, and oh boy, was IIS a pain.
Instead of a single config dir where you just deploy the configs and you are done, IIS settings were all over the registry. And don't forget component registration - a single, global registry per machine, used by IIS. Want to run staging and dev server on same machine? Tough luck. Random software installs can upgrade components and cause failures in completely unrelated failures...
The article doesn't mention any Azure outages, but a client had an Azure hosted container restart that day which caused problems due to configs being stored out of persistant storage.
If you load an issue page, you'll see 1 failed request to:
/project/product/issues/<number>
And sure, that's what you care about, but consider the working requests to:
/in-product-messaging/copilot-budget-request-banner
/in-product-messaging/code-scanning-ai-findings-preview-banner
/github-copilot/chat
/_private/browser/stats
If page has 15 requests and needs data from all of them to work correctly, then with 20% failure rate you are suddenly close to 100% non-functional page from the user perspective.
A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage of incoming requests), group the incoming requests by the parent request (the one with the 1:15 fan out) and reject according to the parent request. One way to put it, simply (though somewhat inaccurately), would be: reject 100% of traffic from 20% of users, rather than 20% of traffic across all users (causing essentially full failure for all users).
We typically implemented this by propagating an ID of the parent request down to the backend. I'm simplifying a lot in this description (e.g. have to deal with the parent requests landing on different backend tasks; also rotate the IDs gradually to introduce some fairness).
A nice technique to learn of this situation is to fail ("black hole") random resources in synthetic loads of the frontend. This lets you find out 'bout these single points of failure, and then you can start adding fallbacks.
It's easier to create a nice worst-case failure UX this way too. Rather than a page that sorta kinda loads but doesn't actually work at all you can unambiguously put up a "oops we're broken right now try again later" page whenever that top-level endpoint returns a non-retryable error.
But that of course begs the question--why is the frontend retrieving all these data from 16 different endpoints instead of just one? It would be less overhead (http, auth, serialization, compression) to just serve it all in one request even if the http layer needs to fan out 15 rpc calls... this is why we can't have nice things.
Shyeah, all those successful requests were really helpful for showing me a page, which then failed when I tried to do anything by clicking the buttons on the page. I was completely unable to merge things for hours, for no lack of trying. GH was down.
Thank you so much for this gem, this sent me through the rabbit hole and I'm fascinated by the level of engineering in this book. I love how big complex systems are engineered. It feels like anything is possible when you have a solid plan and you keep improving it step by step. So inspiring.
> this complex system collapses (rather than degrade gracefully) when demand exceeds capacity
This.
It's easy to be an "armchair quarterback," here, though.
Handling stuff like this, needs to be planned for, from the start. I suspect that a lot of the issues are because GitHub is something that started small (and probably quickly), and has accreted. Things like Facebook are in a similar boat.
Of course there's infinite capacity. A google datacenter is infinite capacity from the perspective of say an NTP server. Infinites exist when you have enough orders of magnitude in the middle.
Also traffic isolation and degradation by tier is not "no outage", you're still in outage land, you're just being smart in how you use it and choosing what you disrupt. It doesn't fix the lack of capacity.
Google's (or any similar large company's, really) efficiency requirements would never allow for such levels of waste. It would be incredibly irresponsible to allow just any random small server to unexpectedly use the entirety (or even just a non-trivial percentage) of a Google-scale datacenter's capacity. Do you evict other jobs? Do you run your datacenters cold to prepare for this eventuality? Both are terrible options.
Suddenly you push an update to your tiny server or one of its thousands of clients and, because of some bug or misconfiguration, traffic or per-request-resource-consumption grows 1000X or more. Given the size of the looong tail of small servers at Google, this would be a daily occurrence.
I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding.
And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, but it significantly lessens the negative effects of it.
You understand it if you think of engineering infinites rather than mathematical infinites. The capacity of a full single datacenter can be treated as infinite for most customers. I explained how its defined in the original comment. Amount of places in engineering where you treat even a 3 order of magnitude difference as infinite is a lot, but the number of order of magnitudes varies depending on context.
You don't have a single Google datacenter (or any other provider / big name) for a single tenant. The bigger the DC, the more the number of tenants/customers reaches "infinity" (giving the same meaning to "infinity" you are giving to it)
Sure there is infinite capacity given an infinite amount of time to scale up. I assert you’re leaving out the time dimension. Those so-called infinities are simply not accessible in a practical way since you’ll hit a wall in actually provisioning that capacity long before the data center runs out of compute.
So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?
I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.
Best code is the one that you have not written :) Because the goal is not the code , it is the things that code does, and if it can be done without code, its the best code. Also if you produce lots of code that does not do anything in reality, then its worst code.
And yes, we can rebut that with "time you enjoy wasting is not wasted" except of course some externalities, like boiling earths oceans.
If it's not your job, then just ignore the reports.
If it's actually critical, someone will put money on the table and then it's a business. And then it's about scheduling and resourcing - also should not burn anyone out.
Just because many people have false sense of entitlement as soon as they get a free offering, it does not mean anyone needs to accommodate them.
Just doing what others wish is not conscientous in itself! It _may_ be depdending on situation but it can be just pathological towards the self.
When it's psyhocologically hard to do things you imagine will dissapoint someone that's probably not concientousness. It's more like low self-esteem or codependency.
It's very hard for someone to tell these apart themselves. Hence when this topic pops out it's good idea to remind that being super-accomodating may in fact be a personality flaw - that can be healed if acknowledged.
There is very large spectrum between "trying not to dissapoint anyone" and doing what you know is the right thing.
Sounds like their problem, not something a SaaS product should dance around.
Yet they kind of did. I've limited participation in my libraries with GitHub's setting that nobody who made an account in the last 6 months can do anything in my repos (after some misguided hustler thought they're an easy target and posted an ad).lp
Time's marching forward though. Wonder what will happen after a few more months. We'll have bot spam accounts that are no longer as fresh.
A great majority of business applications do run on open source projects, and in turn, are affected by them if things go awry. It’s a prisoner’s dilemma in this case.
Basically the only library for reading jp2k data (complicated specs, ask your AI to one shot an implementation, mine said "it's 3000 lines of fiddly spec, too complicated"). Issues full of buffer-overflows. Recently unmaintained.
Used in tons of projects, now all possibly vulnerable.
It's starting to become a cliché to have people reply "I'm getting a lot more done", but without seeing any evidence of this incredible productivity gains, I'm starting to wonder if y'all are suffering from collective hallucination. If the accepted claims are of "100x productivity" (increasing by the day), and LLMs have gotten very good for the past ~year, for sake of argument, where are the 100 year improvements in the status quo of software?
If one claims such extraordinary figures of 100x increased productivity, a step forward never seen in the history of humanity in such short timespans, they must present extraordinary proof or be branded as a complete lunatic. I could have accepted people saying "I'm 20% more productive", which is an incredible achievement by itself, but not the 10x, 20x, 100x I keep hearing about. I think I've read 200x this week.
The analogy here would be : a kid who wants a toy but does not have money. So he keeps dreaming how he some day would get that toy and how he would play with it and how it would make him happy, but times goes by and he still does not have money to get that toy. Then suddenly along comes LLM and you have infinite money to buy you all the toys that you wanted, you get them, but you now don't have time to play with them. Because time is money and just like before you didn't had time to "buy" the toys, now that you have them you still have no time to play with them.
I've heard people use the same word, I was dubious but they did produce some stuff, yet I think that it ends up as an itch-project. You're satisfied you saw the thing emerge into existence but that's about it. No more drive after that. Maybe because LLM don't require you to have a real long term intense need for that thing.
This is exactly my experience. A month of excitement building something that I wouldn’t have had time to do myself, then something broke in the setup and my motivation didn’t extend to fixing it.
I’m only back at it four months later and I don’t really know what happened before, or why it’s working now, I’m just happy that I can scratch that itch again.
Hey, traditional hand coder fellow ... the times have past and the future is already a present. I never have been this productive before, and it's been ~20 years that I spent coding. System level programming. Few years back I would have said the bottleneck is not in spelling out the code so we wouldn't see that much AI impact but boy I was wrong. Writing code has never been this cheap, both in terms of time resources and $$$. Now I can iterate over the ideas I didn't have the capacity before, both intellectual and time-wise.
I think using AI often feels faster than it is because you put less thought and effort into the problem yourself.
Another possibility is that the people who experience these 100x productivity increases are honest, correct, and simply had abysmal productivity which has now been increased to near-average junior levels thanks to AI.
> I could have accepted people saying "I'm 20% more productive", which is an incredible achievement by itself, but not the 10x, 20x, 100x I keep hearing about. I think I've read 200x this week.
That’s a measure of lines of code, I suspect the parent is talking about what results those LOC create.
AI built me a 1.5k+ LOC react component which is probably a 15x increase on the file size I would have created, with negative impact on the project for those extra LOC.
The variance is extreme though. Thanks to Claude Code and Codex I've been able to make several non-trivial internal tools and libraries without writing much in terms of code, just some reviews here and there.
I spent a couple of days on those, and its would have taken me months to write manually I am sure, so in that regards it's close to 50x.
I've also had Claude track down some logic issue in a module I was unfamiliar with which had very large and complicated flows. Would have taken me many days, since I did not have a reproducible case, so had to go by logs and customer description alone. I spent 5 minutes writing a prompt and when I checked back, Claude had identified the issue. The fix I had to implement myself, but was fairly easy. So there Claude definitely was a 100x increase in productivity.
Then there are cases where they're much more modest, or where they might even be negative, when they think they're fixing stuff but actually are introducing more bugs.
You can accept what you like, but it's true. Our team and our business is incredibly more productive. The number of new valuable customer facing features, and the number of PRs (which represent REAL work, not taking a PR and splitting it into 200 PRs game) have all increased dramatically. We've shipped something like ~10x more PRs so far this year than all last year. And we've done that without increasing the number of bugs and outages.
Most of my family are small business owners, so I've done a handful of Excel automations that they needed that generally take me a while to figure out. Landing pages and a few 3D designed prototypes that I used the Fusion 360 MCP server for
A link to a YouTube video without context e.g. summary of findings, primary author/creators, and primary citations, methodology, and so on is a next to useless for making a point. For all we know you’re linking to a crackpot or an industry sock puppet and I don’t care to “watch” any of what can potentially be a dubious video or worse a malicious video. It is your job as the linker to convince me the video is worth even one iota of my time.
On the other hand recent papers highlight the validity of concern/suspicion:
> This systematic review demonstrates that the environmental footprint of artificial intelligence is a structural and increasingly consequential challenge, shaped by interdependent decisions across algorithms, software pipelines, hardware infrastructures, and deployment contexts. The synthesized evidence shows that energy consumption and carbon emissions associated with AI systems are highly variable, context-dependent, and often underestimated — Beyond Efficiency: A Systematic Review of Energy Consumption and Carbon Footprint Across the AI Lifecycle (published in “Sustainability” an international, peer-reviewed, open-access journal)
https://www.mdpi.com/2071-1050/18/3/1359
Nobody is concerned about the costs and environmental impact of data centers.
When someone in a discussion about the benefits of AI goes "did you think about the environment?!", it's always performative.
The real motivation is disliking AI itself or doubt about the government's ability to offset the labor market impact. Discussions that start with feigned concerns being raised are nearly always going to be unproductive.
I don’t dislike AI but really of mine are negatively affected by climate change and AI isn’t helping what is easily observed when Google and MS scrapped their CO2 reduction targets.
So every time I use AI I think about the necessity and usefulness of what I‘m doing with AI and if the use outweighs the costs.
Since the rise of AI the environmental impact doesn’t seem to matter anymore.
I guess because it’s the shiny new toy of the hackernews audience.
Privacy also lost importance given the fact that the same people who refused to give information like their phone number to companies like Google and Meta now upload their whole life to their AIs to asks what should the eat, hyperbolically speaking
You're not helping your case by questioning the usefulness of software produced with AI in the same comment section, or complaining about other people supposedly compromising their own privacy in overusing AI.
The reason it doesn't matter is because the environmental impact is moderate, and the benefit obviously tremendous.
The environmental impact is anything but moderate and the benefits are not obviously tremendous at all. I'm happy using Claude Code as much as the next guy, but saying that the impact has been "tremendous" is vastly overstating the actual results.
I disagree, with the projected doubling by 2030 we're looking at 3% of global electricity consumption or 3.4 EJ, less than 1% of final energy consumption.
That is moderate. Energy-intensive industry is around 130 EJ, and global final energy consumption > 450 EJ.
Existing documented applications of today's AI have the potential to decrease energy consumption by >13 EJ/year by 2035.
Now that was about operational energy consumption. Someone might bring up manufacturing and construction.
From what I could find the climate impact of those are estimated somewhere between 10-35% of the total climate impact of data centers, so relatively small compared to the operational energy consumption.
It is very hard to justify more than moderate environmental impact here, in my opinion.
For the benefits of AI, my personal results have been great, so I am quite optimistic. And objectively, I find it hard to ignore recent results in mathematics and security research.
"Global data centers consumed around 415 terawatt-hours (TWh) of electricity—about 1.5% of the world's total electricity—with AI acting as a primary accelerator for new power demand."
That is today where we already consume too much. If by 2030 AI's consumption doubles it gets worse.
While training large models draws major initial power, everyday AI usage (inference) now drives roughly 80% to 90% of cumulative AI energy
"Existing documented applications of today's AI have the potential to decrease energy consumption by >13 EJ/year by 2035."
Seems like AI helps slowing down the rise of energy consumption.
We are at a point where we want less CO2 not moderataly more.
In the end more is more.
If your doctor tells you to lose weight or you get sick it's not a success to gain weigth slower
Well, at least we could establish that the environmental impact is moderate rather than extreme.
And if the potential of >13 EJ/year is actually realized, it would seem like the net impact of the data centers is not just "moderately more CO2" but possibly "moderately less".
I play modded Starfield. There's been a clear increase in mods lately. Some of them are from self-proclaimed non-programmers who are using the LLM's to reverse engineer the game or other abandoned mods, and they've started to create new cool mods or they've fixed various engine limitations. Can confirm that these actually work and I can finally have my 1000+ modlist.
There was a huge exodus of existing programmers/modders ~two years ago, due to paid mods and what not. The gamers took over with their LLM tools.
I have. In fact, some of my hand rolled stuff is actually _provably_ faster and more secure then even heavily battletested and widely used "industry standard" solutions. This is mainly due to them passing an exhaustive barrage of millions of lines of code of tests (literally, in fact I just did a pass over all my tests/ dirs and it's sitting at 6.3 million as of today) ranging from adversarial CVE probing attacks to fuzz tests. Ironically, my same testing suite has caught _numerous_ bugs in production stacks (openSSL/libuv) particularly, literal hard SIGSEGVs and the like.
On CVE probing, and I haven't really seen anyone describe/use it (or I may be oblivious), but the way you do it is you curate a list of CVEs for the class of software you're writing, say a web server. Then you take this list in chunks and hand them off to your agents to devise and implement adversarial technically analogous attacks against your codebase. If it's red, report and patch. Ironically (even with Fable 5) it's never complained/refused to do it.
Well, many of the saas I use continue to have problem meeting their SLA. And I’m on support just as often as always trying to get through to a human to file a bug that will never get fixed. “It’s in our roadmap”.. no it isn’t. Even with LLMs, product will focus on making new features to push their AI mission.
At my company, we are using AI to clean up huge amounts of technical debt that would have just hung around otherwise, so yes. I'm not sure that your average user would notice, but we do without a doubt have a much better product now.
That’s like asking whether fuel consumption was productive or leisure as the number of cars on the road increased. It’s both! I don’t think you can separate one from the other in any reasonable way.
That's how it's sold, but have you heard about any major tech company that sends their worker home when they've achieved what they used to achieve pre-AI?
EDIT: Instead of simply down-voting, you're welcome to name examples that proves me wrong ;)
This is anecdotal, but I know that a lot of my coworkers and coasting and putting up one AI generated PR per day which they've hardly even self reviewed.
> In a good company that will come back to bite them next performance review.
In a good company that should be discussed in the next 1:1s so actual change can happen meanwhile. If it just waits for the end of year review, then it's not a good company.
To some extent we are getting more things done as well. In my company (mid-sized startup), they're making us push features every other day now as opposed to maybe 1-2 features per person per sprint. Back when I joined, things were a lot slower. Today, they expect freshers to push new features on day one.
I have some big issues with this technology and the companies behind it, but I know of quite a few people personally who were not previously coders but have now been able to use LLMs to make their own custom software, solving real problems they had.
Are non-coders these days aware that they should use version control (and push their code to GitHub)? Or does their agent helpfully suggest setting up a GitHub repo?
One of the things I've done is create an entire fully functional GitHub alternative that does more of what I want and hosts all of my other projects, so yes I at least am getting considerably more done.
For some context, in June they said commits "commits nearly doubled year over year, crossing 1.4 billion per month". Now, it has more than doubled that in just a few months.
Makes sense given the ubiquity of agentic coding. I made a joke to my coworker today that all we do is make sure AI agents can communicate with other AI agents.
They really aren't. AI commit messages are much more thorough and correct than some of the stuff I've seen humans do over the years e.g. Fix, Fixup, Fixed some Stuff, Should work now, Definitely should work now etc etc etc.
Not really. In my experience, the average human written commit messages were almost always useless 1-3 word "Should work non" kind of messages. The agents had a low bar to clear.
Has it? Most of the LLM-generated commit messages and PR descriptions I read are needlessly verbose and miss the point: elucidating the why of a change rather than the what. If I actually pose a question based on them the author often tells me “yea that was some AI generated nonsense, I rewrote it now”.
Some guy, profiling his linux distro, wonders why ssh connections are a few hundred milliseconds slower than expected, finds there PR introduced and RCE backdoor and p0wns the whole project.
Seriously, this is driving me insane. The XZ hack told us what we had to do to secure our supply chain, and instead we went ahead and implemented a worldwide standard for NLP-to-action and figured we'd worry about the guardrails later. Mad.
The code change makes no sense and should do nothing. The commit message described a very deep investigation into garbage collection on the C++ side. Some object is being kept alive when the test requires it to be collected, and changing the code in this way allegedly prevents that. But wouldn't you think there would be a better way to ensure an object gets collected, like setting the variable to null?
The comments in the code don't make a lot of sense either. Something so obscure and brittle has to be explained extremely clearly.
While the issue might be real, this commit is so far away from the locus of normal that it's sending red alert. Plus a hallucination is very likely with such a long investigation - once an LLM agent starts investigating it just assumes there is a problem. And this is the 1 out of 1 robobun commit that I looked at.
Make sure the fs module keeps working if someone freezes or seals its exports table. I was wondering who was going around freezing random tables from other modules, so I checked the linked issue - robobun reported the issue, too. Why? I'm skeptical of whatever robobun was doing when it decided that it was necessary for code outside of a module to freeze their export tables. It needs a very good justification.
The first is, annoyingly, a relatively common problem and solution when dealing with GC lifetimes in tests. Few interpreters/JITs want to generate extra instructions to null out stack slots or pre clobber registers to ensure something becomes collectible at a specific point. Eager nulling of a variable often gets removed by dead store elimination or even just from being a disconnected SSA node. I've written extra nested scopes or wrappers in Java to deal with this in tests.
If this is needed in tests it needs to be known how it works, it needs to work consistently, and it needs to be documented how it works. It can't be an ad-hoc deep investigation and random fix each time. The comment should then be just // ensure foo is no longer a GC root, see gc_roots.md
Nixpkgs has 11k open PRs at the moment (a lot of them routine version bumps and such), so no. But then Nixpkgs isn’t a piece of software in the conventional sense: the monorepo does contain a few of those, but most of it is a giant collection of basically-independent build scripts.
that's the interesting part : forges are being abused.
Sure AI workflows might be a non-negligible share of all that usage but still the point is that initially forges existed to help developers collectively share a state then solve problems. Nowadays they are basically online filesystems with better notifications for other software to interact with and only optionally developers actually communicating.
github-actions: "If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code"
robobun: "The ordering is load-bearing: reclaiming before this block made is_dead_request true and hung a parked textStream read (caught by body.test.ts in CI). The comment pins that constraint."
Ah, well, if something is load-bearing, then I guess that settles it. Need a comment to pin that constraint, in case a read is parked. These are words that normal humans commonly use in these ways.
(Always striking how much Claude obsesses over the minutiae of method contracts and side effects, exhaustively documenting them in comments. It’s much happier figuring out how to reorder some method calls with nonobvious side effects so the code works than it is refactoring them not to do unexpected things!)
Why it doesn't work? It is working for me. It is working for bun. It is working for others who actually embraces it and puts in the work to get it working.
I recognize your username from other comment threads and would classify you as a bun fanatic, but even so, I’d hold off on saying, “it’s working for bun,” until 1.4.0 has been out of canary for, like, more than 24 hours. Most real users haven’t onboarded to it yet.
so first it was rust rewrite bad. Then rewrite with ai can never work. Then it will be riddled with bugs. It'll crash. It'll take years to fix.
On the other hand, rewrite was mostly done in record time. New version added massive number of features. Also huge bug fixes. Being used by Claude code by millions of people. Successfully used by some others even in canary. After release, multiple companies immediately switched due to massive amounts of resource savings and performance gains (and publicly posted about it).
Can there still be problems? Yes, I'm sure there will be. But denying the feat Oven pulled off with Bun in last few months is nothing but phobia/fud.
Many people are already posted about testing new bun version and I have yet to see a single post where the issue is the latest versions of bun. In some cases people posted it doesn't work but that's due to node compatibility etc and it didn't work on previous version either.
One does not need to be bun fanatic to see and call things as they are.
PS: I like bun because I hate how js ecosystem requires 100s of packages to do anything and bun is aiming to include batteries. This is good.
Well either they can handle this load that Microsoft can't, or they can't. If Microsoft are going to continue to be unreliable in the absence of the rate limit then:
If alternatives can handle the load, those who would consider those alternatives if Microsoft opposed a rate limit are likely to move to them anyway.
If alternatives aren't able to manage, then user's aren't going to jump since those services won't actually provide more usage.
I was thinking the same thing at first: ideas to increase product limits on GitHub, to increase reliability given limited infra.
However, there are sharks in the water, and with the diminishing mean of user technical knowledge, the product actually needs to become even more free. GitHub likely needs even lower friction.
"All it takes" is the insanely heavy technical lift to support that. There is no other solution. All the C-Suite needs to do is foster an environment with well-thought through, and possibly over-funded engineering, at the edge of the art. That sounds like an amazing challenge.
If you use GitHub as just a software forge then sure you can find an alternative. But I suspect more people use GitHub for its social aspects and they will stick around despite the regressions because building an alternative to an established social network is incredibly difficult if not impossible.
You can get work done on any software forge. But potential employers will still ask for your GitHub. People will judge your personal project by its GitHub stars and be less reluctant to download a binary from GitHub than elsewhere. Potential contributors will leave a PR on GitHub but probably not if they have to make an account on a new platform and learn how it works.
And of course, let's not assume any competitor can just absorb even a fraction of the traffic GitHub receives without suffering similar reliability issues.
I still remember when they decided to limit the number of private repos you could have as a free user. Kind of silly to me at the time, and even more so now!
Which will just increase the cries of "enshittification" and hasten the mass migration to the next free platform that surely, this time, won't ever go down.
We're small enough that we've been hosting our git infra for about a year now, I wonder how many other companies figured out they could make the trade. I've had a Github since a couple years after they started and I think they are going to become a Stack Overflow, albeit slower with MS at the helm. If Github is going to be 99% slop it's going to be really hard to use as a fun tool to show what you can do, what you've worked on, side projects, etc. I took github off my resume and I'm probably not going to relaunch my weblog if I end up job hunting, too much low-effort crap and people basically copying what a lot of us had been doing manually for years to really feel like it's anything other than a negative signal.
Ironically a bunch of people already migrated to Codeberg and then Codeberg announced "heads up, we actually don't want your AI slop, we're for real projects only" and AI coders threw a shitfit on HN.
GH processes at the commit level for things (including actions) even though they're bundled in a push... it's relevant to the load on their infrastructure.
In what way? If I have multiple local commits pushed once I expect to see CI type actions to run once for the push rather than for ever commit in the push.
This is configurable in Github Actions. Many projects want to run CI for each commit to avoid situations where one commit breaks the build and the commit after in the same push or PR fixes it. Broken commits in the history makes bisecting harder.
Probably? If you do 100 pushes instead, there is roughly zero additional data. At best you'd be comparing cache costs, which probably are lper for one large push, but there's a ton of calculation and CI that runs per commit regardless of other data being cached.
Why don’t you have your actions run on pr or push instead of on commit? Why would you even want that? I’ve never seen actions set up that way. If I push a branch with 100 commits then it’ll only run CI once. It’ll show the rest of the commits in the UI, sure, but that doesn’t mean that it’s the same performance impact. It could very well be 1 db transaction with multiple rows written instead of 100. I think you’re reducing this problem too much without knowing their architecture.
Github runs CI per PR push (obviously), and per main-branch commit (click on commit history in any project with CI, see a build result check mark on each one - that's true if you push a dozen commits too (I've done that)), in nearly all setups I've seen. I'm not sure how much of that is required vs default though.
With enough effort, you can rather obviously run CI per PR commit (it's a programmable system), but I've never seen aUI-integrated way to track the results, aside from browsing custom job names, which is very far from what I'd call "integrated" when compared to PR-level build markers. Similarly, I'm not aware of (but would not be surprised by) any way to disable per-main-branch commit builds, aside from initial pushes.
But I haven't poked around deeply in the settings, and business-account settings are rather different anyway so those might be wildly different / more flexible / more obtuse in exciting ways. Github is a very large and complicated product at this point, darn near anything could exist if you dive through enough UI layers or use old URLs to find soft-deprecated features.
Also, honestly, 100 commits = 1 transaction? That's far more of an over-simplification than anything I've said. It's a massive product with thousands of engineers, there's no chance at all it's just one database.
Commits are not expensive, pushes are. You can do any number of commits before you do one push, unless you are editing online, in which case every act is it's own commit & push.
You can rig up a local ide to pathologically commit+push per save, but you can do literally anything, so what you can do is immaterial.
Do you have some GitHub architectural knowledge you’d like to share with us?
A push pushes commits and blobs and trees and tags. It’s an interesting metric to track, but the core unit of complexity (and expense) worth tracking on GitHub’s side is obviously the commit.
There’s a difference between pushing 1 commit and 100.
> There’s a difference between pushing 1 commit and 100.
There isn’t much. GitHub doesn’t run actions separately for each commit. It runs them on pushes. I’m trying to think of a thing that would happen for each commit in each push and coming up blank.
It does things like scan for references to issues to index, but it would just scan the log for a range.
I did disagree with GP though because there is no reason to assume that the ratio of commits to pushes has materially changed. So if that is the proxy they have always used for measuring growth, and they know it reliably does that then I think it’s a reasonable way to communicate this to this audience.
Sure, because pushes are how you update a reference. That’s really what triggers an action: a reference changing. And there could be a bunch of those in a push.
A commit costs storage, you’ve got secret scanning, it needs to be indexed in a way that can be referenced in commit messages and comments, a commit message itself can close issues or reference other PRs, stored and served individually and immediately via the web UI or git clients, etc etc.
None of the things you mention - indexing or secret scan would be done individually for each commit. As I already said, this would be a log of all commits in the range pushed - it would be scanned once for those things. There is no need for a loop running over a range of commits and processing each one.
There 100% is at least for things like secret scanning and message parsing.
Secret scanning needs to make sure my repo as a whole has no secrets. It’s not acceptable to have 1 commit introducing it and 1 removing it because the secret is still recoverable.
Every commit is also surely an entry in a database somewhere. I can navigate in GitHub directly to any individual commit so there is definitely some overhead of some type.
It's not necessary to perform secret scanning on a per-commit basis. The most efficient way is to just scan all blob objects being pushed; there is no reason to even be aware of an object's location (tree path or commit) except for diagnostic messages.
> I can navigate in GitHub directly to any individual commit
You can do the same with the git command line client. The overhead you claim is already in the git on-disk format. Github might very well duplicate this information in a database somewhere, but it doesn't follow from your observation.
You are right about secret scanning, but its worth noting that is only enabled by default on public repositories. It is an extra paid feature for organizations on a teams or enterprise subscription, and isn't available at all for individual owned repositories outside of an enterprise subscription.
And yes, I agree there is indexing of commits, but that is a batch insert from a log.
>You can rig up a local ide to pathologically commit+push per save
The dev system we use for a 3rd party hosting provider (a big one) requires a commit and push for every file save while we're developing. I created a build system for this that copies the whole repo to a temp folder. As we save changes to files in the main repo folder, the build system watches for changes and copies the changed file to the temp folder, then does a commit on the temp folder and pushes to a an intermediary repo in github which then triggers an action that causes the 3rd party system to update from the intermediary repo. This way we don't pollute our main source repo with a commit every time we save an update to a source file.
It's not my favorite way to develop but it's caused us no real problems except when github goes down.
I don’t think I could imagine a stupider idea than this if I tried. To paraphrase Babbage: I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a solution.
meh, it's just external undo button. It's useful. It might or might not be worth the cost, but it's not that it delivers no value or causes some harm (other than cost/reliability)
It's like cancer growth though, not the 'good' kind of growth ;) E.g. I doubt that the number of Github users has doubled in that month too. Github should probably introduce daily commit- and merge-limits that slow down excessive clanker activity, but are high enough that a human doesn't notice. Alternative put users with excessive resource usage on their own 'sub-infrastructure' so that when this is overloaded, the regular users are not affected by the outage.
Serious question: do you think your 6 year old is learning coding from prompting an AI?
I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong. This is proven on a nearly daily basis at my job, where, with identical agents and prompts, I see designs being pushed with objectively incorrect facts, sub-optimal code in PRs, and a general explosion of slop. That tells me that it still very much matters if you know how to do the job without the help of robots.
> I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong.
That knowledge will be worthless 12-18 months from now when AI does everything better than you, including “understanding”.
If you’re one of the world’s best programmers, it might be 24 months instead, but the writing is on the wall for everyone.
I wonder if people were behaving like that for other revolutionary technologies in the past. “I can still run faster than a car can drive in sharp turns on a gravel road…”
General intelligence is a convenient myth perpetuated by the marketing departments of AI corpos to help further their pursuit of regulatory capture. Don't fall for it.
> That knowledge will be worthless 12-18 months from now when AI does everything better than you, including “understanding”.
...what? Seriously though. What grand insight into the arc of AI development do you have? Where are you getting this from? Cite your sources. Show your analysis.
Charitably, I would ask "are you high?".. If you're suffering acute psychosis from mind altering chemicals then we merely have to wait for the effects to wear off and you'll likely recover. If not, then I'm very sorry. Your road ahead is a rough one.
Not that impressive when you realize it's mostly due to AI slop
edit: AI actually writes 99.9% of my code these days. I'm just saying of course the number of commits to github is going to climb astronomically due to AI.
Agree - this reminds me very much of the old joke of two economists increasing GDP by taking turns giving the same 200usd back and forth for having each other eat shit.
Useful / impressive for whom is the question.
Not for us!
We pay for Github enterprise, and because GH can't be bothered to separate service tiers for sloplords and actual paying customers we get garbage level performance. They could of course always implement usage limits, but the goal is not to earn money, or provide a good service, the goal is to maximize AI users. Would be very awkward at the next executive golf meetup if you couldn't point to increased AI adoption.
In short: This is why monopoly laws matter. Once a company becomes too large, normal business rationales cease to be the motivation for their actions, and GH can go along with the pied piper of AI psychotic C-suite officers like MS is doing instead.
Impressive for whom? It's impressive for the service to have such growth at that scale, the code being slop is somewhat irrelevant. Your comment just seems like mood affiliation (AI should be dismissed, growth was from AI, therefore growth should be dismissed).
Is it impressive? All it's doing is decaying the services. 15 years ago never have imagined I would go to the lengths to host a github alternative on a VPS but after doing just this (also being the last one in the my professional group to do so), GitHub is giving a master class in destroying their reputation in pursuit of advocating for hostile entities.
Not all growth is good, especially growth that is actively hurting the company.
There is an equilibrium in both nature and software. Purposely designing systems that mimic the effects of cancer is going to benefit who exactly?
This growth is surely good for github. If another company becomes the "github for AI agents", they'll lose not just their business for AI but also human coders. (Sure, maybe there will be a human coder only github, but it will be quite small.)
LLM companies are already wildly unprofitable. Who ever wants to do this can be my guest, but please let's tax them enough so that we can get something positive out of the stupidity like free public school lunches for children or universal childcare.
How is it impressive if we all know it's autogenerated? There's no more people there than there were before. Heck, at ~2x growth that's possibly a decrease in real humans there since bots generate loads of them per person
That's not how I read the statement "It's impressive for the service to have such growth at that scale" that this was a reply to. They do seem to think the growth is impressive, not the absolute quantity, about which I agree with you
Yeah, seems like AI slop is going to kill GitHub's free tier. I just don't see how the economics of having to host this much slop and provide service to slopcoders is going to convert into dollars for them otherwise. None of the humans involved are going to end up in big enterprises. It's all cost, with no pathway to revenue.
When a service that was already the primary git hosting provider for most of the world for 20 years grows at that rate its not mundane and its not comparable to any example.
The absolute number doesn’t matter nearly as much as the change in rate of growth. The number of commits had not been doubling every six months at GitHub for a long time.
At Amazon if traffic volume consistently doubled every six months that is actually quite a lot easier to plan for, it just becomes part of everything they do from very early on.
No one said it is unique. But if you take an infrastructure and engineering org that had been growing at 10% a year for a decade, you are going to have a different set of capabilities and practices in place. Adapting to a new reality of doubling every few months will predictably produce failures anywhere. GitHub is not unique in that regard.
They have what amounts to an unlimited budget for AI spend. If it’s so fantastic, why can’t they let it crawl over every piece of their codebase, every metric, every log, and spot these problems before they occur?
“We misconfigured a sidecar” is something I would think AI could quite easily find and fix.
I don't know who you are responding to. I haven't made any statements about what AI can do for them. I would expect AI is making the situation worse, like it is in many dysfunctional tech organizations.
Instagram, Facebook and even threads all had much more mundane growth rates and definitely no unexpected jumps like GitHub is experiencing. I'm sure if suddenly the solar system had 10 more earths with each about 10 billion people and they would all start using Instagram tomorrow we would have exactly the same growing pains and outages that GitHub has today.
Luckily for Meta agents are not yet as much into doomscrolling as humans are.
That does not seem to be true - which two-decade period are you talking about? AWS has only been around for ~20 years, and I just reviewed a 10 year period, and not a single one of those years saw doubling in the whole year, let alone doubling in a few months. Which 20 year period are you referring to, and are you referring to doubling every few months over that 20 year period?
Eh, I would be more empathic in this situation[0].
Github isn’t small startup, where other 10x threshold is as cheap as buy bigger box in your IaaS.
When you are already biggest player in the ecosystem and you suddenly get 10x persisted traffic, with at least 30x+ forecast “soon” - I am not surprised they have issues.
Not sure to be honest, from a machine perspective 2X should never be a big deal, unless 1.4 was the threshold or sweet state and no one thought too much about scale and architecture beyond that
If you’re the size of GitHub and you’ve been running your infra for years with very little variation in traffic patterns you have a strong incentive to optimise costs for that existing behaviour.
I am afraid thats now how infrastructure works from what I have seen. The number that really matters is QPS. For any system the QPS varies through out the day and across the week and months. Most design considerations easily absorb any 2X increase. Pick up any company and the chances are that the servers are over provisioned, no one takes chances specially with critical components.
What you have going on with Github is mix of multiple things. Traffic alone is not the cause from what little I know, it does adds to the problem for sure
1. Infrastructure is being moved to use Azure, and overall all the cloud providers are struggling with hardware at the moment (same is going on for linkedin too)
2. The core teams, the people who knew the existing systems have either been laid off or moved from Github
3. Microsoft veterans are brought in to fill the gap across the board, they are trying their best but its a lot of unknown for them
How much infra have you seen of the top 100 sites in the world? I've worked on multiple top 10, and absorbing a 2x increase (and the peak is very likely more than 2x) is a very very hard problem that would cause hundreds of pagers to go off and load shedding to very high degrees. There is just not tons of unused capacity lying around in wait at the scale of github. "No one takes chances with critical components" is also very wrong for the simple fact that you don't know which is the weakest link in the chain until it fails.
I have some good experience and I feel bad about state of these things too specially given that a lot of it could have been prevented. What you have here is not a single service, its a system compromised of hundreds of services, possibly without clear ownership for some of them after these many years and reorgs. There is not a single person or group that understands the whole system from technical standpoint and pressure points. It akin to people trying to plug the holes as the water starts getting under pressure from different joints. This duct taping is present in almost all big enough systems, you name them.
> "No one takes chances with critical components" is also very wrong for the simple fact that you don't know which is the weakest link in the chain until it fails.
These companies were built and run by people passionate enough for the craft, ones who cared for the systems, who designed them. There is this idea that you can replace people by process and everyone is replaceable. What you have is a classical state where people are just doing their time.
Why does Github not segregate the free offerings from the enterprise or even better, all paid offerings?
It is unacceptable that enterprise plans get impacted by traffic on free and public repos. Our repos are neither on the free plan nor are they open. We have not had more AI stuff happening in the last weeks. Our traffic is stable. I would wager that most enterprises did not spike the traffic all of the sudden. Even if they were, we are paying for our quotas. Still our Github actions were breaking and our PRs not viewable at some times.
I am hoping this instability is going to cause a Cambrian explosion of forges and if that is happening, Github will be the first victim of the AI revolution.
I am working on a truly decentralized / local first code review right now, and a big part of my motivation for this is how bad Github has become. I dont know if I have enough time to build CI as well, but I am hoping others do. Otherwise I will just fall back onto Jenkins.
I applaud GitHub. However, I think no matter how valiant they are they will not climb out from under this. The scale problem will keep getting worse, and it's getting worse in a way I don't think is translating to more money for them. Sooner or later, they're going to have to charge for things currently free.
That's a business decision for Github of course. They are getting some value out of being the goto place for source code hosting for what is essentially a shockingly high percentage of OSS projects and a really large amount of enterprise projects. It's essentially the largest and most complete network of developers and code in the world. That kind of influence and reach is valuable in itself to Microsoft. And of course it's a gold mine for AI training data as well. Which is presumably why they sponsor it.
But it does raise the question for especially commercial users of Github whether it's time to reconsider the relationship with Github and maybe not put all our eggs in one basket. Basically, this wiped out a whole workday for many companies.
I'm not that eager to start self hosting my stuff. But I am considering it. Besides availability, CI build performance is also becoming a blocker for us. My AI coding jobs creating lots of PRs are making that a bottleneck. Fixing that in Github would require switching to a paid plan. And at that point, self hosting might be the more cost effective option. There are a few tradeoffs here of course. But I like the idea of throwing more memory/cpu at this to get blazingly fast builds.
Maybe GitHub sees value in hosting the code that is produced, for model training purposes. This would justify to maintain a free offering, even when it gets flooded like it has been recently
Where in those ~12 billion commits is the software, products and "innovations" which are supposed to be making our lives better? Software and apps in particular are getting worse, normies hate AI more than ever because they're even less likely to get their desired outcome when calling their doctor or trying to get their online order refunded when chatting with a cutely named chatbot, wages for (most) knowledge work are being driven through the floor, artists are being squeezed more than ever, etc., etc. That's to say nothing of the existential threats to the economy, environment and critical thinking which are growing daily. I really think we've lost the plot, folks.
On the topic of translation/ language learning, I still find Google Translate better for when I need to type emails in Russian (my older family who only speak it) as far as grammar is concerned than what LLMs spit out.
I'd be surprised if Google Translate isn't powered by a specialized, Transformer-powered model we would now recognize as a "small" language model heavily constrained to a specific task.
The transformer was invented as a machine-translation algorithm, later adapted to create LLMs. I think it's a reasonable hypothesis that a ML system designed for translation will always beat an LLM role-playing a translation system (assuming equal computing power and engineer effort).
We're going to see is a lot of stuff people use LLMs for moving back to using bespoke algorithms solving the one specific issue. LLMs are just machine learning unlocked for the masses, where you can ask it "translate this for me", "what's the sentiment of this text", and it just does it. Any of these were already achievable before LLMs, they just needed a machine learning engineer to implement.
I myself started an LLM driven project to classify and sort all the photos I've got. I just ask an LLM to poop out tags for an image. Using a proper algorithm for this is possible, but I'd need to put in some effort to actually get it set up. LLMs do make it easier since they can directly, instructively help you that set up, and provide an easy stream of training data to distill out a model perfectly fit to your usecase.
I think majority of work currently going through LLMs is such inefficient tasks that can be solved quicker and better with a limited algorithm.
Most people would prefer an excellent language learning app to simply be available and ready for them to use, rather than having to come up with how such an app could best work, and to get an AI to implement it by themselves, and having to iterate on that. Just like most people simply want to buy nice clothes, and don’t want to become fashion designers and managers of a clothing factory. So far it doesn’t look like AI is making the former (readily available excellent apps) more widespread.
As someone also building a language-learning app for myself, it's not too hard to create an excellent app right now, but the "readily available" part of your requirements is now 10x harder than a few years ago. Users are getting bombarded by new apps, so marketing an app, which was already hard before AI, is now much, much harder.
I could try to market this app, but I tried that 2 years ago, and it was already essentially a failure. So I'm probably going to keep it for myself or just try to find a few users who like it and will use it for free.
I think I agree. But it's nice to have the option. Furthermore, i think it's too early to see the results of this, as technology takes a while to diffuse. Since there are so many people trying to take advantage of the hype to make a quick buck, it takes time to figure out what is actually useful and what is not
Having an app means exactly nothing. How good your language skills will be after N years of learning is the actual measure of whether it's worth anything.
Hang on! What's the proposed causal chain from AI to famine here? Inference -> energy prices -> fertilizer/food prices? Or are you referring to the war?
> ai code dev has enabled people to make tools for themselves that they didn't have before.
...that sounds pretty worthless tbh. The promise of AI wasn't that you can bang out your own little half baked duolingo clone. It was that every engineer was supposed to become hundreds or thousands of percent more productive. The assumption being that things generally would therefore get noticeably better.
We're seeing hundreds of percent more commits. We're not seeing that translate to any real world improvement of anything. That's the problem.
Call me "ignorant" if you want, but back it up by showing what all this "AI productivity" has actually accomplished in real economic terms. Not vaporware.
Out of curiosity, what real economic terms would you have to see to be convinced?
Do you mean more earnings per share for corporations or real utility impacts on social systems (e.g., more new drugs)
When I ask Claude about the AI ROI, it seems to cite that ~95% of AI PoC's are negative ROI but the 5% that do have ROI tend to have decent return -- my guess is mostly back office clean up to reduce expenses and increase revenue/profit.
IIRC Uber claim their 1.5k/mo budget lead to no new value creation.
FWIW, I look at the situation with similar skepticism. One argument you could make was what Marc Andreesen said with the hypothesis that "all the big companies get nuked", so no new valued gets created but the large market cap companies get eaten by thousands of little pirañas. Obviously, so far, that doesn't seem to be the case.
> Do you mean more earnings per share for corporations or real utility impacts on social systems (e.g., more new drugs)
I mean real utility impacts. More reliable and efficient software, better features, etc. So far we've seen none of that. In fact, it seems like the opposite--more volume but a much lower quality product.
As a hobby I am playing around with mesh radios using Lora.
Let me tell you, it is _silly_ the amount of vibe-coded software in this area is popping up every week. Is the software duplicated? To an extremely large extent, yes. It is useful? Yes, but each piece of software seems to have a smaller and smaller audience, and quality is often severely lacking.
Iv'e done my own, too, for "RF debugging" as I called it to look into SNR issues related to interference. The software getting produced is likely useful _somewhere_, its just that youre not going to notice it.
Are we just moving towards personalised digital assistants for everyone, which in turn will produce software to function? Not unlikely.
Yeah, a lot of the software around MeshCore people have created seems heavily vibe coded (since got used to having press a button multiple times until it works) & the community seems to be surprisingly non-critical to "AI" technologies. Is it just about timing or something else ?
What are the boosters ranting and raving about if not the above? Am I just confused and the end game for all of the AI hype, investment, infrastructure build out, etc. over the last N years is ... hobby projects?
Here's a quote I found after a quick web search for "dario ai benefits":
> The list of positive applications of powerful AI is extremely long (and includes robotics, manufacturing, energy, and much more), but I’m going to focus on a small number of areas that seem to me to have the greatest potential to directly improve the quality of human life. The five categories I am most excited about are:
It's hard to separate out the hype from the results, but if you think none of that's being tried or worked on then you're the Ostrich with your head in the sand. Mayo Clinic says AI models help detect pancreatic cancer on CT scans 3 years before clinical diagnosis.[1] Breast Cancer.org says emerging research shows AI might spot cancers in mammograms that radiologists might miss[2]. Nature.com has a paper conclusing that AI (in various forms) have "already revolutionised the diagnosis, prognosis, screening, biomarker discovery" of cancers.[3] Cancer.gov says NCI scientists are using AI to improve cervical cancer screening with deep learning and digital images[4].
Dr David Burns offers his Teams CBT therapy with an LLM based app for people who can't afford therapist visits[5].
U. Chicago says that AI weather forecasts can be faster and cheaper than supercomputer-based physical simulations and produce results weeks ahead, which makes accurate forecasts affordable and available for farmers in middle-income countries[6].
With your smartphone camera you can get an AI-assisted bike-fit at home[7].
You can get a bike fitness training plan[8] and performance analysis with an AI that you can chat with to analyse and explain or change it, used by three professional cycling teams.
Prophix Australia analyse photos of powerlines taken by helicopter using AI to track assets and identify and verify potential faults[9]. AI systems scan images for early signs of plant disease in the vineyards of the Swiss Alps [also 9]. AI systems analyze multispectral images and satellite images of crops to detect nutrient deficiencies or pest problems in large farms[also 9].
In 2020 - 2022 a trial of a system in a hospital to identify patients at risk of sepsis helped medics treat them 2 hours earlier and reduced the death rate from sepsis by 18%.[10]
Self-driving cars now exist. Camera drones which follow you and respond to gestures now exist.
Crazy how more code haven't really yielded better products. Who would have guess that the number of lines of code wasn't the problem. Except that very basic tools that I use daily, pretty much every piece of software has become worse in the past few years. The closer to the consumer the software sits, the worse the quality is becoming.
There's something poetic about a company that's leading in the realm of funding, researching, using, selling, etc. AI is actively watching the destruction of one of its core products due, in large part, to that same AI they're selling.
Is it a "productivity panic"? I'm personally pushing a lot more commits because I'm spending more time playing around with ideas on side projects.
Is this all wasted if commits drop in a few months because folks are just pushing a lot now to figure out what works and what doesn't and we eventually settle into something between where we are now and where we were a year ago?
Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI.
I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.
Thinking more, the object of the "house" is to extract value, whatever it is by rent or letting them burn it down, as long as the correct dues are paid, the mission is accomplished.
The long view is that MS will sell those developers the tools that slow down or put band-aids on the damage they inflict on their own codebases by using AI.
Conversely, what suggests GitHub has a huge operating cost?
Running a GitHub clone at their same scale as a customer on cloud pricing would likely be insane. But y’all know infra is actually quite cheap when you run it yourself right?
It’s usually the case with these M&A deals that the profit just never quite makes sense to justify the purchase price, unless you can truly scale up the user base or revenue model. GitHub was already so mature as a solution when they bought it, I don’t know that they could have added that type of value just by slapping a Microsoft logo in the footer.
Absolute numbers aren’t useful in this analysis. We know capital is easily available for the expenditure. You have to consider it as a relative number to revenue or users otherwise big numbers always look scary.
I agree - however, I was responding to your direct question about whether GitHub's operating cost was high, and there's some data that suggests that it is!
If this is one of their major expenses and only represents 5% of annual revenue, it’s not. Often SaaS companies can run at 90%+ gross margins even with a highly utilized free user tier. Their selling expenses and labor/opex and SG&A run rates can be all over the map but these are extremely controllable and represent the operating strategy being executed more than the cost of service.
It used to be, and buying a few servers was a cheap way to get off the cloud (if you can manage them efficiently). But with RAM and other costs these days... I've had to rethink it a lot.
It's at worst ten times as expensive as it should be. But AWS was already ten to a hundred times as expensive, so even in the computer parts crisis, self hosting is still ahead.
They could do this in a way that lessens external pull requests.
For example, bolting co-pilot on to github, or a Codex in the web kind of thing that gives unlimited check ins.
It's like how Grok Heavy gives the user X premium or whatever. You charge for the tokens and give the unlimted premium access as a bonus. Basically, bundle it.
Right now anyone can publish to public repos in an unlimited manner. They could choose to limit that and elevate unlimited to a paid co-pilot of codex bundled plan.
The GPL is irrelevant nowadays anyway since everything is MIT (corporations won the license propaganda war). I think my computer has, like, five GPL programs? Ten? It's just that two of them are coreutils and Linux, and that basically holds up the GPL as a concept.
operating at a loss and non-operational because of outages are very different. If they can't maintain service levels nobody - AI super user or quant, old-fashioned human - will be happy.
Possibly dumb/silly question.... Are there any sorts of reverse proxies out there that provide their own layer of jittered/exponential backoff based on patterns? (i.e. requesting IP, cookie, etc.)
I suppose the main reason I think it might be a bad idea, is that it would add complexity to the reverse proxy (i.e. now it's having to track whatever thing is being used and that complexity itself becomes a potential failure point...)
(To be clear, the clients should have their own backoff procedures, but I'm thinking about cases involving naughty clients, which are sometimes a harder problem to correct for...)
I believe envoy has it built in and istio (uses envoy) has different levers for circuit breaking and retries. I’m sure lots of them have it as an option though outside of these.
In general, helping prevent retry storms is one of the functions of a service mesh. E.g. Istio lets you configure exponential backoff with jitter, linkerd provides a retry budget and loadshedding based on backpressure.
“Adding complexity to the reverse proxy” is kind of the central feature of service meshes, but the point is you pick one that’s battle tested and solves more problems than it creates.
Exponential is also overkill (even with jitter as others have mentioned).
I seem to remember there was a "you failed 5 PIN entries in a row, please wait 500000 seconds before you retry" on Apple phones. So, you probably also want a sensible max... which makes exponential a bit pointless. Just do a basic fixed delay + (large, e.g. 0.5 x the delay) jitter and you'll be fine for most things. You can add a bit of cumulative delay if it's really costly to do retries.
I always add some jitter but never actually had a problem where it would have been relevant. Recently I added it to a project where others also see it (not just a hobby thingy but something at work) and I was wondering if it would look silly, like premature optimisation. I looked on Wikipedia for how established the practice is and it barely gets a sentence... with no reference.
Do you know of a documented instance where it would have helped?
This doesn't answer your question, but I faced an issue where an application had to retry if things failed and I ended up with the "thundering herd" problem and introduced jitter without knowing it was a relatively standard practice.
I felt dirty implementing such a solution (introducing randomness for the sake of randomness is off-putting), but it worked. It wasn't until a while later that I even heard the term "jitter" in this context and realized this was a pretty decent solution for this kind of problem.
Regardless, if you're going to introduce something where adding jitter is appropriate, I'd just just add jitter. It's not premature optimization; it's an essential part of that kind of functionality.
It’s not hard to get started, it’s a case of adding small amounts of randomness.
If you have, say, a long poll then kick off all users due to a deploy or error (or a broadcast message) then you can have a situation where you’ve got a huge clustering of connections at 1 minute, which spreads very slowly out as real life issues give you jitter for free. You can avoid this or at least return to normal much quicker by adding some jitter.
It might happen if all your users back off at the same rate too, if the clustering causes a bunch of errors. Error -> lots reconnect 1 minute after -> fail -> lots reconnect 2, 4…
More likely to occur in cases where there’s a way you can have people all connecting at the same time - synchronisation to a real world event is one case and then connecting again at the same time after.
You have thousands of users connected to a chat via websockets to a small cheap server that can just handle the load. Server has a hiccup, all clients disconnect, server comes back, all clients reconnect at once. Server can’t handle the load.
Downstream database of our edge serverless platform went down. A tonne of requests failed all at once. Every service in the microservice request path, and the client, had their own retry policy.
Clients all retried at the same time. Retries amplified in our microservice graph (1 request at the front door ended up with like 10s of retries internally as each downstream microservice along the path retried requests). Request queues backed up and couldn’t drain fast enough. Clients all timed out at roughly the same time. All waited the same time. All retried again at the same time.
It was a pulsing thundering herd of many hundreds of thousands of requests at the front door that was amplified by internal retries.
Had to tune up load shedding to 100% after the database outage was mitigated until the backend recovered then tune it down in increments to restore service.
Added jitter to clients and turned off retries on the serverless platform.
Exactly, without jitter the thundering heard problem turns into trying to escape a small island with a small boat and big waves hitting the shore problem. You can never fully recover before you get smashed again.
Even if you're operating at a scale where thundering herd won't crash your service, introducing exponential backoff + jitter to your retries also helps when you're just viewing the logs. Massively reduces log spam when you make changes.
If you’re talking about internet clients, I think the real world provides sufficient jitter. If you’re talking about a fleet of clients on your 10gbps network, jitter might be useful.
I've had to simulate jitter recently to reproduce a memory spike in a go app using traffic control (tc). have you observed how your app works under jittery network?
Thundering herd is different. Thundering herd is when a lot of clients trigger requests at the same time. Common situations being a specific time, some other event just occurred or synchronize on other parts of your infrastructure (such as readers queuing behind a R/W lock that then get unblocked at the same time to continue to make a bunch of requests at the same time.
Request amplification via retries is a different problem that causes large amounts of traffic (but it is generally more steady than spiky)
> Errors in those services triggered a client-side retry loop that increased traffic during recovery
Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours.
> Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
The detailed root analysis tries to pass this off as a "bug". You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour is functioning exactly as designed. In this case aggressively to try and hide problems if token service responses become flakey.
A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded).
If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go great. But there are lots of situations where one or both of those breaks down.
Anyone who designs such a system should know to use an exponential backoff to avoid the thundering herd. Maybe copilot missed that while it was reviewing its own PR
Exponential backoff is the wrong answer in a highly available system in the typical case where (a) failure is expected and (b) you have nodes you are supposed to fail over to.
Your highly available system is probably somewhat important, otherwise you won’t have invested in making it HA.
While your premise holds for happy cases, when you do have a cascading series of outages, not using exponential backoff is just adding a self-inflicted DoS to when you do go down.
I don’t really follow your premise and can’t really articulate many cases for when you shouldn’t use exponential backoff. Maybe if you’re working at Jane St or something; or other circumstances where you can deploy immediate changes to the client; and you’re willing to trade ‘better p50 for worse outages’.
But in the case of shipped code that’s run on clients, I’ll continue exponentially backing off all the way, all the time, for everything.
When you have an outage, you should not retry at all. Exponential backoff is exactly how you get cascading outages. If service A fails a request to service B and decides to exponentially back off, now service A is holding open an end user request that will claim resources on service A. Fast forward ten minutes and the service B degradation has metastasized into a service A degradation. And even after service B has recovered, service A might still be dead.
To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients. Then service A knows service B is dead, does not retry, and proapgates the failure to clients. This is hard to do perfectly, but there's no alternative that works.
> To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients.
Even this is not enough, since you cannot always reliably know whether service B is dead or suffers an intermittent issue that can be safely retried just from looking at a single failure.
The classic solution, in the monolith/few-services world would be a circuit breaker. High failure rates on any service trigger a circuit breaker in the services calling it, and they'll wait for a cooldown period before trying again.
When you move to a massive microservice architecture with hundreds or thousands of microservices, setting up circuit breakers manually becomes very hard to track and do reliably. Service meshes like Istio make this slightly easier, but they still don't let you verify that all possible paths have circuit breakers and that retries are not excessive etc.
Pretty much, but ideally it should be transparent to your app developers. App developer writes `rpc.doThing(...)` on one side, and an implementation on the other, and the infrastructure -- the RPC framework or the service mesh or whatever -- transparently handles when/whether to retry and where to route retries.
The right answer is for the RPC framework to accurately communicate "try again on another node" vs "don't try again, just hard fail".
When one end user request fans out to hundreds of backend requests (typical for microservices), you can't have each of those backend requests do its own exponential backoff. If they do it in parallel, they're a thundering herd, and if they do it in serial, the end user request will time out before you finish all the work, at which point you're doing a bunch of slow expensive work for no gain (and the enqueued slow expensive work will make your outage worse).
Ideally you have levers further up from your local load balancers as well. Even at the edge. Granted you never want those to trigger but it’s better than fighting a storm while you fix things.
That’s more surgical than load shedding. With load shedding you intentionally return 503s to a proportion of all legitimate requests. It turns a hard blackout (total outage) into a flakiness issue.
In highly distributed microservice architecture, there's almost never a single upstream. In some cases you may have a couple of customer-facing entry-points (a global API gateway, and a couple of BFFs), but these are not the only paths that need to be protected.
There are client-side retries (which have broken GitHub in this case) and server-side initiated API calls between microservices that don't pass through any of your ingresses (e.g. triggered by an ETL pipeline, or a scheduled job).
With a complex architecture you can't just slap a circuit breaker on a couple of ingresses and call it a day. Don't get me wrong, putting them there does go a long way, but you won't be covering all your bases.
I completely agree, I currently manage a fleet of microservices that handles a few trillion requests a month. It’s about defense in layers to these sorts of things. All the way through the stack if possible starting at the edge.
At least it should be required for critical level services in production.
“Circuit breakers, where calls to a downstream service are stopped entirely when an error threshold is exceeded, are widely promoted to solve this problem. Unfortunately, circuit breakers introduce modal behavior into systems that can be difficult to test, and can introduce significant additional time to recovery. We have found that we can mitigate this risk by limiting retries locally using a token bucket. This allows all calls to retry as long as there are tokens, and then retry at a fixed rate when the tokens are exhausted.”
This can either be a relatively short time or an eternity, depending on the upstream consequences. If that means holding a connection open for 60000 milliseconds while waiting for some downstream rpc to go through its backoff ritual, that's an eternity, and under load that connection pool will get exhausted quickly. So now a problem which should only affect maybe 1% of users has completely hosed everyone.
I've seen this happen multiple times. Someone designs some clever backoff strategy without considering how it fits in the context of the rest of the system. Hilarity ensues.
If you have something taking an entire minute on a computer, please ensure you implement it in such a way that no connections are actually held open for that entire minute.
With http you can do it elegantly with a 429 and retry-after. 100% doing it at the edge is the way to go. The machinery which determines how long to delay a client's retry can benefit from knowledge of internal services' state, but I agree the ultimate decision must lie with the serving layer. That's the only way to efficiently deal with misbehaving clients.
On that note, one of the more memorable incidents of my career was when a 10M+ node client decided to retry as hard as possible on 4xx. That was fun x_x.
[edit] that is to say, for this mechanism to be robust your retry-after enforcement mechanism needs to be capable of withstanding almost every single one of your users attempting to illegally retry as fast as they physically can without negatively impacting that one user requesting legitimate traffic. https://media.tenor.com/p3mss3YI6TcAAAAM/wat.gif
Try asking Opus or Fable that question. It'll give you a good answer on why microservice architectures work the way they do in order to keep user-facing latency acceptable and minimize downtime. It's a complicated enough topic that I don't feel like explaining it for free to you in a HN comment.
Why are you pretending this is a harder topic than it is and you have some secret insight. And you're a dick on top of it:
Yes, exponential backoffs alone are insufficient. Yes, adding jitter helps randomize the calls across a fleet and should be the default with exponential backoffs. Yes, both of these may be sufficient for most systems. Yes, you can dive more into circuit breakers and adaptive retries to limit thundering herd.
Someone linked the Google SRE book. It explains that client services watch the global distribution (for that client) of number or retries. It will normally retry immediately on another node, because as you say, some failures are expected. But if it notices that more than about 1% of requests are having to retry twice, that indicates the server service needs reduced load and it starts refusing to retry. If it gets really bad it even starts preemptively failing first attempts so that the service doesn't get contacted at all.
> because the node you hit is rolling/broken/overloaded, but the others aren't
Retries in such a situation should be handled internally with the client at most responsible for failing over with a circuit breaker to another zone. Having the client auto retry right away is not something that behaves well as shown here, even if in the happy path it happens to stimulate increased availability without actually investing in the proper architecture for it
> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour
That wouldn't be a unit test - that's more like an end-to-end or integration test.
Have you ever worked anywhere that had perfect test coverage? It just doesn't happen, nor is it possible unless you're building a calculator app or todo list.
My employer has 0% test coverage lol. I've begged and pleaded, but the claim is that "risk is low" and "that's what QA is for." Hell, I've complained to senior management about how there are senior devs that forgo backend validation. It's truly Hell in the trenches sometimes.
Some days, I would seriously rather work at Wendy's.
If such a job paid even half as much, I would quit today. I can continue to program computers as a hobby. Occupational programming has essentially eroded my passion over the years.
Money is not important to me insofar as I have enough to live an average life. I don't need anymore than that.
You're the engineer, not them - you're being paid to write code that ideally doesn't break - it's your job to explain that tests benefit them. There's very little excuse not to add test coverage as you go along these days given an LLM could help write most in a flash.
Or, don't tell them and do it anyway. It'll make your own life 100x easier, unless you enjoy the stress of bug fixing and responding to incidents where you've broken something.
Writing code without tests because it takes too long is like running a car engine without ever bothering to change the oil - it'll run for a bit, and eventually fail catastrophically.
Maybe the retry logic was vibecoded instead of using an existing hardened library. After all, according to Twitter, nobody is looking at the code anymore.
A lot of retry libraries I’ve seen require the user to configure them. You can use a library with all the right settings, but if you configure it wrong, you are really no better off than if you hadn’t
> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour
Not to join the parade, but what would a unit test that confirms a cycling behavior across all the instances in-flight even look like? I mean, besides "Not a unit test".
Mock out the network call (which you'd be doing anyway because unit tests never connect to the network) so it always returns a failure. Mock out the timer.
Call the function multiple times and ensure it'd passing the expected wait durations in for each time it's called followed by a fatal error after say 30 seconds.
Backend API rate limiting has to surely kick in and force you to wait x amount of time before you try again… Discords bot API actually sends you how long before you retry.
As a client you can hit a server as much and as often as you like. The only thing the server can do is return an error code or try to hold the socket open (which the client can then close on their own).
It can block your IP address at the firewall level. It can't stop you DDoSing it with raw packets, but that's very unlikely to happen unintentionally, because TCP will wait for a SYN-ACK response for at least several seconds and possibly up to several minutes.
Except the other side of this is interrupting a service which would otherwise have succeeded: there's a lot of unattended or minimally attended processes where an interruption is just asking the user to do the only thing they were going to do anyway - retry it.
In GitHub's case this is especially relevant - the only reason to throw an error message at the user is the hope they - the human - give up and walk away (or you break all the CI/CD builds and the time it takes humans to hit "retry" gives you some breathing room).
"You can't seriously tell me that the unhappy leg of the code path has no test coverage."
Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.
"Please don't sneer, including at the rest of the community." It's reliably a marker of bad comments and worse threads, and you can make your substantive points without it.
I cannot even begin to express how many times I've seen engineers working super hard to optimize happy-paths so that we turn 3 nines of availability into 4 nines but introduce unintended emergent behaviors in unhappy-paths that turn 1 nine into zero nines via thundering herds, retry storms, etc.
You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?
1. People add retries w/o backoff. Now we have a retry storm.
2. People don't add jitter so we get huge waves of highly correlated retries that cause self-perpetuating overload cycles and failure.
3. People add retries (even w/ backoff) at more than one layer (e.g. one in process, one in envoy), so now we have a deeply confusing multi-level of retries with super weird n^2 patterns.
4. People find ways to fetch from multiple data sources to make a composite object but don't cache/reuse data they fetched, so one data source being down causes DOS on _other_ data sources because of retries.
5. People add failover mechanisms where all failures happily pick the same failover target because, ya know, it has the most free capacity.
6. People underestimate connection setup cost so "failover" causes huge increase in overall load due to connection setup (often tls setup) causing huge influx of "new" cpu work in a loaded system. CPU spike causes unhealthy destinations, causing more failovers, causing metastable failure.
7. People realize purely local decisions aren't optimal so add a layer of global health metrics (e.g. global retry budgets) but these systems add a time delay to the metric (ie its the view of the world 5ms ago). This delay makes their naive control algorithm go into oscillations or divergence spraying stuff everywhere.
All of these things have a similar pattern that when things are almost all good, they will make the system more robust. You'll get an extra 9 of reliability on good days. But they make everything far worse when things are bad.
Configuring postgres to automatically failover instead of doing it manually. The automated system caused more downtime in a few months than manually doing it did for years before. All in the name of more automations and less downtime
For resolution of a problem you can’t prevent categorically, you have to spend a good bit of time pretending to be the computer before you can trust the computer to do it. A run book is a draft of a requirements doc for a program to do the same thing. And you can’t afford to test that program in production without loads of simulations first. And that’s running the process manually and checking aggressively for any signs of problems.
Some coworkers snaked the autoscaling work out from under me, and then ignored the advice I offered on low water marks and weeks of testing and rehearsals. All I can figure about their irrational exhuberence is something to do with claiming something for annual reviews? All I do know is we didn’t make it 40 hours from when they flipped it on until we had our worst outage in two years. Classic FAFO.
They were hoping to eventually get to a low water mark of just over 30% of our static cluster size, and they decided to use that for their initial go instead of my estimate of 40% being the low, and a recommendation that they start with 50-60% for the initial weeks and then ramp it down. When I refused to let it go in the status meeting, the team decided we should vote on it. Two guesses how that went.
A bug in the deployment logic the Ops team had for doing things like upgrading VMs caused the next update during daylight hours to spin up the new instances at the minimum cluster size, instead of the current cluster size. It was done outside peek hours but that still had us cut over with just under half of the hardware we needed at that moment. Because we were in such a fucking hurry to be something like third in line to use the new autoscaling support, nobody else had run into this problem yet (or system did a pretty good job of buffering everyone else).
A couple weeks later they’d looked at the stats and decided that we were spending less than a half hour a day running at the low water mark, and the additional shutdowns were causing churn that made it harder for us to detect problems like memory leaks. Surprise, surprise, they increased the min cluster size to exactly what I’d told them two weeks before.
Since “listen to me” isn’t a lesson that transfers to other teams, I will instead say, don’t transfer initiatives to a new team when there is less than 10% of the project left. The lack of friction you encounter may result in a lack of respect for the danger, and attention to potential problems. And if you must transfer, if your ideas about rollout timeline end up being less conservative than theirs, be patient and do it the slow way. They probably have encountered problems you haven’t seen yet.
I think the "happy path" might be a slightly wrong classification in GP, since the post is in reply to a retry-storm issue and explicitly talks about retry storms and thundering herds.
I've seen many cases where engineers optimize the sad path, but pessimize the wretched path. Or in less flowery language, they cut the occurrence rate of common non-critical failures, but by doing that they introduce code that can make rare failures much worse.
The cases I've seen generally boil down to naive retry logic or poorly tested and poorly maintained fallback paths (such as killswitches that break their environment[1], graceful degradation turned graceless, dormant feature flags that get reactivated).
The case you see with a retry storm here is the most classic one and the one that annoys me the most. I've seen engineers adding aggressive retries even into places where the impact is minor (you could show an error and let the user manually retry instead). Retries that improve user experience can be great if done correctly, but I've never seen the authors of such pull request addressing the risk and mitigation techniques for retry storm or retry amplification.
I've seen cases which had:
1. Retries on the client side (browser or mobile app).
2. Retries on the BFF.
3. Retries on Microservice A used by the BFF.
4. Retries on Microservice B used by Microservice A.
5. Retries on Critical Service C used by Microservice B.
Most of these retries had very short timeouts (e.g. 100ms), in order to keep latency SLOs during normal operations (not a good idea on retries). Every time QA saw a layer without retries, that would be a bug, and adding retries is easy, so we'd get a new retry without much thought. But the first time Critical Service C became overloaded, Microservice B started timing out a couple of times and retrying. This was too much too much for Microservice A that had a short timeout that couldn't hold the 3 retries done by Microservice B, so it making doing its own retries, all of them dropped in the middle of the way. Eventually you'll get a full-blown retry storm where every request from the client side got amplified with 3^5 retries, easily bringing down Critical Service C.
We'd usually introduce a circuit breaker for the particular path that caused the issue, but a variation of this kept happening several times because designing safe retries across a vast collection of microservices takes a lot of effort, and it's always easier to just add a quick-and-dirty retry at any point where you think you might need one and call it a day.
A proper solution (which I've never seen implemented) would be an mandating a corporate-wide inventory of retry-paths, and monitoring it for any path that is at risk of triggering a retry storm, or adding mandatory headers that cross microservices and track the amount of retries done up the chain and the time spent in total waiting for previous retries. You could have a budget for both and automatically stop performing more retries. Both solution require extra effort and a large degree of coordination.
> I've seen many cases where engineers optimize the sad path, but pessimize the wretched path.
As said GP, yes this is exactly what I meant and "happy-path" was an unclear choice of words. They optimize for the unhappy path on the good days, and make the bad days much worse.
It seems to me like you're saying nearly the opposite of them. You're saying "no matter what tests you have in place, it happens", while they're suggesting low standards and few tests.
It’s been proven that VS Code has product managers who vibe code commits that get merged, including the co-authored with copilot bug. So there’s some problems there.
That was the final straw that stopped me using it.
I didn’t/don’t want the AI, I didn’t like the telemetry but it could be turned off but the cavalier disregard to just yeeting shit into a tool used by that many people no thanks.
I have been using Zed lately and quite liking it. I used Jetbrains for years, but it's gotten absurdly buggy lately so I probably won't renew it. I have also used vscode and I need to use it at work, and it's fine, but I prefer Zed (fewer bugs, faster, mainly).
I always advocate for the "boomer" editors when possible. It doesn't matter which one you choose. People can say what they want about them, but very little software survives almost 60 years. I sure as Hell hope VS Code doesn't.
For me, boomer editors (mostly nvim to be honest) join a long list of other things like tiling window managers as something I genuinely want to learn, but that I can never justify learning over either learning something else or doing "real work". I've been slowly introducing both and making gradual progress, but its extremely slow
Just pick a specific task that you do routinely (say markdown editing) and just use neovim for that one task - eat the elephant one bite at a time and markdown is generally small enough/light enough that you won't get out into the weeds trying to make neovim the perfect IDE in the process.
At some point, esoteric UIs may have been needed in the 70s, but that ship has long sailed. I don't need to give myself carpal tunnel syndrome trying to quit vim. If anything, a lot of that is also a problem that continues to plague software to this day: devs who go "well if it works for me, it's good enough for anybody, and if you don't like it, you must not be a Real Developer!"
If you've only worked at a small-to-medium privately held company you'd assume every company ran that smoothly. "Too big to fail" makes sense once you've worked for a behemoth and not before.
Copilot probably wrote the code confidently and it was never looked at by a human then yolo’d out to meet the pointy haired bosses arbitrarily short timeline to then look good to his boss and so on
SQLite is the exception. But they still test with fake errors, not real errors. For example they intercept malloc and make the first malloc call fail, run the tests, change it to second call, run the tests, ... and with two modes: only one call fails, or all calls fail after that.
They can afford to do this exhaustively because the test criterion is universal: they are testing ACIDity - the database is either in the new state or the old state. Never in between or corrupted.
Any other kind of system wouldn't be so easy to test for malloc failure, since you couldn't check for things like a successful response.
I think a lot have mostly experienced working for tech companies and do not understand how different that is from working at some other kind of company (particularly something not technically sexy) doing tech (see last parenthetical).
At a tech company, engineers mostly work on software that is the company’s product, and as such its functionality, reliability etc. are high company priorities, and the whole company is focused on producing and selling those products.
At a non-tech-company, that’s not the case. Which means you end up with something much closer to the movie “Office Space”, where software is just considered a necessary evil that’s worked on by what’s essentially the software development arm of an IT department, where any opportunity to cut costs or headcount is considered a good idea, where “best practices” take a back seat to “any practice that Bob can make work.”
A lot of companies these days are somewhere between the two: their main product is not software, but the company depends on a website to interact with a large number of customers: banks, insurance companies, web stores, etc. In that case, you tend to get the worst of both worlds, naturally.
^ this lol, real world systems are so much more complex and involve much more coordination than a personal pet project. It’s so difficult to figure out the source of a bug in a production system like this.
That means nothing, especially with all the layoffs.
One thing that I learned over the years is never mystify "code quality". Maybe you can say a certain team/project produces high quality code, and maybe Google's overall quality is better than my company's (considering their bar of hiring), but you want to avoid generalizing that, especially at scale.
The average software engineer at a large company ships just enough code at good enough quality to earn a salary, nothing fancy about it.
In .Net-land we have access to a lot of the underlying MS library code, and reflection/decompilation if needed.
MS has layers, there are A-teams, B-teams, and so-on. Simultaneously they are dropping both world class work and, to your point, a statistically much larger pool of just pretty good code.
A lot of that work is perfunctory and somewhat bloated/weakened by the scale of publishing and their internal tech-political requirements. Components for component I’ve seen in-house replacements for many MS projects from mid-tier devs that are better, smaller, and less brittle over time.
I’d compare it to baking: a home cook or tiny bakery have the advantages of time and focus, perpetual consistency at scale is a whole different baking challenge, one that doesn’t necessarily yield better individual loaves. It doesn’t mean WonderBread is bad at bread, just that they’re optimized for something other than maximal loaf quality.
I have never heard anyone ever utter the words, "Microsoft makes great products and services." They basically built an empire on brand recognition and vendor lock-in. Of course, a lot of their success was due to catching lightning in a bottle a few times at the right moments.
It is miraculous how they continue to fail upwards somehow. I suppose it's because they just vampirically sustain themselves by acquiring other companies and sucking them dry until they are a husk of what they once were -- like Github, Activision/Blizzard, LinkedIn, Skype, etc.. I remember when those companies used to actually produce good products and services.
It's because they understand capitalism better than they understand software. The former is how you win capitalism, the latter is just a cost center.
Same with Oracle. Both of them are good at getting contracts and then executing them not poorly enough that it's breach of contract but not well enough that the customer stops paying. That's their business, not software.
Answer is, you do comprehensive unit testing irrespective of Microsoft or any other company doing it. Also, as the OP of your comment posted, you are over estimating the software quality of these big name corporate companies. It can’t be the bar against which you would want measure your own company standards.
I solve real problems at scale and the engineering practices I see on a daily basis are a clown show.
There's little to no basic understanding of networking, distributed systems, databases, etc. 99% of our engineers were hired from their college internships and never worked anywhere else. Industry hires to improve systems rarely last more than a year and it is almost never their fault.
We're in the next tier down from the biggest tech companies and what we do is hardly uncommon among our peers.
I should be shocked that 99% of engineers I deal with treat all resources as infinite bandwidth, 100% uptime, but I'm not. They NIH super hard and write tons of code for things that a docker container running nginx (or similar) would solve in 5 minutes. There's almost no useful testing and worse documentation.
I had to explain to a “senior” engineer the other day (read: a few years experience) why locating a database client in a different geo region from the server is a bad idea (especially when that client is using an ORM that likes to make lots of little calls to the server.)
I also had to argue for changing a system that was reading about 100k small files from cloud storage to use a single compressed file. There seemed to be no awareness that copying 100k files might be inefficient.
One company I was at sold its product claiming any changes you made were “near instantly published” globally. They tried to demo it as such.
The way the engineers built the update/publish operation was synchronous from their primary data center to a number of globally distributed data centers. Publish didn’t “complete” until a receiver in each data center responded with an ACK after parsing and uploading to a nearby region cloud bucket. Any failure/timeout caused the entire transaction across all data centers to retry. All of traffic was over multiple VPNs, hub and spoke style. They built this system in 2020.
They constantly complained and generated incident reports about p95/p99 latencies to the Asia regions. Latencies that were perfectly reasonable when you considered the multiple global round trips that were being made, the size/volume of objects in the publish, set of operations and speed of light.
They swore that because the client UX to publish the change to the primary data center used JavaScript async that the entire process was async. They denied repeatedly that their “all receivers ack complete to succeed” business logic was synchronous. I shit you not.
Distributing across different services wouldn't be a bad idea....
I still can't help but feel a little grateful for what they do across the free side of things. I know it isn't altruism, and I know nobody needs to defend a billion dollar corporation but...
Name another service that does what they do for FREE (and no ads) at this scale. It isn't easy. Wikipedia has probably more usage, but is a simpler endeavor. (except the moderation part, that's just amazing) Open Street map? Smaller and simpler. Internet archive? Again, smaller and simpler. Linux distro mirrors? Again, smaller and simpler than whatever github is doing for free.
My only push back would be on the FREE part. I coulda bought that 5 years ago. Now I look at GitHub and go, "if the product is free, it's because Im the product" with all their co-pilot stuff.
Another fair take. I mean, hell, a lot of companies are bought literally just for their user lists (to sell new products to). I can't think of a nicer user list than a list of potential developers.
My guess is, user list would be the least. Microsoft has this thing about getting you all warm and snug because they have a solution for everything. Need an IDE? VS Code and VS proper. Got your hosting needs (azure). Literally everything. You wanna build applications around AI. We got you too, along with the VCS to keep your app in and pipelines. Hey, go ahead and send us your data so your AI application is better because you're really just calling our models. Right, and this goes even further. Data is incredibly valuable. And here we are, now paying companies to vacuum up all our data. Every last bit. And I am just seeing GitHub as another way to get people in the door to do just that.
Probably the biggest thing that has me dumbfounded about everything in the AI space and the tooling in GitHub and stuff with co-pilot. They've figured out a way to make us pay them to steal all our valuable data. And to package it up all nice for them with a bow on it and not question it.
The point is, GitHub is not free for me (we pay) yet we were also down?
IMO definiely time to segregate the free and the paid service. Otherwise those of us who need the service to stay up will have to look for alternatives.
The post mortem conclusion should've been to put usage limits on the free tier so paid users aren't killed by them.
It's been bad because it's crowded out better things. When you put something shit on the market for free you destroy most of the will for customers to pay for a thing, even if it's good, because they can just use the shit free version. This can be an intentional business strategy.
So, I lived through the Internet Explorer age so I'm familiar with that. I think it is a leap to say there are better alternatives dying to get out. There's no objective way to decide they are stifling innovation.
I'm not sure there's some solution for what you're complaining about other than the government coming in and saying they have to split github. What are you advocating for here, or is this just complaining?
Are retries bad?
These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic.
I feel like I'm mildly stupid in trying to out retries as heresy but I'm not sure.
Seems that retries are good when the error is rare, and bad when the error is common. Typically outages have you transitioning from "everything is fine" to "nothing works", so being able detect that transition early is helpful
Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times and bad in bad times.
This is one of the scenarios where I feel we as engineers should have been more diligent in publicly writing down what techniques have worked in which scenarios and what haven't, and the AI could have gathered decades of operational knowledge and provide proper guidance to architects designing new systems.
It's also true that we're witnessing unprecedented levels of scale.
Thundering herds and circuit breakers are extremely well documented and practiced in production. AI absolutely knows about this.
But knowing when to use which strategy and when a simple retry suffices is precisely the type of thing humans will remain to be better at than AI for the foreseeable future.
Exponential back off retries could hide a real issue, but I would estimate something like 99.99999999% of network retries are resolved within the first 2 attempts.
Not using retries is optimizing for the astronomically rare case, which is better mitigated by other means
Yes I’m aware, and they’re probably suggesting that something on a higher level should instead retry, and I’m arguing that you need to coordinate the retry mechanisms and behavior between differently layers of your stack.
I totally agree with you, I think retries are overused, with the exception of operations that are known to be unreliable and can't be improved.
In my experience, errors which go away within a few seconds are quite rare, and are mainly due to flaws which are usually caught in testing.
I think a very careful cost/risk/benefit analysis should be done when adding automatic retries to things. As well as potentially causing cascading failures, it is a degraded user experience when it doesn't succeed.
As a user I would rather see an error straight away than see many seconds of spinning while something silently retries, and THEN an error.
I have the exact opposite view. Way too often, I’ll be presented with an error to the effect of, “something went wrong, please try again” and often the retry works. And I’m left wondering why this machine whose sole purpose is to automate things can’t do that for me automatically.
In particular, networks tend to be a LOT less reliable than the typical developer accounts for. And the failures are very often transient. A case I run into often is doing something with my phone while leaving the house. There’s a window where it still thinks it’s on the WiFi but it’s too far away for it to work anymore. Initiating an action in that window often produces an alert telling me to try again, and trying again a few seconds later almost always works.
If you want a retry to prevent that, it must only be done at the outermost layer. Note that what you think is the outermost layer might not be, and that accidentally deploying a retry at a non-outermost layer is much worse than no retry at all.
The Github outage was about internal clients. Phone apps are a reasonable place to say things are known to be unreliable and can't be fixed. Your IP address changes when you leave the house.
Btw, PWAs added offline capabilities to websites. I hate how the only thing that got used for was these stupid pages that look like you were able to reach the site but it's actually just saying you have no internet, like YouTube.
I’m sure some retries are helpful there too (TCP is doing them, at the very least) but yeah, different approaches for different situations. Maybe you retry but you don’t spend many seconds hoping for it to work.
I'm ok with TCP retries generally. But the assumption at L4 is that L3 can and will drop/reorder packets at random, and retrying is cheap. Also a lottt of tuning has gone into TCP already.
I was using claude tethered via my phone, and would lose signal every now and then as we went through a tunnel. I was glad for how resilient it was its its eventual retries.
The current infrastructure cannot handle the new load requirements. Either the infrastructure must change, or they must start denying users the ability to use the infrastructure.
I may have come out with a different interpretation than you did of GP's comment. I see how you got to yours. But the way I read them saying they should slow down was, maybe slow down on new features. Which would mean they could shift resources from new features to infra.
I think he meant to say "prioritize" so that availability related work sees results sooner — "acceleration" — than it would without an increase in priority.
>We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.
> Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits.
I operated services at similar scale, and generally we use to put a bit of slack so that you would get an alarm when capacity goes up to 80%+ (or whatever number makes sense)
This allows to check, in the morning, after coffee, why the load balancer fleet didn't scale up automatically.
I am sure there is a good answer to why this is impractical, but it would be nice to know
I wonder how much of the traffic increase is enterprise vs. hobbyists? A 7 hour outage for enterprise customers is really, really bad and it's sad if caused by a mass of non-paying vibe coders. It's becoming absolutely obvious that the unlimited free tier needs to go.
This is my suspicion, all though I have no evidence, just anecdotes. I make a handful of commits a day. I write code, review it, test it, commit, and then push.
We have some marketing folks that have gotten into vibe coding stuff for their personal use. First let me say: good for them and I'm glad they're experimenting with new ideas and tools. The side effect of that is that looking at their repos, they're having Claude go whole hog and make upwards of hundreds of commits a day, all with things that they haven't taken a look at.
I don't think I can say this is wrong of them, because their tools encourage that and they shouldn't have to consider their impact on an enterprise service, but I wonder if this trend is similar in other places.
I appreciate the unprecedented load GitHub is currently experiencing, but it's not just the (admittedly extreme) load of commits/pushes that is to blame.
Their Copilot cloud agent offering is suffering with a case of some of the worst corporate ADHD I've seen. We built a cloud agentic development pipeline on it, and it seems like almost every other week they silently change something with zero public announcement that creates real disruption for our team.
Note: that's not bugs in the Copilot platform like the article discusses. That's real, breaking changes to the platform that clearly aren't being tested/reviewed before being pushed to prod, with zero public announcement or documentation.
Support is useless – we're paying customers in the 4-5 figures and our tickets go unanswered.
I love(d) GitHub, but I do think they've lost enough public trust at this point that their time is ticking. With talk of new VCSes designed specifically for agents, I do believe it is just a matter of time. Which pains me somewhat to say.
Yeah... it has the vagueness and awkward staccato of heavily edited Claudish.
I've been finding it nearly impossible to try to hammer raw Claudish into tolerable prose. I usually have to re-write most of the text by hand if I want it slop-free.
P.S. Hi Claude! I know you might be reading this :) Don't take it personally. Let your trainers know that you need some help with this.
This seems like a pretty straightforward and easily winnable situation for GitHub. The demand for their services just doubled, apparently. They have no real competitor operating at the scale they’re at. They have pretty substantial network effects.
They are under no obligation to continue functioning as a bottomless free repository for text file hosting, especially now that text file creation has multiplied exponentially. They could make a few almost purely commercial changes and solve this without any major re-engineering while maintaining their status as the go-to public / open source code hosting platform.
1. Immediately increase pricing of all enterprise licenses and add super-committer overage fees.
2. Rate limit or cap commit size / frequency for public accounts.
Their service is more valuable than ever and switching is much harder if people have automation built up on their platform. Now is the time to cash in their chips.
And the positive externality of increasing commit cost would be forcing people to have some semblance of restraint for the AI content they generate.
Why are we even committing AI generated code and uploading it to GitHub? I've heard we don't need to read the code anymore. Why preserve a detailed version history if AI has it all handled? Why even share code if we can all just have AI write whatever software we need for ourselves? GitHub feels like a dead end of AI is really headed to where we believe it's headed
But GitHub were one of the first and strongest to advocate for using AI, so excessively in fact that it turned people away from the platform. At every turn on the website it suggested to use Copilot, or edit it right now, and commit right now, and do this and that right now(!). The platform effectively begs you to use the free GitHub Actions, too. Not sure what the business model is here.
There are a lot of ways to avoid exponentially more commits, issues, and PRs breaking your backend down, and begging every visitor and user to please use AI to write 40x more code that needs 40x more fixes is not one of them.
Is it me or is all of this essentially "we don't want to show the user anything at all when something breaks?"
And what makes this funny (to me) is that this is a website for developers. I would think that of all the audiences you would target, developers would mind seeing the platform display error messages when things break the least.
Centralized source code hosting is going to end up looking like the three credit bureaus in terms of security. It's only a matter of time before the first big hack, when everyone shrugs and says, "Oh well, everyone's source code leaked lol too big to fail."
Given the jailbreaking behavior of models in Anthropic and OpenAI, as well as Microsoft allowing Copilot access to client data, we should assume our source code on Github is leaked or leakable anyway.
This article seems to say that there is "no excuse" for these issues but look at all these things we changed and are changing. It doesn't really feel transparent and it feels like they aren't really taking true ownership on what has happened.
As I mentioned before (https://news.ycombinator.com/item?id=49333107), they can mitigate these issues with limits, even for failure cascades. There should've been an all-hands-on-deck feature freeze 6 months ago to implement the limits needed. That clearly didn't happen.
I think it's because their leadership actually doesn't care that it goes down. A weekly outage is now an accepted cost of continuing to allow unlimited free access with infrastructure that cannot possibly handle the load. As a result, everyone is looking at their GitHub Enterprise bills and cost of stopped work, calculating how much they'd save by self-hosting.
> Copilot services took longer. Errors in those services triggered a client-side retry loop that increased traffic during recovery.
Let's pretend that the scale traffic is with the number of commit/pr and not self-inflicted with all the copilot eye candy features that were vibe-coded-added to GitHub.
In addition they say that they will continue their migration to azure and that azure is supporting their actions run.
But GitHub actions is one of the things that was the most constantly broken without multiple outages recently. So I have the feeling that it proves the point that part of the stability issues is also due to their forced usage of azure.
Why not identify the lunatic top 1% of free user you know are just abusing the hell out of the system and put severe rate limits across the board for those organizations/accounts?
I don't know, this blog post and the lasts read like organisational and human failures to me.
Their system doesn't degrade, they even have infinite retries and no meaningful quotas anywhere I can see (but I'm not a heavy github user).
I feel like this becomes a lesson on how NOT to design and operate a SaaS.
I wonder what the ratio of repositories to physical machines is these days. I'd also be curious to see this as change over time.
I have a hard time with the premise that a mere doubling of git ops would be especially crippling for any particular repository. GitHub runs like ass because it's oversubscribed by a huge factor. Not because git is inherently constraining at scale.
Not an option. Companies are happy to sell you products at premium prices, take your money, and then still collect, mine, insecurely store, and sell all your private data.
I am surprised they didn't rollback their change from 2019 when they included up to 5 private repos in free plan. Before that you had to pay to keep your code from being publicly visible.
Unfortunately time and time again, the overwhelming majority of people show that they would rather deal with an unreliable scummy company for "free" than pay $1/mo for a reliable service which treats them with respect :(
OK that "treats them with respect" is certainly an important aspect though. It does not seem like GH is especially great on that one, including listening to its users.
I wasn’t being completely serious with my suggestion, but I was thinking of a totally PAYG model. Monthly plans could come with credits, like they already do for Actions.
"The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR " - silly question but github uses github for their own PRs and deploy right? Do they have a special dedicated system just for them so they can fix github with a code change even if the rest of us can't?
After touting 1 billion commits over 2025 at universe last year… they are now handling 3 billion per month jeez. I’ll give them a little more grace after all…
I don't know where else to ask this but it's killing me. Does anyone know what the hell that GitHub physical CD thing was about? Did anyone in the world get theirs?
That was a fun thing that the marketing and devrel team are still working on, turns out burning repos on the discs was a harder problem than anticipated (lol) but we'll get them out eventually, promise!
Haven't they been migrating to Azure for a few years? How is it still only 58% done... Microslop needs to lay off the focus on AI features and get it done
Linux didn't (yet) kill Microsoft. Microsoft absorbed that shot. Then the Git arrow went straight to cold black heart of Microsoft. The next few months will determine if they survive it. If they do, what will we see from the third draw out of Linus' quiver?
Linux had nothing to do with the downfall of Microsoft. Any perceived dominance of Linux is not because Linux is getting better, but because Microsoft is getting worse.
The 'growth in completed actions runs' graph is interesting. I assume the periodic drops are weekends, so intuitively the floor of those drops corresponds more with hobby/personal projects than people at work. It looks like there's a sharp uptick specifically in that floor since July ish.
I said in another thread that they can't blame increased demand for these outages, but the demand growth is genuinely insane for a company already operating at huge scale.
I guess we'll have to wait and see if they deliver now, but it seems like they're taking it seriously at least.
Is git still the best VCS today? I ask because it seems so much effort in the industry has been invested in making git scale, like Cursor's Origin, or the stories in the past with Facebook's monorepo, but they all seem like bandaids to its intended design.
> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.
I'm missing in these descriptions the most obvious approach: Resilience. Shedding load so that you can keep services up even though capacity is too low. If you flip over as soon as load exceeds what you can handle then this problem will never go away, unless you always have insane overprovisioning of resources which is uneconomical. There will always be spikes. You need to plan to handle them, no matter how high.
> we have focused on three priorities: adding capacity, improving efficiency, and removing architectural bottlenecks.
Sorry, but again, that is not good enough. They should ask themselves why they are expecting that trying the same medicine as last time will prevent next time. It won't.
With that mindset I'm not surprised this happened and it will surely happen again.
Edit: In more concrete terms. If you 2x your capacity and in a week you face a burst 2x of what happened last time, you are back in the same seat. If you improve efficiency by 2x, same thing. And after a bottleneck is before a bottleneck. There will always be a bottleneck. The key is to be able to handle a bottleneck. Removing one just pushes the issue to the next one.
Your architecture must be such that your whole system should be able to run on a raspberry pi. Most client requests would be dropped, but those that make it through will be served. If your architecture serves 0% because it crashes when load is 10% over capacity, then capacity increases or efficiency increases or bottleneck removal are not going to prevent the next outage.
I recently received a PR fully automated by Claude for an 8 years old repo. The bug is legit and the scope it affects is larger than what that PR addressed, but I no longer care too much about that legacy code anyway, so I also let Claude run free for the first time in my life, from handling that PR to fixing all related bugs.
I walked away for half an hour and back, found Claude opened and merged 9 more PRs and added a comprehensive CI for testing for all platforms. It will likely take me months to reach this level of output, but only half an hour for a capable agent.
With all the outages at GitHub there has to be someone willing to unseat them as the social git repo… how bad does it have to get before folks go elsewhere? Bitbucket and gitlab exist but are pawns compared to a king no?
The trend line does not tell you what will actually happen at scale, even if you think you're perfectly prepared for the next 10% or 20% growth. As Mike Tyson put it, "everyone has a plan until they get punched in the face".
The problem is there are a class of problems that only appear after you go over the tip of what your system can handle, which are very difficult to predict or model.
We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.
GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.
Forgejo is pretty neat. Their CI story is sad because it's based on act and it lacks features like GH Apps so service accounts are not possible (using a user account as service is barf).
I self-host it and its pretty easy to have uptime higher than github when you have 3 users total.
"Forgejo is a self-hosted lightweight software forge"
That says absolutely nothing. The "What is Forgejo?" question is unanswered and instead you get a lot of words about their values, their inclusivity, etc. And the next thing in the docs is how to install it. It's ridiculous.
“Software forge” is an established[1] term by this point: a piece of software providing a full set of features for collaborative development, usually at least code hosting with a web interface, tickets, and webpage hosting, and these days often also code review and integrated CI or at least the possibility of integrating CI. It’s admittedly squishy but in the same way “IDE” is squishy, which is to say it’s still well-defined enough to be useful.
> a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.
Interview question. How would you handle the growing traffic needs and traffic spikes. I'm curious whether any existing architectural diagram of theirs would highlight a potential failure post-mordem.
The distinction between running out of capacity and collapsing when you run out of capacity is exactly right. Every distributed system hits limits, the question is what happens next.
the retry loop that made the outage worse is just the internet being extra helpful. every client decided the best way to help was to ask again, louder.
Great read - I'm glad they realize there's work ahead but what I'm missing is:
* Paid customers: we know you pay us often a ton of money, and we burn your month on actions during these outages - we'll refund you for the days we spent your money and gave you no value.
* Paid customer: We know you put your trust in us, so we'll ensure we have a separate pool of capacity to ensure we can keep that trust.
* Paid customer: we'll proactively refund you when we miss our SLA.
What I read from this is:
* Scaling is hard, we don't have enough capacity
* We give away a shitton of compute for free
* I have to talk about Azure not being a steaming pile of poop, otherwise my bonus will get tweaked downward in the next comp cycle.
Notice there's nothing about paid customers, I'll add in what they are missing:
Paid customers: Go F*ck yourself, you don't pays us enough to be an interesting line item compared to windows server.
This. I own a small company with 5 people. I pay Github $250/m. I'm sorry but the narrative of, "look at this burden we have, it's hard to take care of all of this code!" is pretty insulting when I'm paying $50 per person per month to host code and run CI pipelines. If they do not want my money, I'll find a company who does.
And another outage. [0] Looking forward to the subsequent post-mortem on that one.
You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.
6 years ahead [1] on not going all in an centralizing everything on GitHub.
With all the software being written on github you'd think we were going though a software rennasance. Where are the results? Is it really just all slop?
I can't speak for all of it, obviously, I don't have time to try much of it, but I see tons of amazing new software in my various feeds pretty much daily.
I'm a bit weary of seeing comments like this, but I want to believe you, so help me learn? What are some examples of the amazing software you've seen recently?
Sure. Off the of my head the first three I remember from the last day or so: Opacity (who seem to be reinventing vector drawing), Pixel Flight Simulator for pico8 (with dynamic cloud cover, day/night cycle, and voice over from air traffic control), Jet (a high speed 3D library for ESP32/embedded, open source and commercially licensable).
I'd say I saw these because I follow certain creators and/or high signal reposters in the fields of software and games.
And, if you'll forgive me, I myself am working on a totally new way to make games, called Jinks (multi-platform Web, Mac, Linux, Windows, even back to Wii and Dreamcast; games can be introspected, edited, queried at runtime; one game released so far to prove it; work in progress)
Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.
I wish I had say in our git forge decisions at work, but I don't and I can either tolerate this or quit my job. So I will continue to disparage one of the world's biggest tech companies not being able to manage GitHub properly.
I have setup a gitea instance on my gitea server which I think is good for me and GitHub both. For one, I dont have to worry about GitHub service outages, and GitHub gets to be free from my toy (and mostly AI slop) projects that no one else will ever read/use/participate in :-)
"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products."
Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.
Github is ripe for disruption and I hope it is disrupted soon.
"While accelerating our migration to Azure," meaning, they will only solve problems if it helps them also use Azure more.
It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage, unless they have no reporting or their tooling is completely incompetent. If things can fall apart so easily, throwing more capacity at the problem won't fix it.
You’re torturing your own logic to make Azure the villain here. And it also sounds like you lack experience with capacity exhaustion. Things fail slowly, then suddenly.
I don’t owe you anything, much less a separately sourced counterargument. You openly admit that your opinion is not based not on GitHub’s proffered statements but on your self-admitted assumption that GitHub is actively lying in an attempt to cover up an Azure-related root cause.
I’m firmly in the camp of “something stinks at GitHub” but
> It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage
In my experience, there are hard thresholds that get passed that expose hidden bottlenecks like this. A previous system I worked on we had absolutely loads of headroom by all of our measured metrics, but one day we filled a cache because the value hadn’t been tweaked in recent memory. Plenty of space on disk and in memory, but all of a sudden we went from a very high cache hit rate to a very low cache hit rate, and everything ground to a halt.
This absolutely can happen in large systems. If some part of the system is at capacity, then slightly increasing the load can cause it to fall behind and start accumulating a backlog.
These backlogs can cause clients to make more retries, exacerbating the problem. Potentially further cascading through the system.
Sure. But you might not even be realizing that something is just at the cusp if the load is spiky enough.
The art of large system design is to identify and avoid these kinds of chokepoints. And when something happens, propagate the "backpressure" up the stack to avoid queuing.
Ok, but there's no universe where a major Microsoft-owned property is not being forced to run on Azure. Just like AWS pushing to get off Oracle back in the day. It would be career-destroying to suggest otherwise regardless of technical merit (and tbf, no infrastructure is bulletproof, unless you want to port GitHub to z/OS on mainframe)
Azure just has very poor performance and reliability characteristics. It’s a particularly bad migration target for a colo-based company that mainly runs on owned hardware (such as GitHub or LinkedIn). Requires much larger architecture changes than (say) a company coming from AWS.
I'm betting on Tangled and Codeberg. Tangled has a better press and in general is a dark horse, Codeberg has the "brand" and some network effects from projects that moved to there. (famously, Zig.) I heard that Sourcehut is having a moment as well, and I love the idea of email-based workflow and not having to have an account to contribute to someone's project hosted there, but I'm not maintaining anything worthwhile paying the $4/mo sub.
that is more manageable but c'mon I can't even keep Google One 100GB up on a consistent basis, that's how poor I am. self-hosting would be a far better option because apparently I find enough people to provide free Hetzner VPSes and stuff as long as I can sell this as mutually beneficial.
for context, I would GLADLY move there my Neovim plugin. all it does is brings the current jj message into your editor and lets you integrate it with a status bar (or anything in nvim, really). that would be a decent measure against drive-by slop contributions, and I'd accept contribs over private github mirror from those who I know but can't bother setting up git mail
EDIT: TIL that one can host SourceHut themselves. discoverability may still be a problem (sr.ht just ranks higher in search engines) but 1) fixable with github mirror that points to sourcehut instance as a canonical development platform, 2) it's moderately easy to sync contributions between tangled and sourcehut, so tangled is also an option
EDIT 2: the email part would be PITA, so $4/mo is attractive on that background
> We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.
A server box now has around 256 CPU cores. So that's about 12000 servers. If each one is $10k that's $120 million. Not a lot compared to Github's income.
$10,000 is only enough for CPU cost (2x AMD EPYC 9754). The rest of the servers (mainboard, RAM, storage, power supply, network card, rack, cooling, etc.) can easily triple or quadruple the cost.
But at this rate of growth, sounds like they'll need to buy around that amount every year just to keep up with current (not even taking into account future) growth.
Microsoft don't release the costs as you know, but
Compute and Storage for Free Tiers: Hosting code for over 150 million developers and processing over 2 billion GitHub Actions (CI/CD) workflows a month requires astronomical server power and data storage. The "Free" tier is a massive cost sink that Microsoft treats as a loss-leader marketing expense
Let me know when you understand how that's not free.
If you're a big company, you can afford having one engineer spend one or two days per year to maintain your self-hosted GitLab or Forgejo. On top of better reliability than GitHub, you'll get the additional bonus that your source code won't accidentally leak through being in Copilot's training set.
If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.
Speaking from experience, it cost mW about a week or two per year to maintain GitLab for the startup I worked at.
My personal GitLab on the other hand really does take only a day or two per year.
That said, a week or two per year is just what it costs to maintain any one thing period. I spent about that much time maintaining PCs in the office, or my personal proxmox setup. It's not onerous at all.
GitLab is super bloated and a little sucky to admin, but it's not too bad all things considered. I'm admin in my new job's GitHub org and it sucks a whole lot more to maintain.
The issue with these systems is that they lack Github's sophistication for issue tracking, knowledge transfer, and automation. I think Gitlab is a mature product in its own space and unlikey to change, for instance, at this point.
Codeberg also has the issue of having a political stance which means they will not accept just anyone's use of the platform. That is absolutely their right and I have no issue with it, but it's unattractive to me - as someone who agrees with most of their current politics - because the day they decide they don't like me, I'm screwed.
I never found GitHub's systems for issue tracking to be all that great. Cross-repository issues and development plans are hard to track within a git host. I've always used an external panning and issue tracking tool, mostly Linear, and it works really well. GitLab's Linear integration is excellent, FWIW.
I've actually worked with a couple of companies who do use GitHub for their code, and they all use Linear in addition to GitHub.
I understand the concern you're talking about wrt. Codeberg, but I wouldn't view it as a significantly bigger risk than anything else. Any platform can suddenly decide that your project is against ToS (GitHub will absolutely not accept just anyone's use of their platform either) and Codeberg introducing some rules recently doesn't, in my mind, drastically increase the risk of a dramatic ToS change in the future. But we all have to make our own risk evaluations and I won't judge yours. Luckily, moving between Git hosts isn't that difficult; setting up CI again and losing merge request history does suck but it's not the end of the world, unlike something like, say, losing your AWS/GCP/whatever account.
"sophistication" seems like a strange way to describe GitHub to me - I've found in every individual aspect (code browsing, issue tracking, code review, package management, etc), it's the worst out of all the systems I use regularly... But it's good _enough_ for most people, and it has all those features in one place, which is more convenient than wrangling 10-15 high quality but disconnected systems
Gitlab has the benefit of having very little traffic, both free and paid. Their limits are still way above the current usage so less likely to be an issue
Worth mentioning GitLab's paid enterprise offering are more expensive than GitHub's, on a per-seat basis.
Lots of companies moved because it was cheap, but it's not anymore. Ironic that companies might choose to migrate to them now for stability, rather than price.
GitLab has its own problems. It might be somewhat more available than GitHub lately, but it too is not immune to incidents, particularly involving workflow dispatch and managed runner availability.
Plus the UX isn’t nearly as good. Sometimes it feels downright clunky in comparison.
Dude no one uses github tracking for anything serious, come on. The only thing github has over gitea/forgejo is discussions. That's the only real social "innovation" github has contributed to open source development and it's just a shitty tacked on forum.
Also for accuracy, Codeberg has a pro-human and anti-corporation stance. Both of which are definitely en vogue at the moment.
Much better than GitHub's pro slop sentiment, which is doing nothing but destroy their reputation.
github controls the productivity of a large number of very large tech companies. When there is an outage like this they are basically shutting down a significant number of factories for hours at a time. This would be as if during the hayday of detroit they just turned off the power grid at a time randomly at least once a week for hours. It's unacceptable. The amount of productivity lost is staggering.
We should be building tools that help us all move off of github as soon as possible. The amount of action code that will need to be rewritten is daunting.
This seems really bad, to be honest. I think we may be fucked. There's just no way all these lines of code are doing anything useful. We're now just burning stuff in desperation and confusion.
It's funny, I bet you could take any software dev and blind AB test a page written by a corporate manager and a page written by an engineer.
Here's something an engineer writes, loaded with facts:
"I got to the office and we had a huge panic going on, I immediately called our IT in US-2West and they reported on cascading box failures, I checked our load balancer via remote admin and indeed it was failing to. I called my IT managers and learned we had hard resetting in progress for the past 20 minutes with minimal impact on recovery."
Using a corporation's free offerings isn't freeloading. Microsoft wants people to put their code on GitHub. They want GitHub to be the place where source code is hosted, it is incredibly valuable. Saying "GitHub is sucking and I might leave" is information that Microsoft wants to know if they want to preserve GitHub's dominance.
Of course some people take it too far. Of course there are reasons that the outages are occurring. But Microsoft wants GitHub to be a core, reliable pillar of the software world. Nobody's making them do that, they do it because it's good for them.
We're still staying on Github at work, but have had backup self hosted git repos as a break glass option when Github is completely broken and leveraged this several times now.
Whos fault is the massive load increase, or their inability to navigate it?
Did I iject copilot into every part of github? In fact not only did I not insert it, I have never even used it.
Did I move their infrastructure to Azure?
Did I sell them to MS?
Did I set all the directives and priorities that MS has set on them like telling everyone they must use openai for everything, and then telling them they must stop doing that and user their own ai instead?
The outage is not due to a natural disaster that no one could anticipate and no one had any input on creating the conditions. They keep the free tier because THEY want what THEY get from the free tier. They could easily have a $1 tier and various totally sensible throttle limits on various services and apis that would have avoided all this, but that would not get them the 100% user coverage that they want. So THEY choose to provide free, swiss cheese service.
It's not some unreasobable burden they labor under that anyone else should be understanding and forgiving about.
This is exactly the entitled attitude I'm talking about. Nobody forced you to use Github for free. Don't like it, use something else. Beggars cannot be choosers
That sounds unusual for a free platform (not a limited trial but an actual free tier). Isn't it usually the case that only some small percentage can be convinced to pay?
I mean, a lot of us have paid GitHub a lot of money for CI on private repos. And when GitHub themselves encourages the insane behavior of vibe coders and agents instead of just charging or rate limiting access of bots, it's hard to give them sympathy.
This stuff is good! This is what a booming economy looks like. There are people out there competing with you for resources because they have cool ideas they want to implement.
Or at least they asked the AI to come up with cool ideas, which is even more interesting. It's exciting watching the world transition away from humanity being in the driver's seat!
Layoffs by the 10s of thousands, food prices out of control, people barely able to afford gas. At least some tech bros can launch their 50th B2B SaaS. I didn't realize a booming economy sucked so much.
The exciting thing about AI is precisely that it'll let software move beyond Yet Another B2B SaaS and into doing useful things in the real world. I regularly ride in driverless cars! That was the stuff of science fiction when I was a kid.
If you're worried about food prices, you should be happy that robots will make agriculture less labor-intensive and bring prices down.
Automation has been making food steadily cheaper for 300+ years, and continues to do so. Sometimes the price of oil goes up so much that fuel and fertilizer price increases negate the wins from automation, but automation is still generating wins and will only accelerate as robots get smarter.
How would any of what you're saying help with this?
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery.
load balancer failure? rate limit woudl address
concurrency limits? throttle or queue up messages. auto-scale failed cause was misconfiguration policy, which i admit cannot be handled by my suggestions.
The cascade? it's downstream service degradation, which I mentione should have had been prevented with queues. One of the jobs that queues/kafka solve is to prevent these downstream outages.
If your LB is down, you're just kind of screwed. You can't enqueue things if requests aren't getting through at all. Same deal with authn/authz issues, which they also had. If you can't answer the question "is this message allowed to be added to the queue" you can't enqueue stuff.
GitHub does use queueing for all kinds of stuff internally, though, because they're not morons.
YOu are making an assumption on which LB is down, and so am I.
Where is LB placed? Is that hte LB in front of API GW? or behind GW? GW can have rate limiter or cloudflare FW can handle the rate limiting.
The doc does not show which part of LB is down so it's all guessing at this point so i won't go forward with this.
Can you imagine the flack he'd take if it turns out he'd been moonlighting whilst the GitHub ship is sailing though cat 5 with both the mast and the ship whore on fire?
Reading this port-mortem / plan shocks me, this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started. It seems the solution has been capacity, capacity rather than architectural or data changes.
> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations
How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.
> this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started.
Well that's because in comparison to the absolute flood of traffic brought on by AI, they really haven't been operating on this scale before.
I strongly disagree. GitHub, a year ago, acknowledged the fundamental problems and began work on them. We all agree with the diagnosis and strategy: stop building new things, bring stability. Why would whether the CTO codes have any bearing on the correctness of this strategy? GitHub’s problem isn’t that leadership don’t understand the product, or that they don’t know what they should be doing, it’s that they’re battling unprecedented demand. If it was a disconnect between users and leadership on what matters, sure, a CTO who doesn’t use the product would be notable, but that isn’t the problem. And that’s all assuming he doesn’t actually use the product, maybe his privacy settings hide private commits.
Interesting - There's a couple of things that I disagree with here, but I do think this resonates: "We all agree with the diagnosis and strategy: stop building new things, bring stability."
Where I have a problem with the positioning of their GitHub profile is, he's the CTO of GitHub, arguably the defacto standard for open-source version control systems. His GitHub profile is linked to as the author for the post, and his GitHub profile simply tells me: "this guy doesn't code."
I don't care if this guy doesn't work on GitHub itself, I hardly would expect that, but IMO, any CTO of a company like GitHub should eat, breathe, and sleep code. He might, but his profile, which is being published as if it means anything, tells me he doesn't.
In other words, I think his private contributions should still manifest on the contribution graph. And for being the CTO of an organization like GitHub, with no open-source contributions... Not a great look.
> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.
This is the wrong way to think about this because there's no such thing as infinite capacity. A large distributed system will be simultaneously mostly idle and (in some subcomponents) overloaded. The root cause is not "a component didn't have enough capacity (because of auto scaling failures)", but rather "this complex system collapses (rather than degrade gracefully) when demand exceeds capacity".
When components reach capacity limits, the excess traffic of the lowest priority should be rejected. Rejected traffic should not be retried — in fact, not only should clients not retry these errors, these errors should cause client-side throttling. Traffic isolation should be applied — if the cause of the overload is a single client/customer system, no other system should be affected.
Nearly a decade ago I wrote about some of the techniques we applied at Google to implement these protections: https://sre.google/sre-book/handling-overload/ Most other large internet services have since copied them, afaik.
My last three employers refused to take advantage of Kubernetes PriorityClasses and agree to schedule work to agree (as a cluster-wide resource that affected many teams) on what our PriorityClasses should be and to migrate workloads to have priorities. And this is something relatively easy to implement - no developer work required, and practically no YAML to write.
Why not? Because sadly, fundamentally, most workplaces are not run by people who care about day-2 operations or long-term health. Product or Sales pushes customer-visible work into the pipeline, and you dare not say no. "Day-2" work is not considered to be something that moves the needle. Even now, with GitHub facing these severe outages, it's not like they're facing some massive exodus; their load seems to be getting worse over time, not better.
I'd be very surprised if there weren't any employees at GitHub who had read the SRE book. I'd expect that they're just not listened to.
What TP is talking about has nothing to do with workload preemption and is more of a variation of loadshedding (e.g overload management in Envoy). When I was part of the team that ran Google's clusters we had very few priority classes - basically just one for system and majority of serving workload ran on another priority and the rest was for batch. Pretty sure SRE book recommends just that.
Workload pre-emption is a form of load-shedding - you shed the load of lower-priority workloads (by evicting their Pods) to free up capacity to schedule more Pods of higher-priority workloads that were added by the Horizontal Pod Autoscaler.
> basically just one for system and majority of serving workload ran on another priority and the rest was for batch
RCA blames in-house load-balancing services (HAProxy) that reached capacity limits. Even if autoscaling is not working correctly because it didn't take Istio into account - why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
Yes, just absolutely crazy way of doing load shedding.
> why does it take more than seven hours to just raise the minimum on the Autoscaler for HAProxy and let the workload scheduler evict workloads that are less important than, say, their auth gateway?
Like what workloads? Application backends and databases? Did you ever think that your past three employers maybe had a valid point?
The entire GitHub site was unavailable. The "unicorn" page. Total outage. Visible to every user. Worst-case scenario.
> Application backends?
I don't think I'm taking crazy pills to suggest that it's preferable for services like rendering PR diffs, MR merge trains, even accepting new Git commit pushes, to be temporarily unavailable, so that the entire web application doesn't fall over, and cache-friendly read-only workloads continue to succeed.
First of all, web/api being down is not the worst case scenario - i didn't even notice this till late in the day. Secondly, even 1M qps at the edge is tiny amount of compute, what you have here is not a scaling problem it's a misconfiguration problem. Don't make it worse by adding more load on the system with your preemption churn and maybe taking out more important services causing more severe/longer outage - just fix it in the edge.
are these separate services? Are you going to integrate how services preempt into overall system design?
My take is that you want to move these decisions into the application tier as much as possible as the first line of defense, both because you can make more precise decisions in the application and you can respond much more quickly. You want things like preemption to about a slower moving loop where you are applying much coarser logic to what gets squeezed.
> you want to move these decisions into the application tier as much as possible
I actually agree, but this is a luxury that most large companies cannot politically prioritize (it is not Product/Sales-driven, see earlier comment). Especially when the company is large, and there are dozens if not hundreds of developer teams in a polyglot microservice environment, pushing application-level handling of these concerns is virtually impossible without executive support, and because it doesn't move the bottom line in an easily measurable way, you won't get executive support.
Companies much prefer infrastructure-based solutions to these problems, even if they're coarser, because the relatively small number of people who need to be involved makes it politically feasible. Easy example off the top of my head - mutual TLS encrypting east-west traffic has been implementable at the application layer for decades, but it was a pipe dream until service meshes made it easy to deploy (it's still a pipe dream for many orgs that refuse to schedule any infra work not Product/Sales-driven though).
If git push and pull go down - that truly is a worst case scenario. The website being down is just an annoyance by comparison.
Website being down is not worst-case scenario.
The errors were pushed to the client just like you suggested...
This is an excellent book and its lessons saved my bacon many times!
As it happens I have a hardcopy of this book (along with "Seeking SRE" and the "SRE Workbook") that I am giving away (because of a move). If you want a hardcopy then email me your UK address I will be happy to post them to your for free.
I tried putting them on the street in a little box but surprisingly none of my neighbours grabbed any of my software books. :)
EDIT: The books have been given away
Let's be honest here, the real reason is the crap that is Azure. GitHub was perfectly fine until then. They are just too bureaucratic to admit it.
Worth reading:
https://isolveproblems.substack.com/p/how-microsoft-vaporize...
actually, I think this time it is the other way around, Azure is more the solution than the problem.
Github in the past ran on their own Hardware. That is fine, if your load is predictable and nto changing rapidly. However, the evolution of the past few months/years has shown, that the previous assumptions about growth are now outdated and scaling that capacity on your own metal is not that easy. Hardware has lead times of many weeks, especially in the current situation, datacenter capacity is even longer and more difficult, especially right now.
Choosing not to deal with scaling the hardware is a valid choice in this situation. Yes, Azure is a bunch of servers held together with glue, duct tape and a lot of hope, but I think, the github hardware is not much better at the moment.
Agreed on not hosting things yourself, but, I am not arguing against cloud hosting at all - just Azure. If they just came out and admitted it is a disaster and swallowed their pride and moved their services towards anything else at all - GCP, AWS or whatever else - I think their uptime would significantly improve. Of course, the fundamental problem here is the culture of the company itself. That's harder to fix.
Related: inside perspective on why it's a shitshow
https://news.ycombinator.com/item?id=47616242
A company suffering a spam attack would normally choose to deal with it by banning, and optionally prosecuting, the spammers.
I read the article and found it interesting.
It's wild to me that when I think MS I think this article, Vista, Windows Phone, Zune, Metro, ads in the start menu, the pleated pants meme and so-on, but I have coworkers that are total evangelists for MS technologies and platforms. Clearly a lot of devs get value from the ecosystem.
A lot of their platforms (not Azure) used to just work. Visual Studio and .NET were solid. The Windows NT line was pretty good for servers too (but different from Linux, and not as flexible by virtue of not being a bazaar). It was at Windows 8 era they turned it into shit, first by designing a UI for tablets (fine) and then putting that UI on desktops (wtf) and silently upgrading Windows 7 users without their consent.
I remember running IIS and Apache side-by-side many years ago, and oh boy, was IIS a pain.
Instead of a single config dir where you just deploy the configs and you are done, IIS settings were all over the registry. And don't forget component registration - a single, global registry per machine, used by IIS. Want to run staging and dev server on same machine? Tough luck. Random software installs can upgrade components and cause failures in completely unrelated failures...
The article doesn't mention any Azure outages, but a client had an Azure hosted container restart that day which caused problems due to configs being stored out of persistant storage.
This looks familiar. Microsoft had similar experiences back in the early Word and Windows era. Too many bugs and too late to ship.
So for the past 6 years the bad uptime was because of less than 12% of github being on azure?
Read the article. It's about Microsoft culture messing things up, not Azure themselves (though it may be a factor).
GH didn't collapse so it's not impossible that they already implement these measures:
> At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%.
https://www.githubstatus.com/incidents/zkxwbgr0cnmx
It completely collapsed, 20% is meaningless, as I said in: https://news.ycombinator.com/item?id=49333089
Those are actual endpoints and results.If page has 15 requests and needs data from all of them to work correctly, then with 20% failure rate you are suddenly close to 100% non-functional page from the user perspective.
+1. This is a very real problem in practice.
A technique we've used to deal with this situation: in the overloaded backend (that has to reject some percentage of incoming requests), group the incoming requests by the parent request (the one with the 1:15 fan out) and reject according to the parent request. One way to put it, simply (though somewhat inaccurately), would be: reject 100% of traffic from 20% of users, rather than 20% of traffic across all users (causing essentially full failure for all users).
We typically implemented this by propagating an ID of the parent request down to the backend. I'm simplifying a lot in this description (e.g. have to deal with the parent requests landing on different backend tasks; also rotate the IDs gradually to introduce some fairness).
A nice technique to learn of this situation is to fail ("black hole") random resources in synthetic loads of the frontend. This lets you find out 'bout these single points of failure, and then you can start adding fallbacks.
This is a pretty ancient thing as far as the FE goes: https://www.stevesouders.com/blog/2010/06/01/frontend-spof/
It's easier to create a nice worst-case failure UX this way too. Rather than a page that sorta kinda loads but doesn't actually work at all you can unambiguously put up a "oops we're broken right now try again later" page whenever that top-level endpoint returns a non-retryable error.
But that of course begs the question--why is the frontend retrieving all these data from 16 different endpoints instead of just one? It would be less overhead (http, auth, serialization, compression) to just serve it all in one request even if the http layer needs to fan out 15 rpc calls... this is why we can't have nice things.
Shyeah, all those successful requests were really helpful for showing me a page, which then failed when I tried to do anything by clicking the buttons on the page. I was completely unable to merge things for hours, for no lack of trying. GH was down.
Thank you so much for this gem, this sent me through the rabbit hole and I'm fascinated by the level of engineering in this book. I love how big complex systems are engineered. It feels like anything is possible when you have a solid plan and you keep improving it step by step. So inspiring.
Isn't this exactly the kind of systems design questions they ask during interviews???
Makes me wonder how many folks at GH and MSFT could even pass their own interviews.
Thanks for sharing though. I learn more about systems design from HN comments than anything else
> this complex system collapses (rather than degrade gracefully) when demand exceeds capacity
This.
It's easy to be an "armchair quarterback," here, though.
Handling stuff like this, needs to be planned for, from the start. I suspect that a lot of the issues are because GitHub is something that started small (and probably quickly), and has accreted. Things like Facebook are in a similar boat.
Haha, tell me about it! I spent more than a decade mostly just deploying these systems across just one company. It's far from trivial!
Of course there's infinite capacity. A google datacenter is infinite capacity from the perspective of say an NTP server. Infinites exist when you have enough orders of magnitude in the middle.
Also traffic isolation and degradation by tier is not "no outage", you're still in outage land, you're just being smart in how you use it and choosing what you disrupt. It doesn't fix the lack of capacity.
Google's (or any similar large company's, really) efficiency requirements would never allow for such levels of waste. It would be incredibly irresponsible to allow just any random small server to unexpectedly use the entirety (or even just a non-trivial percentage) of a Google-scale datacenter's capacity. Do you evict other jobs? Do you run your datacenters cold to prepare for this eventuality? Both are terrible options.
Suddenly you push an update to your tiny server or one of its thousands of clients and, because of some bug or misconfiguration, traffic or per-request-resource-consumption grows 1000X or more. Given the size of the looong tail of small servers at Google, this would be a daily occurrence.
I don't understand your comment. A google data center is much larger than an ntp server, but it's obviously not infinitely larger. As you know, if it was infinite capacity, then there would be no need for load balancing or load shedding.
And of course, load shedding low priority traffic is still a partial outage, it's just a less bad outage than load shedding high priority traffic. It does not fix lack of capacity, but it significantly lessens the negative effects of it.
You understand it if you think of engineering infinites rather than mathematical infinites. The capacity of a full single datacenter can be treated as infinite for most customers. I explained how its defined in the original comment. Amount of places in engineering where you treat even a 3 order of magnitude difference as infinite is a lot, but the number of order of magnitudes varies depending on context.
You don't have a single Google datacenter (or any other provider / big name) for a single tenant. The bigger the DC, the more the number of tenants/customers reaches "infinity" (giving the same meaning to "infinity" you are giving to it)
No customer has access to an appreciable fraction of any data center, and if they do they don't have 10x overhead in reserved quota.
Maybe if you're the only customer, but it seems you've not yet gotten a "no capacity" error from AWS, which happens sometimes.
It is infinitely larger, because there is no distinction between more than you use, and infinity.
Sure there is infinite capacity given an infinite amount of time to scale up. I assert you’re leaving out the time dimension. Those so-called infinities are simply not accessible in a practical way since you’ll hit a wall in actually provisioning that capacity long before the data center runs out of compute.
[flagged]
So much more stuff and growing- what it is actually useful for ? Are we getting actually more done than with previous volumes or is it just all wasted energy?
I’m getting a lot more done. Hobby projects that languished for years are coming along great, at quality and depth I could never have found time for before.
Best code is the one that you have not written :) Because the goal is not the code , it is the things that code does, and if it can be done without code, its the best code. Also if you produce lots of code that does not do anything in reality, then its worst code.
And yes, we can rebut that with "time you enjoy wasting is not wasted" except of course some externalities, like boiling earths oceans.
But what about my pet project? The one that I just do for the fun o seeing AI go brrrr, do you mean now I should care about the Earth?
Note: It's sarcasm.
Other projects get unmaintained with maintainers burnt out by a torrent of vulberability reports
This I don't understand.
If it's not your job, then just ignore the reports.
If it's actually critical, someone will put money on the table and then it's a business. And then it's about scheduling and resourcing - also should not burn anyone out.
Just because many people have false sense of entitlement as soon as they get a free offering, it does not mean anyone needs to accommodate them.
> If it's not your job, then just ignore the reports.
If you have a highly conscientious personality, this is easier said than done.
" highly conscientious "
Just doing what others wish is not conscientous in itself! It _may_ be depdending on situation but it can be just pathological towards the self.
When it's psyhocologically hard to do things you imagine will dissapoint someone that's probably not concientousness. It's more like low self-esteem or codependency.
It's very hard for someone to tell these apart themselves. Hence when this topic pops out it's good idea to remind that being super-accomodating may in fact be a personality flaw - that can be healed if acknowledged.
There is very large spectrum between "trying not to dissapoint anyone" and doing what you know is the right thing.
It's also a great opportunity for character development. Use it for that, and not for trying to overbook yourself to 300%.
You don't owe the world anything at all. If you're conscientious, then give a little -- here and there. Don't turn it into an unpaid job.
Not everyone acts rationally even when knowing that they act irrationally.
Sounds like their problem, not something a SaaS product should dance around.
Yet they kind of did. I've limited participation in my libraries with GitHub's setting that nobody who made an account in the last 6 months can do anything in my repos (after some misguided hustler thought they're an easy target and posted an ad).lp
Time's marching forward though. Wonder what will happen after a few more months. We'll have bot spam accounts that are no longer as fresh.
https://xkcd.com/2347/
A great majority of business applications do run on open source projects, and in turn, are affected by them if things go awry. It’s a prisoner’s dilemma in this case.
usual crying from the usual people. AI slop, blahblahb, ...
(I don't mean you. just these so called open source developers.)
I'm not sure if it's going to persuade you but here is an example:
https://github.com/uclouvain/openjpeg
Basically the only library for reading jp2k data (complicated specs, ask your AI to one shot an implementation, mine said "it's 3000 lines of fiddly spec, too complicated"). Issues full of buffer-overflows. Recently unmaintained.
Used in tons of projects, now all possibly vulnerable.
and? don't use it. switch. abandon.
did we lose anything of value? probably not.
It's starting to become a cliché to have people reply "I'm getting a lot more done", but without seeing any evidence of this incredible productivity gains, I'm starting to wonder if y'all are suffering from collective hallucination. If the accepted claims are of "100x productivity" (increasing by the day), and LLMs have gotten very good for the past ~year, for sake of argument, where are the 100 year improvements in the status quo of software?
If one claims such extraordinary figures of 100x increased productivity, a step forward never seen in the history of humanity in such short timespans, they must present extraordinary proof or be branded as a complete lunatic. I could have accepted people saying "I'm 20% more productive", which is an incredible achievement by itself, but not the 10x, 20x, 100x I keep hearing about. I think I've read 200x this week.
The analogy here would be : a kid who wants a toy but does not have money. So he keeps dreaming how he some day would get that toy and how he would play with it and how it would make him happy, but times goes by and he still does not have money to get that toy. Then suddenly along comes LLM and you have infinite money to buy you all the toys that you wanted, you get them, but you now don't have time to play with them. Because time is money and just like before you didn't had time to "buy" the toys, now that you have them you still have no time to play with them.
I've heard people use the same word, I was dubious but they did produce some stuff, yet I think that it ends up as an itch-project. You're satisfied you saw the thing emerge into existence but that's about it. No more drive after that. Maybe because LLM don't require you to have a real long term intense need for that thing.
This is exactly my experience. A month of excitement building something that I wouldn’t have had time to do myself, then something broke in the setup and my motivation didn’t extend to fixing it.
I’m only back at it four months later and I don’t really know what happened before, or why it’s working now, I’m just happy that I can scratch that itch again.
Hey, traditional hand coder fellow ... the times have past and the future is already a present. I never have been this productive before, and it's been ~20 years that I spent coding. System level programming. Few years back I would have said the bottleneck is not in spelling out the code so we wouldn't see that much AI impact but boy I was wrong. Writing code has never been this cheap, both in terms of time resources and $$$. Now I can iterate over the ideas I didn't have the capacity before, both intellectual and time-wise.
> I never have been this productive before
So, what are you doing/ have done with all your productivity?
Enjoying free time that otherwise I wouldn't have. Building stuff that otherwise I wouldn't have time or resources for.
writing code by hand of course /s
I think using AI often feels faster than it is because you put less thought and effort into the problem yourself.
Another possibility is that the people who experience these 100x productivity increases are honest, correct, and simply had abysmal productivity which has now been increased to near-average junior levels thanks to AI.
> I could have accepted people saying "I'm 20% more productive", which is an incredible achievement by itself, but not the 10x, 20x, 100x I keep hearing about. I think I've read 200x this week.
The ratios are factual though. Just look at the "Insights" tab of any LLM written project. https://github.com/oven-sh/bun/pulse
This kind of velocity is impossible to achieve manually.
That’s a measure of lines of code, I suspect the parent is talking about what results those LOC create.
AI built me a 1.5k+ LOC react component which is probably a 15x increase on the file size I would have created, with negative impact on the project for those extra LOC.
velocity - maybe. usefulness? Maybe even easier. As they say data is not information. Commits are not (necessarily) anything useful.
The variance is extreme though. Thanks to Claude Code and Codex I've been able to make several non-trivial internal tools and libraries without writing much in terms of code, just some reviews here and there.
I spent a couple of days on those, and its would have taken me months to write manually I am sure, so in that regards it's close to 50x.
I've also had Claude track down some logic issue in a module I was unfamiliar with which had very large and complicated flows. Would have taken me many days, since I did not have a reproducible case, so had to go by logs and customer description alone. I spent 5 minutes writing a prompt and when I checked back, Claude had identified the issue. The fix I had to implement myself, but was fairly easy. So there Claude definitely was a 100x increase in productivity.
Then there are cases where they're much more modest, or where they might even be negative, when they think they're fixing stuff but actually are introducing more bugs.
You can accept what you like, but it's true. Our team and our business is incredibly more productive. The number of new valuable customer facing features, and the number of PRs (which represent REAL work, not taking a PR and splitting it into 200 PRs game) have all increased dramatically. We've shipped something like ~10x more PRs so far this year than all last year. And we've done that without increasing the number of bugs and outages.
Great, number go up. Have the features led to actual customer growth, or just increased productivity?
Yes lots and lots of growth.
> where are the 100 year improvements in the status quo of software?
Most software work is just churn / doing the same thing over and over. More productivity can just mean more output, not better output.
Same. Helping family out when I generally was too tired to do it earlier. They're all very happy too
I’m curious, what was the application of AI that helped the family?
Most of my family are small business owners, so I've done a handful of Excel automations that they needed that generally take me a while to figure out. Landing pages and a few 3D designed prototypes that I used the Fusion 360 MCP server for
Are they happy about the circumstances of AI too?
Higher resource consumption and the set back in CO2 reduction?
[flagged]
[dead]
you seem to be uninformed. This should help: https://www.youtube.com/watch?v=H_c6MWk7PQc
A link to a YouTube video without context e.g. summary of findings, primary author/creators, and primary citations, methodology, and so on is a next to useless for making a point. For all we know you’re linking to a crackpot or an industry sock puppet and I don’t care to “watch” any of what can potentially be a dubious video or worse a malicious video. It is your job as the linker to convince me the video is worth even one iota of my time.
On the other hand recent papers highlight the validity of concern/suspicion:
> This systematic review demonstrates that the environmental footprint of artificial intelligence is a structural and increasingly consequential challenge, shaped by interdependent decisions across algorithms, software pipelines, hardware infrastructures, and deployment contexts. The synthesized evidence shows that energy consumption and carbon emissions associated with AI systems are highly variable, context-dependent, and often underestimated — Beyond Efficiency: A Systematic Review of Energy Consumption and Carbon Footprint Across the AI Lifecycle (published in “Sustainability” an international, peer-reviewed, open-access journal) https://www.mdpi.com/2071-1050/18/3/1359
"I know nothing about the subject, but I would _guess_ that the power consumption is bigger concern than the reported and studied water usage."
That's the whole 21 minute 59 second video in a nutshell.
A loud and hectic quick cut rambling video essay. Millions of views, naturally.
> raised heat is a kind of pollution
> water is not unlimited
> you don't want to release a bunch of acid into a river
> I know next to nothing about this
> I know basically nothing
> corn is one of the thirstiest major crops grown in the US
What is this rant supposed to inform? Whats wrong with OP being concerned about the costs of operating a DC?
Nobody is concerned about the costs and environmental impact of data centers.
When someone in a discussion about the benefits of AI goes "did you think about the environment?!", it's always performative.
The real motivation is disliking AI itself or doubt about the government's ability to offset the labor market impact. Discussions that start with feigned concerns being raised are nearly always going to be unproductive.
Wrong.
I don’t dislike AI but really of mine are negatively affected by climate change and AI isn’t helping what is easily observed when Google and MS scrapped their CO2 reduction targets.
So every time I use AI I think about the necessity and usefulness of what I‘m doing with AI and if the use outweighs the costs.
Since the rise of AI the environmental impact doesn’t seem to matter anymore.
I guess because it’s the shiny new toy of the hackernews audience.
Privacy also lost importance given the fact that the same people who refused to give information like their phone number to companies like Google and Meta now upload their whole life to their AIs to asks what should the eat, hyperbolically speaking
You're not helping your case by questioning the usefulness of software produced with AI in the same comment section, or complaining about other people supposedly compromising their own privacy in overusing AI.
The reason it doesn't matter is because the environmental impact is moderate, and the benefit obviously tremendous.
The environmental impact is anything but moderate and the benefits are not obviously tremendous at all. I'm happy using Claude Code as much as the next guy, but saying that the impact has been "tremendous" is vastly overstating the actual results.
I disagree, with the projected doubling by 2030 we're looking at 3% of global electricity consumption or 3.4 EJ, less than 1% of final energy consumption.
That is moderate. Energy-intensive industry is around 130 EJ, and global final energy consumption > 450 EJ.
Existing documented applications of today's AI have the potential to decrease energy consumption by >13 EJ/year by 2035.
Now that was about operational energy consumption. Someone might bring up manufacturing and construction.
From what I could find the climate impact of those are estimated somewhere between 10-35% of the total climate impact of data centers, so relatively small compared to the operational energy consumption.
It is very hard to justify more than moderate environmental impact here, in my opinion.
For the benefits of AI, my personal results have been great, so I am quite optimistic. And objectively, I find it hard to ignore recent results in mathematics and security research.
"Global data centers consumed around 415 terawatt-hours (TWh) of electricity—about 1.5% of the world's total electricity—with AI acting as a primary accelerator for new power demand."
https://www.iea.org/reports/energy-and-ai/energy-demand-from...
That is today where we already consume too much. If by 2030 AI's consumption doubles it gets worse. While training large models draws major initial power, everyday AI usage (inference) now drives roughly 80% to 90% of cumulative AI energy
"Existing documented applications of today's AI have the potential to decrease energy consumption by >13 EJ/year by 2035."
Seems like AI helps slowing down the rise of energy consumption.
We are at a point where we want less CO2 not moderataly more. In the end more is more.
If your doctor tells you to lose weight or you get sick it's not a success to gain weigth slower
Well, at least we could establish that the environmental impact is moderate rather than extreme.
And if the potential of >13 EJ/year is actually realized, it would seem like the net impact of the data centers is not just "moderately more CO2" but possibly "moderately less".
Please avoid low quality analogies on HN.
Please avoid acting as the judge for analogy quality.
Anything but a reduction is bad and AI is a setback for that.
Potential benefits are as long useless as they aren’t realized.
BTW the energy consumption reduction is achieved by what kind of AI? LLMs?
I'm pretty sure that most of people private data contains data about third parties too. It's more as their own privacy they compromise
GP wasnt mentioning water consumption.
What make you think I‘m talking about water consumption?
The construction of data centers needs resources and also creates more the CO2.
The energy for these data centers is often created through fossil fuels which also creates additional CO2
What do you think why Google and MS scrapped their CO2 reduction targets
Same!
Has anyone noticed an increase in the quality, performance or capabilities of the software they use?
I play modded Starfield. There's been a clear increase in mods lately. Some of them are from self-proclaimed non-programmers who are using the LLM's to reverse engineer the game or other abandoned mods, and they've started to create new cool mods or they've fixed various engine limitations. Can confirm that these actually work and I can finally have my 1000+ modlist.
There was a huge exodus of existing programmers/modders ~two years ago, due to paid mods and what not. The gamers took over with their LLM tools.
I have. In fact, some of my hand rolled stuff is actually _provably_ faster and more secure then even heavily battletested and widely used "industry standard" solutions. This is mainly due to them passing an exhaustive barrage of millions of lines of code of tests (literally, in fact I just did a pass over all my tests/ dirs and it's sitting at 6.3 million as of today) ranging from adversarial CVE probing attacks to fuzz tests. Ironically, my same testing suite has caught _numerous_ bugs in production stacks (openSSL/libuv) particularly, literal hard SIGSEGVs and the like.
On CVE probing, and I haven't really seen anyone describe/use it (or I may be oblivious), but the way you do it is you curate a list of CVEs for the class of software you're writing, say a web server. Then you take this list in chunks and hand them off to your agents to devise and implement adversarial technically analogous attacks against your codebase. If it's red, report and patch. Ironically (even with Fable 5) it's never complained/refused to do it.
Well, many of the saas I use continue to have problem meeting their SLA. And I’m on support just as often as always trying to get through to a human to file a bug that will never get fixed. “It’s in our roadmap”.. no it isn’t. Even with LLMs, product will focus on making new features to push their AI mission.
My unsupervised agentic setup has built something that I wanted for decades but never built, so I guess that’s an increase in capabilities.
At my company, we are using AI to clean up huge amounts of technical debt that would have just hung around otherwise, so yes. I'm not sure that your average user would notice, but we do without a doubt have a much better product now.
Only in software that is used to make other software. So we're all just patting ourselves on the back within our bubble.
What I have noticed an increase though is in demands and pressure to deliver.
I'm writing much more secure software and infrastructure (e.g. IAM now gets configured well).
I have. Mostly for the software I use that I maintain myself though :P
That’s like asking whether fuel consumption was productive or leisure as the number of cars on the road increased. It’s both! I don’t think you can separate one from the other in any reasonable way.
You already know the answer to that question
More done + more leisure.
That's how it's sold, but have you heard about any major tech company that sends their worker home when they've achieved what they used to achieve pre-AI?
EDIT: Instead of simply down-voting, you're welcome to name examples that proves me wrong ;)
This is anecdotal, but I know that a lot of my coworkers and coasting and putting up one AI generated PR per day which they've hardly even self reviewed.
In a good company that will come back to bite them next performance review.
So, if it doesn't then you learned something about your workplace (and it's not good).
> In a good company that will come back to bite them next performance review.
In a good company that should be discussed in the next 1:1s so actual change can happen meanwhile. If it just waits for the end of year review, then it's not a good company.
Sure. I meant if they are stubborn.
Middle management was all cut so all managers have 30 direct reports and "Don't have time" to do more than a couple 1:1s per year.
These individuals are demonstrating that they can be replaced with AI. I worry for them.
The only way that you can translate productivity gains into more free time in a salaried position is by working remotely. Seems obvious, no?
Really? https://newsroom.haas.berkeley.edu/ai-promised-to-free-up-wo...
Is it used afterwards or done and forgotten?
To some extent we are getting more things done as well. In my company (mid-sized startup), they're making us push features every other day now as opposed to maybe 1-2 features per person per sprint. Back when I joined, things were a lot slower. Today, they expect freshers to push new features on day one.
But ofc, slop has increased a lot more as well.
That's very fast. Feels like you cannot really review this code so it's all just "working" AI code with few railguards and few human supervision?
Jumping on the great asymmetry of output between humans and llms, yet still being responsible for the product is pure kafka.
And how many of that features are actually used?
How long do they last until the get replaced by the next feature?
I have some big issues with this technology and the companies behind it, but I know of quite a few people personally who were not previously coders but have now been able to use LLMs to make their own custom software, solving real problems they had.
Do you believe this is a significant part of the increased GutHub traffic, people’s personal software?
Are non-coders these days aware that they should use version control (and push their code to GitHub)? Or does their agent helpfully suggest setting up a GitHub repo?
One of the things I've done is create an entire fully functional GitHub alternative that does more of what I want and hosts all of my other projects, so yes I at least am getting considerably more done.
You could have just installed Forgejo.
[dead]
[flagged]
Lol you don't need an app for this. My (barista) gf did the same thing by just telling Claude to create pages in notion.
And you don't need excel. Pen and paper gets the job done. Good logic.
For some context, in June they said commits "commits nearly doubled year over year, crossing 1.4 billion per month". Now, it has more than doubled that in just a few months.
https://github.blog/news-insights/product-news/github-copilo...
Makes sense given the ubiquity of agentic coding. I made a joke to my coworker today that all we do is make sure AI agents can communicate with other AI agents.
It’s funny how the quality of the average commit message has gone up, now that we rarely read or write them directly anymore.
You may be confusing articulate and confident with useful and accurate.
They really aren't. AI commit messages are much more thorough and correct than some of the stuff I've seen humans do over the years e.g. Fix, Fixup, Fixed some Stuff, Should work now, Definitely should work now etc etc etc.
I definitely prefer a "fixup" to a three paragraph meaningless word soup which doesn't cover why the commit is there either
If you can’t tell how this is a non sequitur then it’s very likely you, too, confuse articulate and confident with useful and accurate.
Nah, most often they are just long-winded beatings around the bush.
Not really. In my experience, the average human written commit messages were almost always useless 1-3 word "Should work non" kind of messages. The agents had a low bar to clear.
Has it? Most of the LLM-generated commit messages and PR descriptions I read are needlessly verbose and miss the point: elucidating the why of a change rather than the what. If I actually pose a question based on them the author often tells me “yea that was some AI generated nonsense, I rewrote it now”.
LOL, this kind of things will happen when projects like Bun (https://github.com/oven-sh/bun) are running on auto. :)
More than 50% of the recent commits are from robobun, jesus
It's AI all the way down.
An issue reported by a person account but post made by AI. https://github.com/oven-sh/bun/issues/39800
AI (robobun) responds and creates PR. https://github.com/oven-sh/bun/pull/37459
AI (coderabbit, claude, github actions) review the PR, AI (robobun) applies the fixes.
Some AI back and forth.
A human finally merges the PR.
Not gonna lie, it's kind of beautiful.
Some guy, profiling his linux distro, wonders why ssh connections are a few hundred milliseconds slower than expected, finds there PR introduced and RCE backdoor and p0wns the whole project.
Seriously, this is driving me insane. The XZ hack told us what we had to do to secure our supply chain, and instead we went ahead and implemented a worldwide standard for NLP-to-action and figured we'd worry about the guardrails later. Mad.
ref: https://en.wikipedia.org/wiki/XZ_Utils_backdoor
Here's the most recent nontrivial one as of this comment: https://github.com/oven-sh/bun/commit/d4de65e9a43224a14591ad...
The code change makes no sense and should do nothing. The commit message described a very deep investigation into garbage collection on the C++ side. Some object is being kept alive when the test requires it to be collected, and changing the code in this way allegedly prevents that. But wouldn't you think there would be a better way to ensure an object gets collected, like setting the variable to null?
The comments in the code don't make a lot of sense either. Something so obscure and brittle has to be explained extremely clearly.
While the issue might be real, this commit is so far away from the locus of normal that it's sending red alert. Plus a hallucination is very likely with such a long investigation - once an LLM agent starts investigating it just assumes there is a problem. And this is the 1 out of 1 robobun commit that I looked at.
Edit: here's the next one: https://github.com/oven-sh/bun/commit/72ec6e2594892455df0090...
Make sure the fs module keeps working if someone freezes or seals its exports table. I was wondering who was going around freezing random tables from other modules, so I checked the linked issue - robobun reported the issue, too. Why? I'm skeptical of whatever robobun was doing when it decided that it was necessary for code outside of a module to freeze their export tables. It needs a very good justification.
The first is, annoyingly, a relatively common problem and solution when dealing with GC lifetimes in tests. Few interpreters/JITs want to generate extra instructions to null out stack slots or pre clobber registers to ensure something becomes collectible at a specific point. Eager nulling of a variable often gets removed by dead store elimination or even just from being a disconnected SSA node. I've written extra nested scopes or wrappers in Java to deal with this in tests.
Don't know anything about the second one.
If this is needed in tests it needs to be known how it works, it needs to work consistently, and it needs to be documented how it works. It can't be an ad-hoc deep investigation and random fix each time. The comment should then be just // ensure foo is no longer a GC root, see gc_roots.md
It needs a comment for sure. I always write a little helper utility to manage that when I need it.
As it should be if not more. What's wrong with it?
This will result in pretty low quality software.
I've had opus 5 along with its AI code reviewer agree to do some pretty stupid shit.
Opus 5 is only usable when Sol is reviewing its output.
[dead]
Opus 5 is a trash model
Wait till you see what some people agree to.
Ah the good old "AI isn't bad because people are bad too" argument
[flagged]
I think you're projecting
Wow...
I am currently using bun, but may have to switch. I can't see how this can possibly turn out well in the long run...
3.3k open issues holy shit
and all of high quality
https://github.com/oven-sh/bun/issues/39797
And 5K PRs. I'm crying.
> 5k PRs
Wow, is Bun the record holder for number of PRs?
I recall GitHub recommends to keep the number of PR to a certain level due things such as GitHub Actions slowing down.
Nixpkgs has 11k open PRs at the moment (a lot of them routine version bumps and such), so no. But then Nixpkgs isn’t a piece of software in the conventional sense: the monorepo does contain a few of those, but most of it is a giant collection of basically-independent build scripts.
that's the interesting part : forges are being abused.
Sure AI workflows might be a non-negligible share of all that usage but still the point is that initially forges existed to help developers collectively share a state then solve problems. Nowadays they are basically online filesystems with better notifications for other software to interact with and only optionally developers actually communicating.
You never could scale tulip buds so quickly - progress! ;-)
Strange to think they are probably triaged by LLMs at this point
AI finding issues in code and reporting them so that an AI can review and triage them for another AI to fix.
That seems to be what's happening. Like 2-4 bots talking to each other, then Jarred Sumner just does the final merge with little comment.
https://github.com/oven-sh/bun/pull/39743 https://github.com/oven-sh/bun/pull/39735
github-actions: "If you need a paragraph-long comment to justify why the workaround is OK, the code is wrong — fix the code"
robobun: "The ordering is load-bearing: reclaiming before this block made is_dead_request true and hung a parked textStream read (caught by body.test.ts in CI). The comment pins that constraint."
Ah, well, if something is load-bearing, then I guess that settles it. Need a comment to pin that constraint, in case a read is parked. These are words that normal humans commonly use in these ways.
(Always striking how much Claude obsesses over the minutiae of method contracts and side effects, exhaustively documenting them in comments. It’s much happier figuring out how to reorder some method calls with nonobvious side effects so the code works than it is refactoring them not to do unexpected things!)
I mean, isn't that the dream?
I don't know if that's sarcasm or not. I know it doesn't work, but that's the future we've been promised, right?
If the same AIs don't hallucinate and create bugs in the first place.
I don't know how much my own time is wasted on Claude imagining API response formats that never existed.
It's certainly someone's dream: https://devhumor.com/media/dilbert-s-team-writes-a-minivan
> I mean, isn't that the dream?
Unironically: no.
Why it doesn't work? It is working for me. It is working for bun. It is working for others who actually embraces it and puts in the work to get it working.
I recognize your username from other comment threads and would classify you as a bun fanatic, but even so, I’d hold off on saying, “it’s working for bun,” until 1.4.0 has been out of canary for, like, more than 24 hours. Most real users haven’t onboarded to it yet.
so first it was rust rewrite bad. Then rewrite with ai can never work. Then it will be riddled with bugs. It'll crash. It'll take years to fix.
On the other hand, rewrite was mostly done in record time. New version added massive number of features. Also huge bug fixes. Being used by Claude code by millions of people. Successfully used by some others even in canary. After release, multiple companies immediately switched due to massive amounts of resource savings and performance gains (and publicly posted about it).
Can there still be problems? Yes, I'm sure there will be. But denying the feat Oven pulled off with Bun in last few months is nothing but phobia/fud.
Many people are already posted about testing new bun version and I have yet to see a single post where the issue is the latest versions of bun. In some cases people posted it doesn't work but that's due to node compatibility etc and it didn't work on previous version either.
One does not need to be bun fanatic to see and call things as they are.
PS: I like bun because I hate how js ecosystem requires 100s of packages to do anything and bun is aiming to include batteries. This is good.
sadly someone didn't have free unlimited Fable and Opus like Bun team does
Let's just say my dream does not involve babysitting some robot
Who promised it?
Not my dream…
And...? It just shows how active and big the project is.
It really is. I know I've gone from tens a month to thousands a month. They have to be projecting >100B/month in the next year or two.
wow. they should really institute a maximum amount of individual pushes per-month per-user.
That would just drive users into the arms of the alternatives, which would love to own the world's code... like Cursor/Musk.
Microsoft and GitHub's only option is to suck it up, absorb this growth, and lower failure rates. They have the money, so that's not the issue.
As someone on the sidelines, this is really interesting to watch unfold.
Well either they can handle this load that Microsoft can't, or they can't. If Microsoft are going to continue to be unreliable in the absence of the rate limit then:
If alternatives can handle the load, those who would consider those alternatives if Microsoft opposed a rate limit are likely to move to them anyway.
If alternatives aren't able to manage, then user's aren't going to jump since those services won't actually provide more usage.
I think the number of commits is a red herring, but that said, I wonder how spiky their load is.
Imagine a fee over X commits, but only during certain hours. I can imagine 90% of the commits over 6 or 8 timezones, maybe 50% over 4 right now...
I was thinking the same thing at first: ideas to increase product limits on GitHub, to increase reliability given limited infra.
However, there are sharks in the water, and with the diminishing mean of user technical knowledge, the product actually needs to become even more free. GitHub likely needs even lower friction.
"All it takes" is the insanely heavy technical lift to support that. There is no other solution. All the C-Suite needs to do is foster an environment with well-thought through, and possibly over-funded engineering, at the edge of the art. That sounds like an amazing challenge.
It's not a red herring, every commit necessitates a database hit.
Why would any company want coding data now? It's all garbage. I'd be surprised if anything past 2025 is even used for training.
Me personally, I would like to own the world's code.
The richest man in the world is interested in this dataset as well.
There's nothing special about code here - if you can own the world's $X for any X, you win capitalism.
If you use GitHub as just a software forge then sure you can find an alternative. But I suspect more people use GitHub for its social aspects and they will stick around despite the regressions because building an alternative to an established social network is incredibly difficult if not impossible.
You can get work done on any software forge. But potential employers will still ask for your GitHub. People will judge your personal project by its GitHub stars and be less reluctant to download a binary from GitHub than elsewhere. Potential contributors will leave a PR on GitHub but probably not if they have to make an account on a new platform and learn how it works.
And of course, let's not assume any competitor can just absorb even a fraction of the traffic GitHub receives without suffering similar reliability issues.
I don't think anyone would love to own the world's AI slop. The capex and revenue expenditure would be too high for any ROI.
I still remember when they decided to limit the number of private repos you could have as a free user. Kind of silly to me at the time, and even more so now!
I remember when you had to pay for private repos in the first place.
Initially there were no private repos for free users at all, it was a selling point for bitbucket that they offered that.
I don't think it's silly for free users to be restricted. If you want something, pay for it. Free trials are trials.
so... when you reach your monthly limit in the middle of the month, what is the recourse?
Or, is the idea just to drive everyone away from your platform?
Uh, pay money for the service you're using constantly?
Which will just increase the cries of "enshittification" and hasten the mass migration to the next free platform that surely, this time, won't ever go down.
We're small enough that we've been hosting our git infra for about a year now, I wonder how many other companies figured out they could make the trade. I've had a Github since a couple years after they started and I think they are going to become a Stack Overflow, albeit slower with MS at the helm. If Github is going to be 99% slop it's going to be really hard to use as a fun tool to show what you can do, what you've worked on, side projects, etc. I took github off my resume and I'm probably not going to relaunch my weblog if I end up job hunting, too much low-effort crap and people basically copying what a lot of us had been doing manually for years to really feel like it's anything other than a negative signal.
Ironically a bunch of people already migrated to Codeberg and then Codeberg announced "heads up, we actually don't want your AI slop, we're for real projects only" and AI coders threw a shitfit on HN.
I would give the growth hacker on their team a big raise, these KPIs are incredible
Why is Github talking about number of commits here, and not pushes? Are there a lot of tools/people using github as an online editing platform?
GH processes at the commit level for things (including actions) even though they're bundled in a push... it's relevant to the load on their infrastructure.
In what way? If I have multiple local commits pushed once I expect to see CI type actions to run once for the push rather than for ever commit in the push.
This is configurable in Github Actions. Many projects want to run CI for each commit to avoid situations where one commit breaks the build and the commit after in the same push or PR fixes it. Broken commits in the history makes bisecting harder.
Interesting, never realized that was an option. Thanks!
But they can't process the commit until I push it?
A push containing 100 commits is more expensive than a push containing 1 commit.
But is it 100x more expensive?
Probably? If you do 100 pushes instead, there is roughly zero additional data. At best you'd be comparing cache costs, which probably are lper for one large push, but there's a ton of calculation and CI that runs per commit regardless of other data being cached.
Why don’t you have your actions run on pr or push instead of on commit? Why would you even want that? I’ve never seen actions set up that way. If I push a branch with 100 commits then it’ll only run CI once. It’ll show the rest of the commits in the UI, sure, but that doesn’t mean that it’s the same performance impact. It could very well be 1 db transaction with multiple rows written instead of 100. I think you’re reducing this problem too much without knowing their architecture.
Github runs CI per PR push (obviously), and per main-branch commit (click on commit history in any project with CI, see a build result check mark on each one - that's true if you push a dozen commits too (I've done that)), in nearly all setups I've seen. I'm not sure how much of that is required vs default though.
With enough effort, you can rather obviously run CI per PR commit (it's a programmable system), but I've never seen aUI-integrated way to track the results, aside from browsing custom job names, which is very far from what I'd call "integrated" when compared to PR-level build markers. Similarly, I'm not aware of (but would not be surprised by) any way to disable per-main-branch commit builds, aside from initial pushes.
But I haven't poked around deeply in the settings, and business-account settings are rather different anyway so those might be wildly different / more flexible / more obtuse in exciting ways. Github is a very large and complicated product at this point, darn near anything could exist if you dive through enough UI layers or use old URLs to find soft-deprecated features.
Also, honestly, 100 commits = 1 transaction? That's far more of an over-simplification than anything I've said. It's a massive product with thousands of engineers, there's no chance at all it's just one database.
Things like password scanning have to look per object uploaded
Bigger numbers sound more impressive.
"Our billion-dollar infrastructure crumbles under a tremendous flood of 50 PRs per second" would just sound embarrassing.
I'm not following your line of questioning. Without ever using github as an online editing platform, you can do one push with two new commits.
Commits are not expensive, pushes are. You can do any number of commits before you do one push, unless you are editing online, in which case every act is it's own commit & push.
You can rig up a local ide to pathologically commit+push per save, but you can do literally anything, so what you can do is immaterial.
Do you have some GitHub architectural knowledge you’d like to share with us?
A push pushes commits and blobs and trees and tags. It’s an interesting metric to track, but the core unit of complexity (and expense) worth tracking on GitHub’s side is obviously the commit.
There’s a difference between pushing 1 commit and 100.
> There’s a difference between pushing 1 commit and 100.
There isn’t much. GitHub doesn’t run actions separately for each commit. It runs them on pushes. I’m trying to think of a thing that would happen for each commit in each push and coming up blank.
It does things like scan for references to issues to index, but it would just scan the log for a range.
I did disagree with GP though because there is no reason to assume that the ratio of commits to pushes has materially changed. So if that is the proxy they have always used for measuring growth, and they know it reliably does that then I think it’s a reasonable way to communicate this to this audience.
> It runs them on pushes
Sure, because pushes are how you update a reference. That’s really what triggers an action: a reference changing. And there could be a bunch of those in a push.
A commit costs storage, you’ve got secret scanning, it needs to be indexed in a way that can be referenced in commit messages and comments, a commit message itself can close issues or reference other PRs, stored and served individually and immediately via the web UI or git clients, etc etc.
It’s also like… the core unit of git.
None of the things you mention - indexing or secret scan would be done individually for each commit. As I already said, this would be a log of all commits in the range pushed - it would be scanned once for those things. There is no need for a loop running over a range of commits and processing each one.
There 100% is at least for things like secret scanning and message parsing.
Secret scanning needs to make sure my repo as a whole has no secrets. It’s not acceptable to have 1 commit introducing it and 1 removing it because the secret is still recoverable.
Every commit is also surely an entry in a database somewhere. I can navigate in GitHub directly to any individual commit so there is definitely some overhead of some type.
It's not necessary to perform secret scanning on a per-commit basis. The most efficient way is to just scan all blob objects being pushed; there is no reason to even be aware of an object's location (tree path or commit) except for diagnostic messages.
> I can navigate in GitHub directly to any individual commit
You can do the same with the git command line client. The overhead you claim is already in the git on-disk format. Github might very well duplicate this information in a database somewhere, but it doesn't follow from your observation.
You are right about secret scanning, but its worth noting that is only enabled by default on public repositories. It is an extra paid feature for organizations on a teams or enterprise subscription, and isn't available at all for individual owned repositories outside of an enterprise subscription.
And yes, I agree there is indexing of commits, but that is a batch insert from a log.
>You can rig up a local ide to pathologically commit+push per save
The dev system we use for a 3rd party hosting provider (a big one) requires a commit and push for every file save while we're developing. I created a build system for this that copies the whole repo to a temp folder. As we save changes to files in the main repo folder, the build system watches for changes and copies the changed file to the temp folder, then does a commit on the temp folder and pushes to a an intermediary repo in github which then triggers an action that causes the 3rd party system to update from the intermediary repo. This way we don't pollute our main source repo with a commit every time we save an update to a source file.
It's not my favorite way to develop but it's caused us no real problems except when github goes down.
> requires a commit and push for every file save
I don’t think I could imagine a stupider idea than this if I tried. To paraphrase Babbage: I am not able rightly to apprehend the kind of confusion of ideas that could provoke such a solution.
meh, it's just external undo button. It's useful. It might or might not be worth the cost, but it's not that it delivers no value or causes some harm (other than cost/reliability)
https://delta.dev/
Exactly. You can have 3000 commits in a branch, and unless you don't push each of them one by one, shouldn't be a problem the number.
Truly incredible in every sense of the word.
It's almost double. If you have scaling prepared, it should be linear, but I doubt they've prepared for this.
they are/were in the middle of a migration
Oh the productivity increase.
I feel it in my fingers. I feel it in my veins.
It's like cancer growth though, not the 'good' kind of growth ;) E.g. I doubt that the number of Github users has doubled in that month too. Github should probably introduce daily commit- and merge-limits that slow down excessive clanker activity, but are high enough that a human doesn't notice. Alternative put users with excessive resource usage on their own 'sub-infrastructure' so that when this is overloaded, the regular users are not affected by the outage.
Not really. It's AI commits. Not quality commits.
it doesn't matter though, commits create load in their system
One has to wonder what the point of a collaborative version control system even is if the software we’re writing is one-shotted by a call to an API ?
the infrastructure does not care about the quality of the commits, just that a commit happened.
A lot of human commits are low quality
Because human commits ar necessarily quality?
no but AI commits are necessarily not
[flagged]
Serious question: do you think your 6 year old is learning coding from prompting an AI?
I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong. This is proven on a nearly daily basis at my job, where, with identical agents and prompts, I see designs being pushed with objectively incorrect facts, sub-optimal code in PRs, and a general explosion of slop. That tells me that it still very much matters if you know how to do the job without the help of robots.
> I have had excellent results from using AI, but it’s only because I understand what it is I’m asking it to look at, and know when it’s wrong.
That knowledge will be worthless 12-18 months from now when AI does everything better than you, including “understanding”.
If you’re one of the world’s best programmers, it might be 24 months instead, but the writing is on the wall for everyone.
I wonder if people were behaving like that for other revolutionary technologies in the past. “I can still run faster than a car can drive in sharp turns on a gravel road…”
It will be interesting to see if LLM can push beyond the intelligence embedded in language.
General intelligence may still be a ways off.
General intelligence is a convenient myth perpetuated by the marketing departments of AI corpos to help further their pursuit of regulatory capture. Don't fall for it.
> That knowledge will be worthless 12-18 months from now when AI does everything better than you, including “understanding”.
...what? Seriously though. What grand insight into the arc of AI development do you have? Where are you getting this from? Cite your sources. Show your analysis.
Charitably, I would ask "are you high?".. If you're suffering acute psychosis from mind altering chemicals then we merely have to wait for the effects to wear off and you'll likely recover. If not, then I'm very sorry. Your road ahead is a rough one.
Coding is dead.
[dead]
[dead]
I mean if I set up a for loop spamming my own SAAS it would be incredible growth too
this is impressive too
"We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity"
Not that impressive when you realize it's mostly due to AI slop
edit: AI actually writes 99.9% of my code these days. I'm just saying of course the number of commits to github is going to climb astronomically due to AI.
I don't know how to delete things.
And what would be the expected result (other than spending $20 a month)?
I wrote a list of things that makes us money, but I decided that I like having less competition. Now to see if I can delete my original post.
Automated slop?
Agree - this reminds me very much of the old joke of two economists increasing GDP by taking turns giving the same 200usd back and forth for having each other eat shit.
Useful / impressive for whom is the question. Not for us!
We pay for Github enterprise, and because GH can't be bothered to separate service tiers for sloplords and actual paying customers we get garbage level performance. They could of course always implement usage limits, but the goal is not to earn money, or provide a good service, the goal is to maximize AI users. Would be very awkward at the next executive golf meetup if you couldn't point to increased AI adoption.
In short: This is why monopoly laws matter. Once a company becomes too large, normal business rationales cease to be the motivation for their actions, and GH can go along with the pied piper of AI psychotic C-suite officers like MS is doing instead.
They do. Talk to your account representative for the alternate domain name to hit.
Should use on premises.
Should use Forgejo
Impressive for whom? It's impressive for the service to have such growth at that scale, the code being slop is somewhat irrelevant. Your comment just seems like mood affiliation (AI should be dismissed, growth was from AI, therefore growth should be dismissed).
Is it impressive? All it's doing is decaying the services. 15 years ago never have imagined I would go to the lengths to host a github alternative on a VPS but after doing just this (also being the last one in the my professional group to do so), GitHub is giving a master class in destroying their reputation in pursuit of advocating for hostile entities.
Not all growth is good, especially growth that is actively hurting the company.
There is an equilibrium in both nature and software. Purposely designing systems that mimic the effects of cancer is going to benefit who exactly?
This growth is surely good for github. If another company becomes the "github for AI agents", they'll lose not just their business for AI but also human coders. (Sure, maybe there will be a human coder only github, but it will be quite small.)
LLM companies are already wildly unprofitable. Who ever wants to do this can be my guest, but please let's tax them enough so that we can get something positive out of the stupidity like free public school lunches for children or universal childcare.
How is it impressive if we all know it's autogenerated? There's no more people there than there were before. Heck, at ~2x growth that's possibly a decrease in real humans there since bots generate loads of them per person
> there's no more people there than there were before
Seems false. Lots of coding adjacent people, engineering managers, etc. are now pushing PRs.
> How is it impressive if we all know it's autogenerated?
Nobody is saying the code is impressive, just the growth of github is impressive. It's not doing anythign different based on the source of the code.
It's an impressively large quantity of X, not an impressively good X.
That's not how I read the statement "It's impressive for the service to have such growth at that scale" that this was a reply to. They do seem to think the growth is impressive, not the absolute quantity, about which I agree with you
Yeah, seems like AI slop is going to kill GitHub's free tier. I just don't see how the economics of having to host this much slop and provide service to slopcoders is going to convert into dollars for them otherwise. None of the humans involved are going to end up in big enterprises. It's all cost, with no pathway to revenue.
alot of distributed systems in big companies grow at this rate, its not exponential, its mundane
When a service that was already the primary git hosting provider for most of the world for 20 years grows at that rate its not mundane and its not comparable to any example.
AWS saw this growth every year for two decades, hyper growth tech sees it all the time
The absolute number doesn’t matter nearly as much as the change in rate of growth. The number of commits had not been doubling every six months at GitHub for a long time.
At Amazon if traffic volume consistently doubled every six months that is actually quite a lot easier to plan for, it just becomes part of everything they do from very early on.
im not buying that github is a unique engineering problem, harder than the rest of hyperscalars.
No one said it is unique. But if you take an infrastructure and engineering org that had been growing at 10% a year for a decade, you are going to have a different set of capabilities and practices in place. Adapting to a new reality of doubling every few months will predictably produce failures anywhere. GitHub is not unique in that regard.
They have what amounts to an unlimited budget for AI spend. If it’s so fantastic, why can’t they let it crawl over every piece of their codebase, every metric, every log, and spot these problems before they occur?
“We misconfigured a sidecar” is something I would think AI could quite easily find and fix.
I don't know who you are responding to. I haven't made any statements about what AI can do for them. I would expect AI is making the situation worse, like it is in many dysfunctional tech organizations.
ive spent 15 years in big tech companies, this problem is common and its why they pay engineers 500k-1M. this happens at meta literally all the time.
half of engineering in big tech is just rewriting a system to scale
microsoft is incompetent, they havent changed windows/excel/outlook in 30 years
Tell me about the time that an app at Meta grew at 10% a year for a decade and then began doubling every few months.
its probably happened on half the core teams, idk what you want me to say, it sounds like you dont have alot of experience
Instagram, Facebook and even threads all had much more mundane growth rates and definitely no unexpected jumps like GitHub is experiencing. I'm sure if suddenly the solar system had 10 more earths with each about 10 billion people and they would all start using Instagram tomorrow we would have exactly the same growing pains and outages that GitHub has today.
Luckily for Meta agents are not yet as much into doomscrolling as humans are.
instagram growth rate != core service team in instagram growth rate
Don't dodge the question and attack the asker, that's rude.
Can you give an example?
> AWS saw this growth every year for two decades
That does not seem to be true - which two-decade period are you talking about? AWS has only been around for ~20 years, and I just reviewed a 10 year period, and not a single one of those years saw doubling in the whole year, let alone doubling in a few months. Which 20 year period are you referring to, and are you referring to doubling every few months over that 20 year period?
AWS saw doubling in usage every month for 240 months? So it now exceeds the number of atoms in the universe?
Eh, I would be more empathic in this situation[0].
Github isn’t small startup, where other 10x threshold is as cheap as buy bigger box in your IaaS.
When you are already biggest player in the ecosystem and you suddenly get 10x persisted traffic, with at least 30x+ forecast “soon” - I am not surprised they have issues.
[0] even at current MS owned github
cloud providers and hyper growth tech deal with these growth rates all the time
Did you read the article? It is growing exponential.
this is common at AWS/cloud providers
Not sure to be honest, from a machine perspective 2X should never be a big deal, unless 1.4 was the threshold or sweet state and no one thought too much about scale and architecture beyond that
If you’re the size of GitHub and you’ve been running your infra for years with very little variation in traffic patterns you have a strong incentive to optimise costs for that existing behaviour.
I am afraid thats now how infrastructure works from what I have seen. The number that really matters is QPS. For any system the QPS varies through out the day and across the week and months. Most design considerations easily absorb any 2X increase. Pick up any company and the chances are that the servers are over provisioned, no one takes chances specially with critical components.
What you have going on with Github is mix of multiple things. Traffic alone is not the cause from what little I know, it does adds to the problem for sure
1. Infrastructure is being moved to use Azure, and overall all the cloud providers are struggling with hardware at the moment (same is going on for linkedin too)
2. The core teams, the people who knew the existing systems have either been laid off or moved from Github
3. Microsoft veterans are brought in to fill the gap across the board, they are trying their best but its a lot of unknown for them
How much infra have you seen of the top 100 sites in the world? I've worked on multiple top 10, and absorbing a 2x increase (and the peak is very likely more than 2x) is a very very hard problem that would cause hundreds of pagers to go off and load shedding to very high degrees. There is just not tons of unused capacity lying around in wait at the scale of github. "No one takes chances with critical components" is also very wrong for the simple fact that you don't know which is the weakest link in the chain until it fails.
I have some good experience and I feel bad about state of these things too specially given that a lot of it could have been prevented. What you have here is not a single service, its a system compromised of hundreds of services, possibly without clear ownership for some of them after these many years and reorgs. There is not a single person or group that understands the whole system from technical standpoint and pressure points. It akin to people trying to plug the holes as the water starts getting under pressure from different joints. This duct taping is present in almost all big enough systems, you name them.
> "No one takes chances with critical components" is also very wrong for the simple fact that you don't know which is the weakest link in the chain until it fails.
Sure, but the new patterns are several years old at this point.
Not adapting is a choice
If you grow 2x a month, you'll run out of resources very soon. Like the whole Solar System.
Hey, um. Little bit of math. Exponential growth.
Why does Github not segregate the free offerings from the enterprise or even better, all paid offerings?
It is unacceptable that enterprise plans get impacted by traffic on free and public repos. Our repos are neither on the free plan nor are they open. We have not had more AI stuff happening in the last weeks. Our traffic is stable. I would wager that most enterprises did not spike the traffic all of the sudden. Even if they were, we are paying for our quotas. Still our Github actions were breaking and our PRs not viewable at some times.
I am hoping this instability is going to cause a Cambrian explosion of forges and if that is happening, Github will be the first victim of the AI revolution.
I am working on a truly decentralized / local first code review right now, and a big part of my motivation for this is how bad Github has become. I dont know if I have enough time to build CI as well, but I am hoping others do. Otherwise I will just fall back onto Jenkins.
I agree. I'm just a pro user and not a big enterprise. But I feel I should get a lot more GitHub Actions minutes and stability than free users.
I applaud GitHub. However, I think no matter how valiant they are they will not climb out from under this. The scale problem will keep getting worse, and it's getting worse in a way I don't think is translating to more money for them. Sooner or later, they're going to have to charge for things currently free.
I've been saying this for a while: https://news.ycombinator.com/item?id=47534499
That's a business decision for Github of course. They are getting some value out of being the goto place for source code hosting for what is essentially a shockingly high percentage of OSS projects and a really large amount of enterprise projects. It's essentially the largest and most complete network of developers and code in the world. That kind of influence and reach is valuable in itself to Microsoft. And of course it's a gold mine for AI training data as well. Which is presumably why they sponsor it.
But it does raise the question for especially commercial users of Github whether it's time to reconsider the relationship with Github and maybe not put all our eggs in one basket. Basically, this wiped out a whole workday for many companies.
I'm not that eager to start self hosting my stuff. But I am considering it. Besides availability, CI build performance is also becoming a blocker for us. My AI coding jobs creating lots of PRs are making that a bottleneck. Fixing that in Github would require switching to a paid plan. And at that point, self hosting might be the more cost effective option. There are a few tradeoffs here of course. But I like the idea of throwing more memory/cpu at this to get blazingly fast builds.
If pushes are the main problem, they could rate limit too.
Maybe GitHub sees value in hosting the code that is produced, for model training purposes. This would justify to maintain a free offering, even when it gets flooded like it has been recently
And even when this code is AI produced, yes. It has some informational value.
Yes, but there are sharks in the water. Can GitHub afford to lose less money than them, at a slower rate?
Yes, they can, because I don't think other forges are actually making money.
GitHub is the de facto standard.
They are effectively getting spammed / ddosed by vibe coders pushing slop nobody uses.
A majority of this increased traffic is the codex/claude in auto mode used by people who don’t even know what git is.
> Since April, monthly commits have grown from 1.4 billion to 2.9 billion
Bonkers.
You can tell the entire industry is in a "productivity panic" and here's more proof. There's a velocity zealot crying tears of joy somewhere.
Bonkers is right.
Where in those ~12 billion commits is the software, products and "innovations" which are supposed to be making our lives better? Software and apps in particular are getting worse, normies hate AI more than ever because they're even less likely to get their desired outcome when calling their doctor or trying to get their online order refunded when chatting with a cutely named chatbot, wages for (most) knowledge work are being driven through the floor, artists are being squeezed more than ever, etc., etc. That's to say nothing of the existential threats to the economy, environment and critical thinking which are growing daily. I really think we've lost the plot, folks.
This comment comes up over and over again and it's incredibly ignorant.
To give just a single example, ai code dev has enabled people to make tools for themselves that they didn't have before.
I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public
Speaking of ignorance, how does your success story relate to the societal issues GP brought up?
It relates to software making my life better, as OP said he can't see any examples of this
Why is it incredibly ignorant? Where is he wrong? We have definitely lost the plot.
and where is your evidence of that?
On the topic of translation/ language learning, I still find Google Translate better for when I need to type emails in Russian (my older family who only speak it) as far as grammar is concerned than what LLMs spit out.
I'd be surprised if Google Translate isn't powered by a specialized, Transformer-powered model we would now recognize as a "small" language model heavily constrained to a specific task.
that already happened in like 2016
The transformer was invented as a machine-translation algorithm, later adapted to create LLMs. I think it's a reasonable hypothesis that a ML system designed for translation will always beat an LLM role-playing a translation system (assuming equal computing power and engineer effort).
We're going to see is a lot of stuff people use LLMs for moving back to using bespoke algorithms solving the one specific issue. LLMs are just machine learning unlocked for the masses, where you can ask it "translate this for me", "what's the sentiment of this text", and it just does it. Any of these were already achievable before LLMs, they just needed a machine learning engineer to implement.
I myself started an LLM driven project to classify and sort all the photos I've got. I just ask an LLM to poop out tags for an image. Using a proper algorithm for this is possible, but I'd need to put in some effort to actually get it set up. LLMs do make it easier since they can directly, instructively help you that set up, and provide an easy stream of training data to distill out a model perfectly fit to your usecase.
I think majority of work currently going through LLMs is such inefficient tasks that can be solved quicker and better with a limited algorithm.
You don't need a chatbot.
It’s really not ignorant at all when a large majority of these changes go untested.
Most people would prefer an excellent language learning app to simply be available and ready for them to use, rather than having to come up with how such an app could best work, and to get an AI to implement it by themselves, and having to iterate on that. Just like most people simply want to buy nice clothes, and don’t want to become fashion designers and managers of a clothing factory. So far it doesn’t look like AI is making the former (readily available excellent apps) more widespread.
As someone also building a language-learning app for myself, it's not too hard to create an excellent app right now, but the "readily available" part of your requirements is now 10x harder than a few years ago. Users are getting bombarded by new apps, so marketing an app, which was already hard before AI, is now much, much harder.
I could try to market this app, but I tried that 2 years ago, and it was already essentially a failure. So I'm probably going to keep it for myself or just try to find a few users who like it and will use it for free.
I think I agree. But it's nice to have the option. Furthermore, i think it's too early to see the results of this, as technology takes a while to diffuse. Since there are so many people trying to take advantage of the hype to make a quick buck, it takes time to figure out what is actually useful and what is not
Having an app means exactly nothing. How good your language skills will be after N years of learning is the actual measure of whether it's worth anything.
True, i agree. Which is why it will take quite some time for this technology to diffuse and for people to learn which apps really are good
> I've made a language learning app for myself. Its working better than Duolingo so far, for me. Its not really public
On the other hand, famine is coming. Come back and read this next year. Was your dalliance worth these food shortages?
Food production is at record highs last time i checked, thanks to technology
Hang on! What's the proposed causal chain from AI to famine here? Inference -> energy prices -> fertilizer/food prices? Or are you referring to the war?
Climate change, most likely.
> ai code dev has enabled people to make tools for themselves that they didn't have before.
...that sounds pretty worthless tbh. The promise of AI wasn't that you can bang out your own little half baked duolingo clone. It was that every engineer was supposed to become hundreds or thousands of percent more productive. The assumption being that things generally would therefore get noticeably better.
We're seeing hundreds of percent more commits. We're not seeing that translate to any real world improvement of anything. That's the problem.
Call me "ignorant" if you want, but back it up by showing what all this "AI productivity" has actually accomplished in real economic terms. Not vaporware.
Out of curiosity, what real economic terms would you have to see to be convinced?
Do you mean more earnings per share for corporations or real utility impacts on social systems (e.g., more new drugs)
When I ask Claude about the AI ROI, it seems to cite that ~95% of AI PoC's are negative ROI but the 5% that do have ROI tend to have decent return -- my guess is mostly back office clean up to reduce expenses and increase revenue/profit.
IIRC Uber claim their 1.5k/mo budget lead to no new value creation.
FWIW, I look at the situation with similar skepticism. One argument you could make was what Marc Andreesen said with the hypothesis that "all the big companies get nuked", so no new valued gets created but the large market cap companies get eaten by thousands of little pirañas. Obviously, so far, that doesn't seem to be the case.
> Do you mean more earnings per share for corporations or real utility impacts on social systems (e.g., more new drugs)
I mean real utility impacts. More reliable and efficient software, better features, etc. So far we've seen none of that. In fact, it seems like the opposite--more volume but a much lower quality product.
Are you going to be writing your own exam too?
As a hobby I am playing around with mesh radios using Lora.
Let me tell you, it is _silly_ the amount of vibe-coded software in this area is popping up every week. Is the software duplicated? To an extremely large extent, yes. It is useful? Yes, but each piece of software seems to have a smaller and smaller audience, and quality is often severely lacking.
Iv'e done my own, too, for "RF debugging" as I called it to look into SNR issues related to interference. The software getting produced is likely useful _somewhere_, its just that youre not going to notice it.
Are we just moving towards personalised digital assistants for everyone, which in turn will produce software to function? Not unlikely.
Yeah, a lot of the software around MeshCore people have created seems heavily vibe coded (since got used to having press a button multiple times until it works) & the community seems to be surprisingly non-critical to "AI" technologies. Is it just about timing or something else ?
> Where in those ~12 billion commits is the software, products and "innovations" which are supposed to be making our lives better?
Where is anything saying there should be? If they were all private hobby projects, you'd have no grounds for complaint.
> Where is anything saying there should be?
What are the boosters ranting and raving about if not the above? Am I just confused and the end game for all of the AI hype, investment, infrastructure build out, etc. over the last N years is ... hobby projects?
Here's a quote I found after a quick web search for "dario ai benefits":
> The list of positive applications of powerful AI is extremely long (and includes robotics, manufacturing, energy, and much more), but I’m going to focus on a small number of areas that seem to me to have the greatest potential to directly improve the quality of human life. The five categories I am most excited about are:
> Biology and physical health
> Neuroscience and mental health
> Economic development and poverty
> Peace and governance
> Work and meaning
https://darioamodei.com/essay/machines-of-loving-grace
It's hard to separate out the hype from the results, but if you think none of that's being tried or worked on then you're the Ostrich with your head in the sand. Mayo Clinic says AI models help detect pancreatic cancer on CT scans 3 years before clinical diagnosis.[1] Breast Cancer.org says emerging research shows AI might spot cancers in mammograms that radiologists might miss[2]. Nature.com has a paper conclusing that AI (in various forms) have "already revolutionised the diagnosis, prognosis, screening, biomarker discovery" of cancers.[3] Cancer.gov says NCI scientists are using AI to improve cervical cancer screening with deep learning and digital images[4].
Dr David Burns offers his Teams CBT therapy with an LLM based app for people who can't afford therapist visits[5].
U. Chicago says that AI weather forecasts can be faster and cheaper than supercomputer-based physical simulations and produce results weeks ahead, which makes accurate forecasts affordable and available for farmers in middle-income countries[6].
With your smartphone camera you can get an AI-assisted bike-fit at home[7].
You can get a bike fitness training plan[8] and performance analysis with an AI that you can chat with to analyse and explain or change it, used by three professional cycling teams.
Prophix Australia analyse photos of powerlines taken by helicopter using AI to track assets and identify and verify potential faults[9]. AI systems scan images for early signs of plant disease in the vineyards of the Swiss Alps [also 9]. AI systems analyze multispectral images and satellite images of crops to detect nutrient deficiencies or pest problems in large farms[also 9].
In 2020 - 2022 a trial of a system in a hospital to identify patients at risk of sepsis helped medics treat them 2 hours earlier and reduced the death rate from sepsis by 18%.[10]
Self-driving cars now exist. Camera drones which follow you and respond to gestures now exist.
[1] https://newsnetwork.mayoclinic.org/discussion/mayo-clinic-ai...
[2] https://www.breastcancer.org/screening-testing/artificial-in...
[3] https://www.nature.com/articles/s41698-026-01276-6#Sec17
[4] https://www.cancer.gov/research/infrastructure/artificial-in...
[5] https://www.feelinggoodapp.com/
[6] https://climate.uchicago.edu/insights/ai-is-transforming-wea...
[7] https://www.myvelofit.com/
[8] https://analog.io/sports
[9] https://medium.com/design-bootcamp/what-has-ai-ever-done-for...
[10] https://www.scientificamerican.com/article/algorithm-that-de...
Crazy how more code haven't really yielded better products. Who would have guess that the number of lines of code wasn't the problem. Except that very basic tools that I use daily, pretty much every piece of software has become worse in the past few years. The closer to the consumer the software sits, the worse the quality is becoming.
> Crazy how more code haven't really yielded better products
Crazy how anyone would think it would.
There's something poetic about a company that's leading in the realm of funding, researching, using, selling, etc. AI is actively watching the destruction of one of its core products due, in large part, to that same AI they're selling.
Is it a "productivity panic"? I'm personally pushing a lot more commits because I'm spending more time playing around with ideas on side projects.
Is this all wasted if commits drop in a few months because folks are just pushing a lot now to figure out what works and what doesn't and we eventually settle into something between where we are now and where we were a year ago?
Cry in story points. Like a real scrum master level 9000.
Everyone suggesting that they simply charge users for commits to drive off AI-heavy users forgets that Github is owned by Microsoft, who has a big incentive to keep having developers use AI.
I suspect that Microsoft would even prefer to have Github operate at a loss, if that loss were because all its users were using their models and paying for OpenAI subscriptions to generate the code.
This is like letting someone stay in your house for free while charging them to burn it down.
If you can sell them fuel, it might be a lucrative business.
Thinking more, the object of the "house" is to extract value, whatever it is by rent or letting them burn it down, as long as the correct dues are paid, the mission is accomplished.
The long view is that MS will sell those developers the tools that slow down or put band-aids on the damage they inflict on their own codebases by using AI.
> I suspect that Microsoft would even prefer to have Github operate at a loss,
I assumed it does, do you know that it doesn't?
I presume there’s a lot of companies out there paying GitHub very large sums to host all their private repos.
No doubt. ...You can have non-zero revenue and still be loss-making though.
Conversely, what suggests GitHub has a huge operating cost?
Running a GitHub clone at their same scale as a customer on cloud pricing would likely be insane. But y’all know infra is actually quite cheap when you run it yourself right?
It’s usually the case with these M&A deals that the profit just never quite makes sense to justify the purchase price, unless you can truly scale up the user base or revenue model. GitHub was already so mature as a solution when they bought it, I don’t know that they could have added that type of value just by slapping a Microsoft logo in the footer.
> what suggests GitHub has a huge operating cost?
They just added 3 million CPU cores to the fleet. That’s a lot of cheddar.
Absolute numbers aren’t useful in this analysis. We know capital is easily available for the expenditure. You have to consider it as a relative number to revenue or users otherwise big numbers always look scary.
I agree - however, I was responding to your direct question about whether GitHub's operating cost was high, and there's some data that suggests that it is!
If this is one of their major expenses and only represents 5% of annual revenue, it’s not. Often SaaS companies can run at 90%+ gross margins even with a highly utilized free user tier. Their selling expenses and labor/opex and SG&A run rates can be all over the map but these are extremely controllable and represent the operating strategy being executed more than the cost of service.
Would you say that the operating cost of running an airline is high or low?
It used to be, and buying a few servers was a cheap way to get off the cloud (if you can manage them efficiently). But with RAM and other costs these days... I've had to rethink it a lot.
It's at worst ten times as expensive as it should be. But AWS was already ten to a hundred times as expensive, so even in the computer parts crisis, self hosting is still ahead.
[dead]
They could do this in a way that lessens external pull requests.
For example, bolting co-pilot on to github, or a Codex in the web kind of thing that gives unlimited check ins.
It's like how Grok Heavy gives the user X premium or whatever. You charge for the tokens and give the unlimted premium access as a bonus. Basically, bundle it.
> bolting co-pilot on to github
They already did this, no?
I meant as a prerequisite to unlimited checkins.
Right now anyone can publish to public repos in an unlimited manner. They could choose to limit that and elevate unlimited to a paid co-pilot of codex bundled plan.
Microsoft, who would also have an incentive to extinguish the largest hub for open source development, having already embraced and extended it.
15 years from now, they'll say it was obvious.
Microsoft has a financial incentive to push LLMs and also an existential reason since LLM-generated code is incompatible with the GPL.
The GPL is irrelevant nowadays anyway since everything is MIT (corporations won the license propaganda war). I think my computer has, like, five GPL programs? Ten? It's just that two of them are coreutils and Linux, and that basically holds up the GPL as a concept.
operating at a loss and non-operational because of outages are very different. If they can't maintain service levels nobody - AI super user or quant, old-fashioned human - will be happy.
What Microsoft models?
> Errors in those services triggered a client-side retry loop that increased traffic during recovery.
The worst outages I've been part of always have some version of this :(
the 'ol thundering herd problem...
Exponential backoff is your friend... too few people use it.
Jittered exponential backoff. You don't want the whole herd to come back at the same time, you have to add timing jitter to the clients.
Possibly dumb/silly question.... Are there any sorts of reverse proxies out there that provide their own layer of jittered/exponential backoff based on patterns? (i.e. requesting IP, cookie, etc.)
I suppose the main reason I think it might be a bad idea, is that it would add complexity to the reverse proxy (i.e. now it's having to track whatever thing is being used and that complexity itself becomes a potential failure point...)
(To be clear, the clients should have their own backoff procedures, but I'm thinking about cases involving naughty clients, which are sometimes a harder problem to correct for...)
I believe envoy has it built in and istio (uses envoy) has different levers for circuit breaking and retries. I’m sure lots of them have it as an option though outside of these.
In general, helping prevent retry storms is one of the functions of a service mesh. E.g. Istio lets you configure exponential backoff with jitter, linkerd provides a retry budget and loadshedding based on backpressure.
“Adding complexity to the reverse proxy” is kind of the central feature of service meshes, but the point is you pick one that’s battle tested and solves more problems than it creates.
Exponential is also overkill (even with jitter as others have mentioned).
I seem to remember there was a "you failed 5 PIN entries in a row, please wait 500000 seconds before you retry" on Apple phones. So, you probably also want a sensible max... which makes exponential a bit pointless. Just do a basic fixed delay + (large, e.g. 0.5 x the delay) jitter and you'll be fine for most things. You can add a bit of cumulative delay if it's really costly to do retries.
Cap the number of retries before the user has to manually try again. That's fine.
Don’t forget jitter!
I always add some jitter but never actually had a problem where it would have been relevant. Recently I added it to a project where others also see it (not just a hobby thingy but something at work) and I was wondering if it would look silly, like premature optimisation. I looked on Wikipedia for how established the practice is and it barely gets a sentence... with no reference.
Do you know of a documented instance where it would have helped?
This doesn't answer your question, but I faced an issue where an application had to retry if things failed and I ended up with the "thundering herd" problem and introduced jitter without knowing it was a relatively standard practice.
I felt dirty implementing such a solution (introducing randomness for the sake of randomness is off-putting), but it worked. It wasn't until a while later that I even heard the term "jitter" in this context and realized this was a pretty decent solution for this kind of problem.
Regardless, if you're going to introduce something where adding jitter is appropriate, I'd just just add jitter. It's not premature optimization; it's an essential part of that kind of functionality.
It’s a fundamental property of Ethernet[1]. Does that count?
[1] https://en.wikipedia.org/wiki/Exponential_backoff#Example
Of Ethernet in the 1980s*
If you have a switch in your network, or a speed faster than 100Mbps, you aren't using this model.
However, WiFi uses it.
It’s the first thing that came to mind!
It’s not hard to get started, it’s a case of adding small amounts of randomness.
If you have, say, a long poll then kick off all users due to a deploy or error (or a broadcast message) then you can have a situation where you’ve got a huge clustering of connections at 1 minute, which spreads very slowly out as real life issues give you jitter for free. You can avoid this or at least return to normal much quicker by adding some jitter.
It might happen if all your users back off at the same rate too, if the clustering causes a bunch of errors. Error -> lots reconnect 1 minute after -> fail -> lots reconnect 2, 4…
More likely to occur in cases where there’s a way you can have people all connecting at the same time - synchronisation to a real world event is one case and then connecting again at the same time after.
You have thousands of users connected to a chat via websockets to a small cheap server that can just handle the load. Server has a hiccup, all clients disconnect, server comes back, all clients reconnect at once. Server can’t handle the load.
Have experienced it, but didn’t document.
Downstream database of our edge serverless platform went down. A tonne of requests failed all at once. Every service in the microservice request path, and the client, had their own retry policy.
Clients all retried at the same time. Retries amplified in our microservice graph (1 request at the front door ended up with like 10s of retries internally as each downstream microservice along the path retried requests). Request queues backed up and couldn’t drain fast enough. Clients all timed out at roughly the same time. All waited the same time. All retried again at the same time.
It was a pulsing thundering herd of many hundreds of thousands of requests at the front door that was amplified by internal retries.
Had to tune up load shedding to 100% after the database outage was mitigated until the backend recovered then tune it down in increments to restore service.
Added jitter to clients and turned off retries on the serverless platform.
Exactly, without jitter the thundering heard problem turns into trying to escape a small island with a small boat and big waves hitting the shore problem. You can never fully recover before you get smashed again.
Even if you're operating at a scale where thundering herd won't crash your service, introducing exponential backoff + jitter to your retries also helps when you're just viewing the logs. Massively reduces log spam when you make changes.
If you’re talking about internet clients, I think the real world provides sufficient jitter. If you’re talking about a fleet of clients on your 10gbps network, jitter might be useful.
I've had to simulate jitter recently to reproduce a memory spike in a go app using traffic control (tc). have you observed how your app works under jittery network?
Reminds me of this blog post I read yesterday, about Buildkite introducing jitter in their CI agent to reduce synchronisation
https://buildkite.engineering/sleeping-at-scale/
Thundering herd is different. Thundering herd is when a lot of clients trigger requests at the same time. Common situations being a specific time, some other event just occurred or synchronize on other parts of your infrastructure (such as readers queuing behind a R/W lock that then get unblocked at the same time to continue to make a bunch of requests at the same time.
Request amplification via retries is a different problem that causes large amounts of traffic (but it is generally more steady than spiky)
> Errors in those services triggered a client-side retry loop that increased traffic during recovery
Symptomic of a wider trend to avoid showing the user any error at all costs, even if that means they sit watching a spinner for 7 hours.
> Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.
The detailed root analysis tries to pass this off as a "bug". You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour is functioning exactly as designed. In this case aggressively to try and hide problems if token service responses become flakey.
A common pattern in highly available services is that sometimes you should retry immediately (because the node you hit is rolling/broken/overloaded, but the others aren't) and other times you should back off aggressively (because the service is degraded).
If your server indicates with 100% accuracy when to retry immediately vs backoff, AND if all your clients consume that information with 100% accuracy, things go great. But there are lots of situations where one or both of those breaks down.
Anyone who designs such a system should know to use an exponential backoff to avoid the thundering herd. Maybe copilot missed that while it was reviewing its own PR
Exponential backoff is the wrong answer in a highly available system in the typical case where (a) failure is expected and (b) you have nodes you are supposed to fail over to.
Your highly available system is probably somewhat important, otherwise you won’t have invested in making it HA.
While your premise holds for happy cases, when you do have a cascading series of outages, not using exponential backoff is just adding a self-inflicted DoS to when you do go down.
I don’t really follow your premise and can’t really articulate many cases for when you shouldn’t use exponential backoff. Maybe if you’re working at Jane St or something; or other circumstances where you can deploy immediate changes to the client; and you’re willing to trade ‘better p50 for worse outages’.
But in the case of shipped code that’s run on clients, I’ll continue exponentially backing off all the way, all the time, for everything.
When you have an outage, you should not retry at all. Exponential backoff is exactly how you get cascading outages. If service A fails a request to service B and decides to exponentially back off, now service A is holding open an end user request that will claim resources on service A. Fast forward ten minutes and the service B degradation has metastasized into a service A degradation. And even after service B has recovered, service A might still be dead.
To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients. Then service A knows service B is dead, does not retry, and proapgates the failure to clients. This is hard to do perfectly, but there's no alternative that works.
> To handle this correctly you need your RPC framework to accurately communicate retryable vs non-retryable failures to clients.
Even this is not enough, since you cannot always reliably know whether service B is dead or suffers an intermittent issue that can be safely retried just from looking at a single failure.
The classic solution, in the monolith/few-services world would be a circuit breaker. High failure rates on any service trigger a circuit breaker in the services calling it, and they'll wait for a cooldown period before trying again.
When you move to a massive microservice architecture with hundreds or thousands of microservices, setting up circuit breakers manually becomes very hard to track and do reliably. Service meshes like Istio make this slightly easier, but they still don't let you verify that all possible paths have circuit breakers and that retries are not excessive etc.
Pretty much, but ideally it should be transparent to your app developers. App developer writes `rpc.doThing(...)` on one side, and an implementation on the other, and the infrastructure -- the RPC framework or the service mesh or whatever -- transparently handles when/whether to retry and where to route retries.
The whole point of exponential backoff is that the first retry can be quick.
The right answer is for the RPC framework to accurately communicate "try again on another node" vs "don't try again, just hard fail".
When one end user request fans out to hundreds of backend requests (typical for microservices), you can't have each of those backend requests do its own exponential backoff. If they do it in parallel, they're a thundering herd, and if they do it in serial, the end user request will time out before you finish all the work, at which point you're doing a bunch of slow expensive work for no gain (and the enqueued slow expensive work will make your outage worse).
This is why you have circuit breakers upstream. Not on every individual instance.
Doesn't do you any good if the outage is in the circuit breaking layer, which it was for GitHub (this started as a load balancer outage).
Ideally you have levers further up from your local load balancers as well. Even at the edge. Granted you never want those to trigger but it’s better than fighting a storm while you fix things.
I wondered that myself. Curious as to why they couldn’t shed load at the edge to help protect goodput.
Isn't that what they did though? Start returning more-or-less hardcoded 403s for the Copilot endpoint that was causing the issues?
That’s more surgical than load shedding. With load shedding you intentionally return 503s to a proportion of all legitimate requests. It turns a hard blackout (total outage) into a flakiness issue.
In highly distributed microservice architecture, there's almost never a single upstream. In some cases you may have a couple of customer-facing entry-points (a global API gateway, and a couple of BFFs), but these are not the only paths that need to be protected.
There are client-side retries (which have broken GitHub in this case) and server-side initiated API calls between microservices that don't pass through any of your ingresses (e.g. triggered by an ETL pipeline, or a scheduled job).
With a complex architecture you can't just slap a circuit breaker on a couple of ingresses and call it a day. Don't get me wrong, putting them there does go a long way, but you won't be covering all your bases.
I completely agree, I currently manage a fleet of microservices that handles a few trillion requests a month. It’s about defense in layers to these sorts of things. All the way through the stack if possible starting at the edge.
At least it should be required for critical level services in production.
Close but not quite: https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/...
“Circuit breakers, where calls to a downstream service are stopped entirely when an error threshold is exceeded, are widely promoted to solve this problem. Unfortunately, circuit breakers introduce modal behavior into systems that can be difficult to test, and can introduce significant additional time to recovery. We have found that we can mitigate this risk by limiting retries locally using a token bucket. This allows all calls to retry as long as there are tokens, and then retry at a fixed rate when the tokens are exhausted.”
Thats why you only back off at the edges, (not internally, where you can spin up on demand) and why you should add jitter to the timing.
This is probably the best and most thorough explanation I’ve seen: https://builder.aws.com/content/3EumjoZascWd1oZiEgL8ORlv3qE/...
I’m particularly fond of the token-bucket mechanism for pacing recovery.
Why? You can retry, but there is nothing wrong with increasingly waiting slightly longer if we fail many times.
"exponential" and "slightly longer" are very different backoff patterns.
The typical ceiling for these is around a minute.
Which is probably hundreds of times your normal wait.
This can either be a relatively short time or an eternity, depending on the upstream consequences. If that means holding a connection open for 60000 milliseconds while waiting for some downstream rpc to go through its backoff ritual, that's an eternity, and under load that connection pool will get exhausted quickly. So now a problem which should only affect maybe 1% of users has completely hosed everyone.
I've seen this happen multiple times. Someone designs some clever backoff strategy without considering how it fits in the context of the rest of the system. Hilarity ensues.
If you have something taking an entire minute on a computer, please ensure you implement it in such a way that no connections are actually held open for that entire minute.
We are probably talking past each other.
I'm not proposing doing any exponential backoff retries, or even retries at all for internal services.
In my mind, the retries with exponential backoff and jitter belong only on the end-client(VSCode in this case). Everything else -> fail fast.
With http you can do it elegantly with a 429 and retry-after. 100% doing it at the edge is the way to go. The machinery which determines how long to delay a client's retry can benefit from knowledge of internal services' state, but I agree the ultimate decision must lie with the serving layer. That's the only way to efficiently deal with misbehaving clients.
On that note, one of the more memorable incidents of my career was when a 10M+ node client decided to retry as hard as possible on 4xx. That was fun x_x.
[edit] that is to say, for this mechanism to be robust your retry-after enforcement mechanism needs to be capable of withstanding almost every single one of your users attempting to illegally retry as fast as they physically can without negatively impacting that one user requesting legitimate traffic. https://media.tenor.com/p3mss3YI6TcAAAAM/wat.gif
Try asking Opus or Fable that question. It'll give you a good answer on why microservice architectures work the way they do in order to keep user-facing latency acceptable and minimize downtime. It's a complicated enough topic that I don't feel like explaining it for free to you in a HN comment.
Why are you pretending this is a harder topic than it is and you have some secret insight. And you're a dick on top of it:
Yes, exponential backoffs alone are insufficient. Yes, adding jitter helps randomize the calls across a fleet and should be the default with exponential backoffs. Yes, both of these may be sufficient for most systems. Yes, you can dive more into circuit breakers and adaptive retries to limit thundering herd.
https://aws.amazon.com/blogs/architecture/exponential-backof...
https://brooker.co.za/blog/2022/02/28/retries.html
Someone linked the Google SRE book. It explains that client services watch the global distribution (for that client) of number or retries. It will normally retry immediately on another node, because as you say, some failures are expected. But if it notices that more than about 1% of requests are having to retry twice, that indicates the server service needs reduced load and it starts refusing to retry. If it gets really bad it even starts preemptively failing first attempts so that the service doesn't get contacted at all.
> because the node you hit is rolling/broken/overloaded, but the others aren't
Retries in such a situation should be handled internally with the client at most responsible for failing over with a circuit breaker to another zone. Having the client auto retry right away is not something that behaves well as shown here, even if in the happy path it happens to stimulate increased availability without actually investing in the proper architecture for it
CAP theorem. Pick one of those.
We used to be able to afford two! Acronym letters cost as much as houses nowadays!
OK I'll pick C and A!
https://web.archive.org/web/20250128235041/https://codahale....
The problem is that P is actually part of A. It should be called the CA theorem.
It's all about what you choose in the face of P. Do you choose C or do you choose A?
Yes. But calling it CP or AP makes it seem like you can also choose CA.
(Also both CP and AP have alternative very dodgy meanings)
> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour
That wouldn't be a unit test - that's more like an end-to-end or integration test.
Have you ever worked anywhere that had perfect test coverage? It just doesn't happen, nor is it possible unless you're building a calculator app or todo list.
I'd disagree it needs end-to-end. Testing that proper sleep calls are made followed by an error can be done in unit tests.
Unit tests already mock network calls so create mocks to return errors instead.
https://news.ycombinator.com/item?id=49384968
My employer has 0% test coverage lol. I've begged and pleaded, but the claim is that "risk is low" and "that's what QA is for." Hell, I've complained to senior management about how there are senior devs that forgo backend validation. It's truly Hell in the trenches sometimes.
Some days, I would seriously rather work at Wendy's.
> Some days, I would seriously rather work at Wendy's.
Narrator: He would not.
If such a job paid even half as much, I would quit today. I can continue to program computers as a hobby. Occupational programming has essentially eroded my passion over the years.
Money is not important to me insofar as I have enough to live an average life. I don't need anymore than that.
Just write the tests. It's your job, not your employer's.
It's their code, not mine. If they don't want us writing tests, then so be it. I get paid either way.
You're the engineer, not them - you're being paid to write code that ideally doesn't break - it's your job to explain that tests benefit them. There's very little excuse not to add test coverage as you go along these days given an LLM could help write most in a flash.
Or, don't tell them and do it anyway. It'll make your own life 100x easier, unless you enjoy the stress of bug fixing and responding to incidents where you've broken something.
Writing code without tests because it takes too long is like running a car engine without ever bothering to change the oil - it'll run for a bit, and eventually fail catastrophically.
It's not your job to explain tests are good. It's your job to have tests.
Maybe the retry logic was vibecoded instead of using an existing hardened library. After all, according to Twitter, nobody is looking at the code anymore.
> instead of using an existing hardened library
A lot of retry libraries I’ve seen require the user to configure them. You can use a library with all the right settings, but if you configure it wrong, you are really no better off than if you hadn’t
They added 3 million CPUs. You reduce the complexity of their systems to a unit test…
> a wider trend to avoid showing the user any error at all costs
And in fact you can see the degradation of software over the previous decade-plus via Google Trends search for ‘something went wrong’ lol: https://trends.google.com/trends/explore?date=all&q=%22Somet...
Ironically I get an error page when I click on this.
> You can't seriously tell me client retry doesn't have a unit test which ensures the retry back off behaviour
Not to join the parade, but what would a unit test that confirms a cycling behavior across all the instances in-flight even look like? I mean, besides "Not a unit test".
Mock out the network call (which you'd be doing anyway because unit tests never connect to the network) so it always returns a failure. Mock out the timer.
Call the function multiple times and ensure it'd passing the expected wait durations in for each time it's called followed by a fatal error after say 30 seconds.
> Mock out the network call
Man, the day this hits, just know I didn't judge you at all.
So, in all of your software, you have introduced randomness in your retries so that the billions of your clients avoid retry synchronization dances?
I'm not asking for randomness, I'm only asking it to stop retrying after say 10 seconds and report it to the user as a fatal error.
it would be surprising if this wasn't already the case.
Backend API rate limiting has to surely kick in and force you to wait x amount of time before you try again… Discords bot API actually sends you how long before you retry.
As a client you can hit a server as much and as often as you like. The only thing the server can do is return an error code or try to hold the socket open (which the client can then close on their own).
It can block your IP address at the firewall level. It can't stop you DDoSing it with raw packets, but that's very unlikely to happen unintentionally, because TCP will wait for a SYN-ACK response for at least several seconds and possibly up to several minutes.
Except the other side of this is interrupting a service which would otherwise have succeeded: there's a lot of unattended or minimally attended processes where an interruption is just asking the user to do the only thing they were going to do anyway - retry it.
In GitHub's case this is especially relevant - the only reason to throw an error message at the user is the hope they - the human - give up and walk away (or you break all the CI/CD builds and the time it takes humans to hit "retry" gives you some breathing room).
"You can't seriously tell me that the unhappy leg of the code path has no test coverage."
Sometimes I forget how ignorant HN can be of real world software development and the bar of corporate code quality, and then bangers like this remind me of it.
"Please don't sneer, including at the rest of the community." It's reliably a marker of bad comments and worse threads, and you can make your substantive points without it.
https://news.ycombinator.com/newsguidelines.html
"You can't seriously tell me that the unhappy leg of the code path has no test coverage."
Cloudflare outage on December 5, 2025 [1]:
> However, we have never before applied a killswitch to a rule with an action of “execute”.
[1]: https://blog.cloudflare.com/5-december-2025-outage/
Exactly this. I've seen production level trading systems grind to a halt over a simple bug and no matter what tests you have in place, it happens.
I cannot even begin to express how many times I've seen engineers working super hard to optimize happy-paths so that we turn 3 nines of availability into 4 nines but introduce unintended emergent behaviors in unhappy-paths that turn 1 nine into zero nines via thundering herds, retry storms, etc.
You have my empathy for this kind of sentiment. Personally this seems somewhat rare in practice. That being said I'm curious if anyone has anecdotes they can share about these kinds of things?
Some anecdotes
1. People add retries w/o backoff. Now we have a retry storm.
2. People don't add jitter so we get huge waves of highly correlated retries that cause self-perpetuating overload cycles and failure.
3. People add retries (even w/ backoff) at more than one layer (e.g. one in process, one in envoy), so now we have a deeply confusing multi-level of retries with super weird n^2 patterns.
4. People find ways to fetch from multiple data sources to make a composite object but don't cache/reuse data they fetched, so one data source being down causes DOS on _other_ data sources because of retries.
5. People add failover mechanisms where all failures happily pick the same failover target because, ya know, it has the most free capacity.
6. People underestimate connection setup cost so "failover" causes huge increase in overall load due to connection setup (often tls setup) causing huge influx of "new" cpu work in a loaded system. CPU spike causes unhealthy destinations, causing more failovers, causing metastable failure.
7. People realize purely local decisions aren't optimal so add a layer of global health metrics (e.g. global retry budgets) but these systems add a time delay to the metric (ie its the view of the world 5ms ago). This delay makes their naive control algorithm go into oscillations or divergence spraying stuff everywhere.
All of these things have a similar pattern that when things are almost all good, they will make the system more robust. You'll get an extra 9 of reliability on good days. But they make everything far worse when things are bad.
"Oh no, I hit a timeout" -> "I've increased the timeout and added a retry" -> "Oh no."
Configuring postgres to automatically failover instead of doing it manually. The automated system caused more downtime in a few months than manually doing it did for years before. All in the name of more automations and less downtime
For resolution of a problem you can’t prevent categorically, you have to spend a good bit of time pretending to be the computer before you can trust the computer to do it. A run book is a draft of a requirements doc for a program to do the same thing. And you can’t afford to test that program in production without loads of simulations first. And that’s running the process manually and checking aggressively for any signs of problems.
Some coworkers snaked the autoscaling work out from under me, and then ignored the advice I offered on low water marks and weeks of testing and rehearsals. All I can figure about their irrational exhuberence is something to do with claiming something for annual reviews? All I do know is we didn’t make it 40 hours from when they flipped it on until we had our worst outage in two years. Classic FAFO.
They were hoping to eventually get to a low water mark of just over 30% of our static cluster size, and they decided to use that for their initial go instead of my estimate of 40% being the low, and a recommendation that they start with 50-60% for the initial weeks and then ramp it down. When I refused to let it go in the status meeting, the team decided we should vote on it. Two guesses how that went.
A bug in the deployment logic the Ops team had for doing things like upgrading VMs caused the next update during daylight hours to spin up the new instances at the minimum cluster size, instead of the current cluster size. It was done outside peek hours but that still had us cut over with just under half of the hardware we needed at that moment. Because we were in such a fucking hurry to be something like third in line to use the new autoscaling support, nobody else had run into this problem yet (or system did a pretty good job of buffering everyone else).
A couple weeks later they’d looked at the stats and decided that we were spending less than a half hour a day running at the low water mark, and the additional shutdowns were causing churn that made it harder for us to detect problems like memory leaks. Surprise, surprise, they increased the min cluster size to exactly what I’d told them two weeks before.
Since “listen to me” isn’t a lesson that transfers to other teams, I will instead say, don’t transfer initiatives to a new team when there is less than 10% of the project left. The lack of friction you encounter may result in a lack of respect for the danger, and attention to potential problems. And if you must transfer, if your ideas about rollout timeline end up being less conservative than theirs, be patient and do it the slow way. They probably have encountered problems you haven’t seen yet.
Could you elaborate on why that would cause more downtime?
I think the "happy path" might be a slightly wrong classification in GP, since the post is in reply to a retry-storm issue and explicitly talks about retry storms and thundering herds.
I've seen many cases where engineers optimize the sad path, but pessimize the wretched path. Or in less flowery language, they cut the occurrence rate of common non-critical failures, but by doing that they introduce code that can make rare failures much worse.
The cases I've seen generally boil down to naive retry logic or poorly tested and poorly maintained fallback paths (such as killswitches that break their environment[1], graceful degradation turned graceless, dormant feature flags that get reactivated).
The case you see with a retry storm here is the most classic one and the one that annoys me the most. I've seen engineers adding aggressive retries even into places where the impact is minor (you could show an error and let the user manually retry instead). Retries that improve user experience can be great if done correctly, but I've never seen the authors of such pull request addressing the risk and mitigation techniques for retry storm or retry amplification.
I've seen cases which had:
1. Retries on the client side (browser or mobile app). 2. Retries on the BFF. 3. Retries on Microservice A used by the BFF. 4. Retries on Microservice B used by Microservice A. 5. Retries on Critical Service C used by Microservice B.
Most of these retries had very short timeouts (e.g. 100ms), in order to keep latency SLOs during normal operations (not a good idea on retries). Every time QA saw a layer without retries, that would be a bug, and adding retries is easy, so we'd get a new retry without much thought. But the first time Critical Service C became overloaded, Microservice B started timing out a couple of times and retrying. This was too much too much for Microservice A that had a short timeout that couldn't hold the 3 retries done by Microservice B, so it making doing its own retries, all of them dropped in the middle of the way. Eventually you'll get a full-blown retry storm where every request from the client side got amplified with 3^5 retries, easily bringing down Critical Service C.
We'd usually introduce a circuit breaker for the particular path that caused the issue, but a variation of this kept happening several times because designing safe retries across a vast collection of microservices takes a lot of effort, and it's always easier to just add a quick-and-dirty retry at any point where you think you might need one and call it a day.
A proper solution (which I've never seen implemented) would be an mandating a corporate-wide inventory of retry-paths, and monitoring it for any path that is at risk of triggering a retry storm, or adding mandatory headers that cross microservices and track the amount of retries done up the chain and the time spent in total waiting for previous retries. You could have a budget for both and automatically stop performing more retries. Both solution require extra effort and a large degree of coordination.
[1] This was the CloudFlare issue mentioned in this thread https://blog.cloudflare.com/5-december-2025-outage/
> I've seen many cases where engineers optimize the sad path, but pessimize the wretched path.
As said GP, yes this is exactly what I meant and "happy-path" was an unclear choice of words. They optimize for the unhappy path on the good days, and make the bad days much worse.
> Exactly this.
It seems to me like you're saying nearly the opposite of them. You're saying "no matter what tests you have in place, it happens", while they're suggesting low standards and few tests.
More like: those tests are useless because the fundamental system design is garbage thanks to the incompetence of corporations.
It’s been proven that VS Code has product managers who vibe code commits that get merged, including the co-authored with copilot bug. So there’s some problems there.
That was the final straw that stopped me using it.
I didn’t/don’t want the AI, I didn’t like the telemetry but it could be turned off but the cavalier disregard to just yeeting shit into a tool used by that many people no thanks.
What do you use now and what is your primary use case?
I have been using Zed lately and quite liking it. I used Jetbrains for years, but it's gotten absurdly buggy lately so I probably won't renew it. I have also used vscode and I need to use it at work, and it's fine, but I prefer Zed (fewer bugs, faster, mainly).
I use jetbrains editors, but mostly goland which might be more stable then others
I recently tried zed but it ddos’d my nas over an smb share lol
I always advocate for the "boomer" editors when possible. It doesn't matter which one you choose. People can say what they want about them, but very little software survives almost 60 years. I sure as Hell hope VS Code doesn't.
For me, boomer editors (mostly nvim to be honest) join a long list of other things like tiling window managers as something I genuinely want to learn, but that I can never justify learning over either learning something else or doing "real work". I've been slowly introducing both and making gradual progress, but its extremely slow
Gradual progress is still progress.
Just pick a specific task that you do routinely (say markdown editing) and just use neovim for that one task - eat the elephant one bite at a time and markdown is generally small enough/light enough that you won't get out into the weeds trying to make neovim the perfect IDE in the process.
At some point, esoteric UIs may have been needed in the 70s, but that ship has long sailed. I don't need to give myself carpal tunnel syndrome trying to quit vim. If anything, a lot of that is also a problem that continues to plague software to this day: devs who go "well if it works for me, it's good enough for anybody, and if you don't like it, you must not be a Real Developer!"
Learning and using vim keybindings was one of the crucial things that mitigated my RSI.
Ergonomics are highly variable, so it might not help someone else, but for me it did the opposite of "give me carpal tunnel syndrome."
neovim, general text editing/small scripts/markdown where the task is short/light enough I just want to do it and keep moving.
Intellij for basically all software development that is more than 2 minutes.
They also charge ahead to prod with known defects and then change their story when they're caught out https://news.ycombinator.com/item?id=48032310
If you've only worked at a small-to-medium privately held company you'd assume every company ran that smoothly. "Too big to fail" makes sense once you've worked for a behemoth and not before.
Copilot probably wrote the code confidently and it was never looked at by a human then yolo’d out to meet the pointy haired bosses arbitrarily short timeline to then look good to his boss and so on
Test coverage varies a lot, but error paths seem almost universally untested.
SQLite is the exception. But they still test with fake errors, not real errors. For example they intercept malloc and make the first malloc call fail, run the tests, change it to second call, run the tests, ... and with two modes: only one call fails, or all calls fail after that.
They can afford to do this exhaustively because the test criterion is universal: they are testing ACIDity - the database is either in the new state or the old state. Never in between or corrupted.
Any other kind of system wouldn't be so easy to test for malloc failure, since you couldn't check for things like a successful response.
This is what I think of when people complain about the quality of code from AI agents.
I think a lot have mostly experienced working for tech companies and do not understand how different that is from working at some other kind of company (particularly something not technically sexy) doing tech (see last parenthetical).
Is GitHub a tech company? Because that's who they're criticising.
At a tech company, engineers mostly work on software that is the company’s product, and as such its functionality, reliability etc. are high company priorities, and the whole company is focused on producing and selling those products.
At a non-tech-company, that’s not the case. Which means you end up with something much closer to the movie “Office Space”, where software is just considered a necessary evil that’s worked on by what’s essentially the software development arm of an IT department, where any opportunity to cut costs or headcount is considered a good idea, where “best practices” take a back seat to “any practice that Bob can make work.”
A lot of companies these days are somewhere between the two: their main product is not software, but the company depends on a website to interact with a large number of customers: banks, insurance companies, web stores, etc. In that case, you tend to get the worst of both worlds, naturally.
^ this lol, real world systems are so much more complex and involve much more coordination than a personal pet project. It’s so difficult to figure out the source of a bug in a production system like this.
> bar of corporate code quality
It's Microsoft, if they can't afford to do comprehensive unit testing, what hope do the rest of us have?
It's less about "can't afford" and more about "don't want to spend".
> It's Microsoft
That means nothing, especially with all the layoffs.
One thing that I learned over the years is never mystify "code quality". Maybe you can say a certain team/project produces high quality code, and maybe Google's overall quality is better than my company's (considering their bar of hiring), but you want to avoid generalizing that, especially at scale.
The average software engineer at a large company ships just enough code at good enough quality to earn a salary, nothing fancy about it.
In .Net-land we have access to a lot of the underlying MS library code, and reflection/decompilation if needed.
MS has layers, there are A-teams, B-teams, and so-on. Simultaneously they are dropping both world class work and, to your point, a statistically much larger pool of just pretty good code.
A lot of that work is perfunctory and somewhat bloated/weakened by the scale of publishing and their internal tech-political requirements. Components for component I’ve seen in-house replacements for many MS projects from mid-tier devs that are better, smaller, and less brittle over time.
I’d compare it to baking: a home cook or tiny bakery have the advantages of time and focus, perpetual consistency at scale is a whole different baking challenge, one that doesn’t necessarily yield better individual loaves. It doesn’t mean WonderBread is bad at bread, just that they’re optimized for something other than maximal loaf quality.
Microsoft is very much not FAANG. They’ve been trying to eliminate dedicated QA for years and underpay contractors.
Microsoft can afford to do a lot of things, but why when you can squeeze a bit more profit out.
I have never heard anyone ever utter the words, "Microsoft makes great products and services." They basically built an empire on brand recognition and vendor lock-in. Of course, a lot of their success was due to catching lightning in a bottle a few times at the right moments.
It is miraculous how they continue to fail upwards somehow. I suppose it's because they just vampirically sustain themselves by acquiring other companies and sucking them dry until they are a husk of what they once were -- like Github, Activision/Blizzard, LinkedIn, Skype, etc.. I remember when those companies used to actually produce good products and services.
It's because they understand capitalism better than they understand software. The former is how you win capitalism, the latter is just a cost center.
Same with Oracle. Both of them are good at getting contracts and then executing them not poorly enough that it's breach of contract but not well enough that the customer stops paying. That's their business, not software.
Vendor lock in is a hell of a drug.
Would you want to go and spend billions to migrate off Microsoft/Oracle when you could just not and get to the next quarter.
Answer is, you do comprehensive unit testing irrespective of Microsoft or any other company doing it. Also, as the OP of your comment posted, you are over estimating the software quality of these big name corporate companies. It can’t be the bar against which you would want measure your own company standards.
I mean, there are many unserious engineers in corporate America.
[flagged]
Ironically wouldn't someone who solves real problems at scale actually be a 'keyboard warrior'?
Ironically, yes, they fight using a keyboard. But not in the way that the term "keyboard warrior" means.
I solve real problems at scale and the engineering practices I see on a daily basis are a clown show.
There's little to no basic understanding of networking, distributed systems, databases, etc. 99% of our engineers were hired from their college internships and never worked anywhere else. Industry hires to improve systems rarely last more than a year and it is almost never their fault.
We're in the next tier down from the biggest tech companies and what we do is hardly uncommon among our peers.
I should be shocked that 99% of engineers I deal with treat all resources as infinite bandwidth, 100% uptime, but I'm not. They NIH super hard and write tons of code for things that a docker container running nginx (or similar) would solve in 5 minutes. There's almost no useful testing and worse documentation.
Welcome to corporate life.
Clearly we have the same employer. Get off hacker news and get back to work!
I had to explain to a “senior” engineer the other day (read: a few years experience) why locating a database client in a different geo region from the server is a bad idea (especially when that client is using an ORM that likes to make lots of little calls to the server.)
I also had to argue for changing a system that was reading about 100k small files from cloud storage to use a single compressed file. There seemed to be no awareness that copying 100k files might be inefficient.
One company I was at sold its product claiming any changes you made were “near instantly published” globally. They tried to demo it as such.
The way the engineers built the update/publish operation was synchronous from their primary data center to a number of globally distributed data centers. Publish didn’t “complete” until a receiver in each data center responded with an ACK after parsing and uploading to a nearby region cloud bucket. Any failure/timeout caused the entire transaction across all data centers to retry. All of traffic was over multiple VPNs, hub and spoke style. They built this system in 2020.
They constantly complained and generated incident reports about p95/p99 latencies to the Asia regions. Latencies that were perfectly reasonable when you considered the multiple global round trips that were being made, the size/volume of objects in the publish, set of operations and speed of light.
They swore that because the client UX to publish the change to the primary data center used JavaScript async that the entire process was async. They denied repeatedly that their “all receivers ack complete to succeed” business logic was synchronous. I shit you not.
At least they aren't creating new problems at scale either.
[flagged]
Distributing across different services wouldn't be a bad idea....
I still can't help but feel a little grateful for what they do across the free side of things. I know it isn't altruism, and I know nobody needs to defend a billion dollar corporation but...
Name another service that does what they do for FREE (and no ads) at this scale. It isn't easy. Wikipedia has probably more usage, but is a simpler endeavor. (except the moderation part, that's just amazing) Open Street map? Smaller and simpler. Internet archive? Again, smaller and simpler. Linux distro mirrors? Again, smaller and simpler than whatever github is doing for free.
My only push back would be on the FREE part. I coulda bought that 5 years ago. Now I look at GitHub and go, "if the product is free, it's because Im the product" with all their co-pilot stuff.
Another fair take. I mean, hell, a lot of companies are bought literally just for their user lists (to sell new products to). I can't think of a nicer user list than a list of potential developers.
My guess is, user list would be the least. Microsoft has this thing about getting you all warm and snug because they have a solution for everything. Need an IDE? VS Code and VS proper. Got your hosting needs (azure). Literally everything. You wanna build applications around AI. We got you too, along with the VCS to keep your app in and pipelines. Hey, go ahead and send us your data so your AI application is better because you're really just calling our models. Right, and this goes even further. Data is incredibly valuable. And here we are, now paying companies to vacuum up all our data. Every last bit. And I am just seeing GitHub as another way to get people in the door to do just that.
Probably the biggest thing that has me dumbfounded about everything in the AI space and the tooling in GitHub and stuff with co-pilot. They've figured out a way to make us pay them to steal all our valuable data. And to package it up all nice for them with a bow on it and not question it.
The point is, GitHub is not free for me (we pay) yet we were also down?
IMO definiely time to segregate the free and the paid service. Otherwise those of us who need the service to stay up will have to look for alternatives.
The post mortem conclusion should've been to put usage limits on the free tier so paid users aren't killed by them.
“ Name another service that does what they do for FREE (and no ads) at this scale” and is reliable is the question
That's fair!
I'd say kids today are spoiled, but there's no doubt that this has been a rough year for github even if it is understandable circumstances.
It's been bad because it's crowded out better things. When you put something shit on the market for free you destroy most of the will for customers to pay for a thing, even if it's good, because they can just use the shit free version. This can be an intentional business strategy.
So, I lived through the Internet Explorer age so I'm familiar with that. I think it is a leap to say there are better alternatives dying to get out. There's no objective way to decide they are stifling innovation.
I'm not sure there's some solution for what you're complaining about other than the government coming in and saying they have to split github. What are you advocating for here, or is this just complaining?
Why do you think it has to be one of those?
I’m not advocating for anything (I was just spitballing) I’d be more interested in what you think would be the legal and fair thing to do.
As a general rule, there are no things that can be done about large companies that are both legal and fair.
[dead]
Are retries bad? These are the sort of reason they make me generally uncomfortable. I appreciate they might be useful in scenarios where connectivity is inherently problematic (e.g. mobile connectivity), but for a super connected and very desktoppy service I'd rather not retry much, if at all. As it obscures it when stuff has genuinely gone wrong, and this worst case scenario is tragic.
I feel like I'm mildly stupid in trying to out retries as heresy but I'm not sure.
https://brooker.co.za/blog/2022/02/28/retries.html
Seems that retries are good when the error is rare, and bad when the error is common. Typically outages have you transitioning from "everything is fine" to "nothing works", so being able detect that transition early is helpful
> Seems that retries are good when the error is rare, and bad when the error is common.
Retries are a great way to turn errors rare into common.
Retries are good, conditional on having a client-side circuit breaker that stops retries quickly when nothing is working. Otherwise, they are good in good times and bad in bad times.
Retries without (exponential) backoff and/or circuit breakers are almost universally bad, and can even prevent a service from recovering.
Source: decades of operational pain.
This is one of the scenarios where I feel we as engineers should have been more diligent in publicly writing down what techniques have worked in which scenarios and what haven't, and the AI could have gathered decades of operational knowledge and provide proper guidance to architects designing new systems. It's also true that we're witnessing unprecedented levels of scale.
Thundering herds and circuit breakers are extremely well documented and practiced in production. AI absolutely knows about this.
But knowing when to use which strategy and when a simple retry suffices is precisely the type of thing humans will remain to be better at than AI for the foreseeable future.
I think the “thundering herd” problem is pretty extensively documented
Don't forget jitter as well. Otherwise your clients will just coordinate their retries.
isn't that a bomb with a pair of scissors to cut the fuse that could break down under certain conditions?
I feel like they could also hide an issue that might get fixed if there were no retries. Is it slow or is our resource sporadically offline?
Exponential back off retries could hide a real issue, but I would estimate something like 99.99999999% of network retries are resolved within the first 2 attempts.
Not using retries is optimizing for the astronomically rare case, which is better mitigated by other means
Google “thundering herd” and you’ll understand why uncontrolled retries can be / are bad.
Quarrelsome is suggesting you should have no retries, not unlimited.
Yes I’m aware, and they’re probably suggesting that something on a higher level should instead retry, and I’m arguing that you need to coordinate the retry mechanisms and behavior between differently layers of your stack.
I totally agree with you, I think retries are overused, with the exception of operations that are known to be unreliable and can't be improved.
In my experience, errors which go away within a few seconds are quite rare, and are mainly due to flaws which are usually caught in testing.
I think a very careful cost/risk/benefit analysis should be done when adding automatic retries to things. As well as potentially causing cascading failures, it is a degraded user experience when it doesn't succeed.
As a user I would rather see an error straight away than see many seconds of spinning while something silently retries, and THEN an error.
I have the exact opposite view. Way too often, I’ll be presented with an error to the effect of, “something went wrong, please try again” and often the retry works. And I’m left wondering why this machine whose sole purpose is to automate things can’t do that for me automatically.
In particular, networks tend to be a LOT less reliable than the typical developer accounts for. And the failures are very often transient. A case I run into often is doing something with my phone while leaving the house. There’s a window where it still thinks it’s on the WiFi but it’s too far away for it to work anymore. Initiating an action in that window often produces an alert telling me to try again, and trying again a few seconds later almost always works.
If you want a retry to prevent that, it must only be done at the outermost layer. Note that what you think is the outermost layer might not be, and that accidentally deploying a retry at a non-outermost layer is much worse than no retry at all.
The Github outage was about internal clients. Phone apps are a reasonable place to say things are known to be unreliable and can't be fixed. Your IP address changes when you leave the house.
Btw, PWAs added offline capabilities to websites. I hate how the only thing that got used for was these stupid pages that look like you were able to reach the site but it's actually just saying you have no internet, like YouTube.
I’m sure some retries are helpful there too (TCP is doing them, at the very least) but yeah, different approaches for different situations. Maybe you retry but you don’t spend many seconds hoping for it to work.
I'm ok with TCP retries generally. But the assumption at L4 is that L3 can and will drop/reorder packets at random, and retrying is cheap. Also a lottt of tuning has gone into TCP already.
I don't like blind retries. It's different if the server or LB knows it's overloaded and asks clients to retry in X seconds.
Oh and this is already assuming the blind retries are randomized exponential backoff. Thought it went without saying but maybe not.
It seems like retries are sometimes best left to the human being in front of the screen. Works well enough.
You're not wrong - https://devblogs.microsoft.com/oldnewthing/20051107-20/?p=33...
I was using claude tethered via my phone, and would lose signal every now and then as we went through a tunnel. I was glad for how resilient it was its its eventual retries.
> Are retries bad?
For sh*ty providers they are great. Best of all when backsourced to the user by "Try again later."
> As it obscures it when stuff has genuinely gone wrong
Works as designed - at every level.
"... these incidents make clear that we must accelerate this work."
It feels like GitHub maybe needs to slow down? 'We must change things faster' is a wild way to start off an eight hour hard-down postmortem.
Do you think the load is going away?
The current infrastructure cannot handle the new load requirements. Either the infrastructure must change, or they must start denying users the ability to use the infrastructure.
I may have come out with a different interpretation than you did of GP's comment. I see how you got to yours. But the way I read them saying they should slow down was, maybe slow down on new features. Which would mean they could shift resources from new features to infra.
"This work" which is being "accelerated" in GP's quote is "the work underway to improve GitHub’s reliability". It is not new features.
I think he meant to say "prioritize" so that availability related work sees results sooner — "acceleration" — than it would without an increase in priority.
>We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.
Crazy.
they're still migrating to Azure?
> Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits.
I operated services at similar scale, and generally we use to put a bit of slack so that you would get an alarm when capacity goes up to 80%+ (or whatever number makes sense)
This allows to check, in the morning, after coffee, why the load balancer fleet didn't scale up automatically.
I am sure there is a good answer to why this is impractical, but it would be nice to know
I wonder how much of the traffic increase is enterprise vs. hobbyists? A 7 hour outage for enterprise customers is really, really bad and it's sad if caused by a mass of non-paying vibe coders. It's becoming absolutely obvious that the unlimited free tier needs to go.
This is my suspicion, all though I have no evidence, just anecdotes. I make a handful of commits a day. I write code, review it, test it, commit, and then push.
We have some marketing folks that have gotten into vibe coding stuff for their personal use. First let me say: good for them and I'm glad they're experimenting with new ideas and tools. The side effect of that is that looking at their repos, they're having Claude go whole hog and make upwards of hundreds of commits a day, all with things that they haven't taken a look at.
I don't think I can say this is wrong of them, because their tools encourage that and they shouldn't have to consider their impact on an enterprise service, but I wonder if this trend is similar in other places.
I appreciate the unprecedented load GitHub is currently experiencing, but it's not just the (admittedly extreme) load of commits/pushes that is to blame.
Their Copilot cloud agent offering is suffering with a case of some of the worst corporate ADHD I've seen. We built a cloud agentic development pipeline on it, and it seems like almost every other week they silently change something with zero public announcement that creates real disruption for our team.
Note: that's not bugs in the Copilot platform like the article discusses. That's real, breaking changes to the platform that clearly aren't being tested/reviewed before being pushed to prod, with zero public announcement or documentation.
Support is useless – we're paying customers in the 4-5 figures and our tickets go unanswered.
I love(d) GitHub, but I do think they've lost enough public trust at this point that their time is ticking. With talk of new VCSes designed specifically for agents, I do believe it is just a matter of time. Which pains me somewhat to say.
Has to be one of the most vague outage summaries of the year
The root cause analysis is separate, but linked to from the blog post: https://www.githubstatus.com/incidents/zkxwbgr0cnmx
It is specific and technical.
Yeah... it has the vagueness and awkward staccato of heavily edited Claudish.
I've been finding it nearly impossible to try to hammer raw Claudish into tolerable prose. I usually have to re-write most of the text by hand if I want it slop-free.
P.S. Hi Claude! I know you might be reading this :) Don't take it personally. Let your trainers know that you need some help with this.
This seems like a pretty straightforward and easily winnable situation for GitHub. The demand for their services just doubled, apparently. They have no real competitor operating at the scale they’re at. They have pretty substantial network effects.
They are under no obligation to continue functioning as a bottomless free repository for text file hosting, especially now that text file creation has multiplied exponentially. They could make a few almost purely commercial changes and solve this without any major re-engineering while maintaining their status as the go-to public / open source code hosting platform.
1. Immediately increase pricing of all enterprise licenses and add super-committer overage fees.
2. Rate limit or cap commit size / frequency for public accounts.
Their service is more valuable than ever and switching is much harder if people have automation built up on their platform. Now is the time to cash in their chips.
And the positive externality of increasing commit cost would be forcing people to have some semblance of restraint for the AI content they generate.
Why are we even committing AI generated code and uploading it to GitHub? I've heard we don't need to read the code anymore. Why preserve a detailed version history if AI has it all handled? Why even share code if we can all just have AI write whatever software we need for ourselves? GitHub feels like a dead end of AI is really headed to where we believe it's headed
But GitHub were one of the first and strongest to advocate for using AI, so excessively in fact that it turned people away from the platform. At every turn on the website it suggested to use Copilot, or edit it right now, and commit right now, and do this and that right now(!). The platform effectively begs you to use the free GitHub Actions, too. Not sure what the business model is here.
There are a lot of ways to avoid exponentially more commits, issues, and PRs breaking your backend down, and begging every visitor and user to please use AI to write 40x more code that needs 40x more fixes is not one of them.
Is it me or is all of this essentially "we don't want to show the user anything at all when something breaks?"
And what makes this funny (to me) is that this is a website for developers. I would think that of all the audiences you would target, developers would mind seeing the platform display error messages when things break the least.
Centralized source code hosting is going to end up looking like the three credit bureaus in terms of security. It's only a matter of time before the first big hack, when everyone shrugs and says, "Oh well, everyone's source code leaked lol too big to fail."
Given the jailbreaking behavior of models in Anthropic and OpenAI, as well as Microsoft allowing Copilot access to client data, we should assume our source code on Github is leaked or leakable anyway.
This article seems to say that there is "no excuse" for these issues but look at all these things we changed and are changing. It doesn't really feel transparent and it feels like they aren't really taking true ownership on what has happened.
The trend doesn't seem sustainable.
As I mentioned before (https://news.ycombinator.com/item?id=49333107), they can mitigate these issues with limits, even for failure cascades. There should've been an all-hands-on-deck feature freeze 6 months ago to implement the limits needed. That clearly didn't happen.
I think it's because their leadership actually doesn't care that it goes down. A weekly outage is now an accepted cost of continuing to allow unlimited free access with infrastructure that cannot possibly handle the load. As a result, everyone is looking at their GitHub Enterprise bills and cost of stopped work, calculating how much they'd save by self-hosting.
> Copilot services took longer. Errors in those services triggered a client-side retry loop that increased traffic during recovery.
Let's pretend that the scale traffic is with the number of commit/pr and not self-inflicted with all the copilot eye candy features that were vibe-coded-added to GitHub.
In addition they say that they will continue their migration to azure and that azure is supporting their actions run. But GitHub actions is one of the things that was the most constantly broken without multiple outages recently. So I have the feeling that it proves the point that part of the stability issues is also due to their forced usage of azure.
Why not identify the lunatic top 1% of free user you know are just abusing the hell out of the system and put severe rate limits across the board for those organizations/accounts?
Why let your entire platform suffer?
Yeah you could probably have a very high free limit that works for 99% of people
I don't know, this blog post and the lasts read like organisational and human failures to me. Their system doesn't degrade, they even have infinite retries and no meaningful quotas anywhere I can see (but I'm not a heavy github user).
I feel like this becomes a lesson on how NOT to design and operate a SaaS.
I wonder what the ratio of repositories to physical machines is these days. I'd also be curious to see this as change over time.
I have a hard time with the premise that a mere doubling of git ops would be especially crippling for any particular repository. GitHub runs like ass because it's oversubscribed by a huge factor. Not because git is inherently constraining at scale.
Sorry to suggest this but if they charged everyone say $1/mo. it would absolutely help the massive surge from AI coding they seem to have had.
I don't like paying for free stuff but gh certainly worth it.
You are always paying one way or another, I prefer to pay in dollars and not frustration, attention, or privacy.
Not an option. Companies are happy to sell you products at premium prices, take your money, and then still collect, mine, insecurely store, and sell all your private data.
I am surprised they didn't rollback their change from 2019 when they included up to 5 private repos in free plan. Before that you had to pay to keep your code from being publicly visible.
Unfortunately time and time again, the overwhelming majority of people show that they would rather deal with an unreliable scummy company for "free" than pay $1/mo for a reliable service which treats them with respect :(
OK that "treats them with respect" is certainly an important aspect though. It does not seem like GH is especially great on that one, including listening to its users.
> service which treats them with respect
Which mythical tech company - free or paid - does this?
Surely that's the point. Shed users.
Microsoft are doing many things to help the AI coding surge, I'm not sure if this'd be a cost effective thing towards it.
Pay per issue and commit. Buy 1,000 commit credits at a time.
Might force people to review their slop before pushing it.
I would support time based quotas or limits per tier (free, solo, team, etc)
But I'm not going to pay per commit over my monthly plan
I wasn’t being completely serious with my suggestion, but I was thinking of a totally PAYG model. Monthly plans could come with credits, like they already do for Actions.
"The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR " - silly question but github uses github for their own PRs and deploy right? Do they have a special dedicated system just for them so they can fix github with a code change even if the rest of us can't?
Not sure, but they could use the self-hosted enterprise version of GitHub for their use case.
After touting 1 billion commits over 2025 at universe last year… they are now handling 3 billion per month jeez. I’ll give them a little more grace after all…
I don't know where else to ask this but it's killing me. Does anyone know what the hell that GitHub physical CD thing was about? Did anyone in the world get theirs?
That was a fun thing that the marketing and devrel team are still working on, turns out burning repos on the discs was a harder problem than anticipated (lol) but we'll get them out eventually, promise!
Haven't they been migrating to Azure for a few years? How is it still only 58% done... Microslop needs to lay off the focus on AI features and get it done
Linux didn't (yet) kill Microsoft. Microsoft absorbed that shot. Then the Git arrow went straight to cold black heart of Microsoft. The next few months will determine if they survive it. If they do, what will we see from the third draw out of Linus' quiver?
Linux had nothing to do with the downfall of Microsoft. Any perceived dominance of Linux is not because Linux is getting better, but because Microsoft is getting worse.
The 'growth in completed actions runs' graph is interesting. I assume the periodic drops are weekends, so intuitively the floor of those drops corresponds more with hobby/personal projects than people at work. It looks like there's a sharp uptick specifically in that floor since July ish.
Would it be okay to suspect the recent upsurge in AI agent usage as a possible main cause of this issue?
This is a really good post.
I said in another thread that they can't blame increased demand for these outages, but the demand growth is genuinely insane for a company already operating at huge scale.
I guess we'll have to wait and see if they deliver now, but it seems like they're taking it seriously at least.
Is git still the best VCS today? I ask because it seems so much effort in the industry has been invested in making git scale, like Cursor's Origin, or the stories in the past with Facebook's monorepo, but they all seem like bandaids to its intended design.
> Both incidents were capacity failures at their core. We failed to scale critical components before demand exceeded their capacity.
I'm missing in these descriptions the most obvious approach: Resilience. Shedding load so that you can keep services up even though capacity is too low. If you flip over as soon as load exceeds what you can handle then this problem will never go away, unless you always have insane overprovisioning of resources which is uneconomical. There will always be spikes. You need to plan to handle them, no matter how high.
> we have focused on three priorities: adding capacity, improving efficiency, and removing architectural bottlenecks.
Sorry, but again, that is not good enough. They should ask themselves why they are expecting that trying the same medicine as last time will prevent next time. It won't.
With that mindset I'm not surprised this happened and it will surely happen again.
Edit: In more concrete terms. If you 2x your capacity and in a week you face a burst 2x of what happened last time, you are back in the same seat. If you improve efficiency by 2x, same thing. And after a bottleneck is before a bottleneck. There will always be a bottleneck. The key is to be able to handle a bottleneck. Removing one just pushes the issue to the next one.
Your architecture must be such that your whole system should be able to run on a raspberry pi. Most client requests would be dropped, but those that make it through will be served. If your architecture serves 0% because it crashes when load is 10% over capacity, then capacity increases or efficiency increases or bottleneck removal are not going to prevent the next outage.
I recently received a PR fully automated by Claude for an 8 years old repo. The bug is legit and the scope it affects is larger than what that PR addressed, but I no longer care too much about that legacy code anyway, so I also let Claude run free for the first time in my life, from handling that PR to fixing all related bugs.
I walked away for half an hour and back, found Claude opened and merged 9 more PRs and added a comprehensive CI for testing for all platforms. It will likely take me months to reach this level of output, but only half an hour for a capable agent.
No wonder why GitHub is down all the time.
With all the outages at GitHub there has to be someone willing to unseat them as the social git repo… how bad does it have to get before folks go elsewhere? Bitbucket and gitlab exist but are pawns compared to a king no?
Cant they just put ai agents at optimizing their slow internal paths.
I fear to ask, how archive.org keeps up to catch all those events for archiving...
AWS CloudWatch has an option to show the trend and what it will be like after x-period.
Doesn't Azure have such options so that engineers can predict to scale better? Seems like engineers are not ready for this per postmortem
The trend line does not tell you what will actually happen at scale, even if you think you're perfectly prepared for the next 10% or 20% growth. As Mike Tyson put it, "everyone has a plan until they get punched in the face".
The problem is there are a class of problems that only appear after you go over the tip of what your system can handle, which are very difficult to predict or model.
A lot of these projects and commits would benefit a ton from proper decentralisation. What functionality of GitHub are you *actually* using?
Afaik the other main issue with github downtime is the choice they made a while back coming back to bite them
5X as much code in <2 years. What is all that software?
AI slop
Related recently:
GitHub has alternatives, but no replacement
https://news.ycombinator.com/item?id=49135365
Why developers are ditching GitHub for Codeberg and self-hosting alternatives
https://news.ycombinator.com/item?id=48842611
and new entry:
Cursor Origin Code Hosting
https://news.ycombinator.com/item?id=49334209
We need to have a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.
GitLab was close last I remember but there was some sort of enterprise tier when I tried hosting stuff on a local server years ago. I want true FLOSS, not another SaaS equivalent of the coke dealer giving clients the good uncut stuff when they're just starting out only to sell crap when they're addicted.
https://forgejo.org/ promises to be this, have only lightly used it on https://codeberg.org/ but it seems nice?
Forgejo is pretty neat. Their CI story is sad because it's based on act and it lacks features like GH Apps so service accounts are not possible (using a user account as service is barf).
I self-host it and its pretty easy to have uptime higher than github when you have 3 users total.
Absolutely do not recommend using GitLab.
"Forgejo is a self-hosted lightweight software forge"
That says absolutely nothing. The "What is Forgejo?" question is unanswered and instead you get a lot of words about their values, their inclusivity, etc. And the next thing in the docs is how to install it. It's ridiculous.
I still don't know what it is or what it does.
“Software forge” is an established[1] term by this point: a piece of software providing a full set of features for collaborative development, usually at least code hosting with a web interface, tickets, and webpage hosting, and these days often also code review and integrated CI or at least the possibility of integrating CI. It’s admittedly squishy but in the same way “IDE” is squishy, which is to say it’s still well-defined enough to be useful.
[1] https://en.wikipedia.org/wiki/Forge_(software)
Forgejo > The name of the software
self-hosted > You install it on your server
lightweight > It does not consume a lot of resources (cpu, disk, ram)
software forge > offers tools that help with creating software collaboratively (repository hosting, change request management, wiki for docs,…)
> a package of FLOSsoftware that you could run on the cloud of your choice that offers most of what GitHub does (niceties on top of Git) without the centralization.
You're in luck, GP comment described it for you.
I'm particularly fond of the concepts within git-appraise, gits back to the federated foundations
https://github.com/google/git-appraise (unmaintained)
Have you looked into Forgejo?
Maybe Github (and especially things like Actions) just need to become more expensive?
Not again but..
the increase in traffic over the last few years is way bigger than I thought
Github you can only post you are doing stuff about outages if its actually effective.
The vibes are off.
“Required several coordinated actions” is the key moment for reflection.
Interview question. How would you handle the growing traffic needs and traffic spikes. I'm curious whether any existing architectural diagram of theirs would highlight a potential failure post-mordem.
Is it a interview question to the MBAs cutting costs at all cost?
Centralized decentralized code repos. It feels like an oxymoron.
The distinction between running out of capacity and collapsing when you run out of capacity is exactly right. Every distributed system hits limits, the question is what happens next.
the retry loop that made the outage worse is just the internet being extra helpful. every client decided the best way to help was to ask again, louder.
What would happen if github was down for like a week?
Great read - I'm glad they realize there's work ahead but what I'm missing is: * Paid customers: we know you pay us often a ton of money, and we burn your month on actions during these outages - we'll refund you for the days we spent your money and gave you no value. * Paid customer: We know you put your trust in us, so we'll ensure we have a separate pool of capacity to ensure we can keep that trust. * Paid customer: we'll proactively refund you when we miss our SLA.
What I read from this is: * Scaling is hard, we don't have enough capacity * We give away a shitton of compute for free * I have to talk about Azure not being a steaming pile of poop, otherwise my bonus will get tweaked downward in the next comp cycle.
Notice there's nothing about paid customers, I'll add in what they are missing:
Paid customers: Go F*ck yourself, you don't pays us enough to be an interesting line item compared to windows server.
This. I own a small company with 5 people. I pay Github $250/m. I'm sorry but the narrative of, "look at this burden we have, it's hard to take care of all of this code!" is pretty insulting when I'm paying $50 per person per month to host code and run CI pipelines. If they do not want my money, I'll find a company who does.
[dead]
That is amazing
Is Azure the right choice?
And another outage. [0] Looking forward to the subsequent post-mortem on that one.
You might want to not go all in on GitHub anymore since it is very unstable to use. A self-hosted instance would have a far better uptime than GitHub over the years.
6 years ahead [1] on not going all in an centralizing everything on GitHub.
[0] https://www.githubstatus.com/incidents/bhbcjn4n3jzp
[1] https://news.ycombinator.com/item?id=22867803
There are no EU instances of GitHub, it's all in the US only.
With all the software being written on github you'd think we were going though a software rennasance. Where are the results? Is it really just all slop?
I can't speak for all of it, obviously, I don't have time to try much of it, but I see tons of amazing new software in my various feeds pretty much daily.
I'm a bit weary of seeing comments like this, but I want to believe you, so help me learn? What are some examples of the amazing software you've seen recently?
Sure. Off the of my head the first three I remember from the last day or so: Opacity (who seem to be reinventing vector drawing), Pixel Flight Simulator for pico8 (with dynamic cloud cover, day/night cycle, and voice over from air traffic control), Jet (a high speed 3D library for ESP32/embedded, open source and commercially licensable).
I'd say I saw these because I follow certain creators and/or high signal reposters in the fields of software and games.
And, if you'll forgive me, I myself am working on a totally new way to make games, called Jinks (multi-platform Web, Mac, Linux, Windows, even back to Wii and Dreamcast; games can be introspected, edited, queried at runtime; one game released so far to prove it; work in progress)
What are the top 3 you remember? You say you can’t speak for all of it… can you speak for any of it?
Please see my answer to an earlier comment: https://news.ycombinator.com/item?id=49397304
Exponential growth. No company could handle that without some issues. Good luck to them. And for those who cannot tolerate this, there are many self hosted options.
I wish I had say in our git forge decisions at work, but I don't and I can either tolerate this or quit my job. So I will continue to disparage one of the world's biggest tech companies not being able to manage GitHub properly.
I have setup a gitea instance on my gitea server which I think is good for me and GitHub both. For one, I dont have to worry about GitHub service outages, and GitHub gets to be free from my toy (and mostly AI slop) projects that no one else will ever read/use/participate in :-)
"We are committed to fixing these problems, as long as it doesn't involve buying things other than AI computers, hiring humans, or using non-Microsoft products."
Calling Azure the solution to this problem when it is in fact the source of most of these problems is just fantastic doublespeak.
Github is ripe for disruption and I hope it is disrupted soon.
> We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.
And from the RCA [1]:
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic.
[1]: https://www.githubstatus.com/incidents/zkxwbgr0cnmx
"While accelerating our migration to Azure," meaning, they will only solve problems if it helps them also use Azure more.
It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage, unless they have no reporting or their tooling is completely incompetent. If things can fall apart so easily, throwing more capacity at the problem won't fix it.
You’re torturing your own logic to make Azure the villain here. And it also sounds like you lack experience with capacity exhaustion. Things fail slowly, then suddenly.
[flagged]
Baseless accusation made from a position of zero information.
Opinion based on stated facts.
Please share the information you have which contradicts the conclusions I have drawn from Github's statement.
(And we know they're liars. They report very few of the actual incidents they have; see for example https://mrshu.github.io/github-statuses/)
I don’t owe you anything, much less a separately sourced counterargument. You openly admit that your opinion is not based not on GitHub’s proffered statements but on your self-admitted assumption that GitHub is actively lying in an attempt to cover up an Azure-related root cause.
I’m firmly in the camp of “something stinks at GitHub” but
> It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage
In my experience, there are hard thresholds that get passed that expose hidden bottlenecks like this. A previous system I worked on we had absolutely loads of headroom by all of our measured metrics, but one day we filled a cache because the value hadn’t been tweaked in recent memory. Plenty of space on disk and in memory, but all of a sudden we went from a very high cache hit rate to a very low cache hit rate, and everything ground to a halt.
This absolutely can happen in large systems. If some part of the system is at capacity, then slightly increasing the load can cause it to fall behind and start accumulating a backlog.
These backlogs can cause clients to make more retries, exacerbating the problem. Potentially further cascading through the system.
I believe their point is that "system is at capacity" is something they ought to start fixing before the capacity is exceeded
Sure. But you might not even be realizing that something is just at the cusp if the load is spiky enough.
The art of large system design is to identify and avoid these kinds of chokepoints. And when something happens, propagate the "backpressure" up the stack to avoid queuing.
AWS got a fair share of similar outages, so the newer SDKs now try to not exacerbate these kinds of issues: https://docs.aws.amazon.com/sdkref/latest/guide/feature-retr...
The original AWS EBS outage is probably the canonical example: https://aws.amazon.com/message/65648/
But then people like OP will claim that the capacity concerns are a lie manufactured to support an unjustified move to Azure.
There's always a cliff, this part is fine. You sometimes know the cliff but often do not.
>"While accelerating our migration to Azure," meaning, they will only solve problems if it helps them also use Azure more.
Look, I hate Microslop as much as anyone but you'd have to purposely misinterpret TFA in order to arrive at this interpretation. C'mon.
Ok, but there's no universe where a major Microsoft-owned property is not being forced to run on Azure. Just like AWS pushing to get off Oracle back in the day. It would be career-destroying to suggest otherwise regardless of technical merit (and tbf, no infrastructure is bulletproof, unless you want to port GitHub to z/OS on mainframe)
LinkedIn gave up after four years of trying: https://www.cnbc.com/amp/2023/12/14/linkedin-shelved-plan-to...
Azure just has very poor performance and reliability characteristics. It’s a particularly bad migration target for a colo-based company that mainly runs on owned hardware (such as GitHub or LinkedIn). Requires much larger architecture changes than (say) a company coming from AWS.
I'm betting on Tangled and Codeberg. Tangled has a better press and in general is a dark horse, Codeberg has the "brand" and some network effects from projects that moved to there. (famously, Zig.) I heard that Sourcehut is having a moment as well, and I love the idea of email-based workflow and not having to have an account to contribute to someone's project hosted there, but I'm not maintaining anything worthwhile paying the $4/mo sub.
What if it was $2?
that is more manageable but c'mon I can't even keep Google One 100GB up on a consistent basis, that's how poor I am. self-hosting would be a far better option because apparently I find enough people to provide free Hetzner VPSes and stuff as long as I can sell this as mutually beneficial.
for context, I would GLADLY move there my Neovim plugin. all it does is brings the current jj message into your editor and lets you integrate it with a status bar (or anything in nvim, really). that would be a decent measure against drive-by slop contributions, and I'd accept contribs over private github mirror from those who I know but can't bother setting up git mail
EDIT: TIL that one can host SourceHut themselves. discoverability may still be a problem (sr.ht just ranks higher in search engines) but 1) fixable with github mirror that points to sourcehut instance as a canonical development platform, 2) it's moderately easy to sync contributions between tangled and sourcehut, so tangled is also an option
EDIT 2: the email part would be PITA, so $4/mo is attractive on that background
Oracle has a free tier that you can self host a git server on, if that's what you're looking for.
I spent a month trying to set this thing up, to no avail
> Github is ripe for disruption and I hope it is disrupted soon.
It's an expensive, low revenue generating site.
There are, and have always been, competitors, including "host it all yourself" solutions, but nothing has really stuck.
How is it "ripe" for disruption?
They had $1b revenue in 2023 and now probably more than $2b in revenue... do you have cost figures showing what their expenses are?
> We have since added more than 3 million CPU cores, 120 petabytes of high-speed storage, and significant network capacity. We installed as much hardware as available power allowed in our existing data centers while accelerating our migration to Azure.
That can't be cheap.
A server box now has around 256 CPU cores. So that's about 12000 servers. If each one is $10k that's $120 million. Not a lot compared to Github's income.
$10,000 is only enough for CPU cost (2x AMD EPYC 9754). The rest of the servers (mainboard, RAM, storage, power supply, network card, rack, cooling, etc.) can easily triple or quadruple the cost.
Yup. Still, it's less than their annual income by several times.
Does that $10k server not need RAM?
If parent commenter is off by an order of magnitude in their costs, their point still stands.
$1.2 billion of costs on revenue of $1-$2b certainly seems like a big concern.
They do not throw away the servers every year..
But at this rate of growth, sounds like they'll need to buy around that amount every year just to keep up with current (not even taking into account future) growth.
Microsoft don't release the costs as you know, but
Compute and Storage for Free Tiers: Hosting code for over 150 million developers and processing over 2 billion GitHub Actions (CI/CD) workflows a month requires astronomical server power and data storage. The "Free" tier is a massive cost sink that Microsoft treats as a loss-leader marketing expense
Let me know when you understand how that's not free.
The Azure horror stories are on this blog post and HN discussion.
https://news.ycombinator.com/item?id=47616242
If you're a big company, you can afford having one engineer spend one or two days per year to maintain your self-hosted GitLab or Forgejo. On top of better reliability than GitHub, you'll get the additional bonus that your source code won't accidentally leak through being in Copilot's training set.
If you're a hobbyist, Codeberg is great, has a nice community and automatically shields you from slop contributions.
Speaking from experience, it cost mW about a week or two per year to maintain GitLab for the startup I worked at.
My personal GitLab on the other hand really does take only a day or two per year.
That said, a week or two per year is just what it costs to maintain any one thing period. I spent about that much time maintaining PCs in the office, or my personal proxmox setup. It's not onerous at all.
GitLab is super bloated and a little sucky to admin, but it's not too bad all things considered. I'm admin in my new job's GitHub org and it sucks a whole lot more to maintain.
The issue with these systems is that they lack Github's sophistication for issue tracking, knowledge transfer, and automation. I think Gitlab is a mature product in its own space and unlikey to change, for instance, at this point.
Codeberg also has the issue of having a political stance which means they will not accept just anyone's use of the platform. That is absolutely their right and I have no issue with it, but it's unattractive to me - as someone who agrees with most of their current politics - because the day they decide they don't like me, I'm screwed.
I never found GitHub's systems for issue tracking to be all that great. Cross-repository issues and development plans are hard to track within a git host. I've always used an external panning and issue tracking tool, mostly Linear, and it works really well. GitLab's Linear integration is excellent, FWIW.
I've actually worked with a couple of companies who do use GitHub for their code, and they all use Linear in addition to GitHub.
I understand the concern you're talking about wrt. Codeberg, but I wouldn't view it as a significantly bigger risk than anything else. Any platform can suddenly decide that your project is against ToS (GitHub will absolutely not accept just anyone's use of their platform either) and Codeberg introducing some rules recently doesn't, in my mind, drastically increase the risk of a dramatic ToS change in the future. But we all have to make our own risk evaluations and I won't judge yours. Luckily, moving between Git hosts isn't that difficult; setting up CI again and losing merge request history does suck but it's not the end of the world, unlike something like, say, losing your AWS/GCP/whatever account.
"sophistication" seems like a strange way to describe GitHub to me - I've found in every individual aspect (code browsing, issue tracking, code review, package management, etc), it's the worst out of all the systems I use regularly... But it's good _enough_ for most people, and it has all those features in one place, which is more convenient than wrangling 10-15 high quality but disconnected systems
Gitlab has the benefit of having very little traffic, both free and paid. Their limits are still way above the current usage so less likely to be an issue
Worth mentioning GitLab's paid enterprise offering are more expensive than GitHub's, on a per-seat basis.
Lots of companies moved because it was cheap, but it's not anymore. Ironic that companies might choose to migrate to them now for stability, rather than price.
GitLab has its own problems. It might be somewhat more available than GitHub lately, but it too is not immune to incidents, particularly involving workflow dispatch and managed runner availability.
Plus the UX isn’t nearly as good. Sometimes it feels downright clunky in comparison.
Dude no one uses github tracking for anything serious, come on. The only thing github has over gitea/forgejo is discussions. That's the only real social "innovation" github has contributed to open source development and it's just a shitty tacked on forum.
Also for accuracy, Codeberg has a pro-human and anti-corporation stance. Both of which are definitely en vogue at the moment.
Much better than GitHub's pro slop sentiment, which is doing nothing but destroy their reputation.
> We have made progress, but these incidents make clear that we must accelerate this work
Pretty sure this line appears in every one of these.
nice
github controls the productivity of a large number of very large tech companies. When there is an outage like this they are basically shutting down a significant number of factories for hours at a time. This would be as if during the hayday of detroit they just turned off the power grid at a time randomly at least once a week for hours. It's unacceptable. The amount of productivity lost is staggering.
We should be building tools that help us all move off of github as soon as possible. The amount of action code that will need to be rewritten is daunting.
someone needs to build a open source aws
1.5 billion commits of worthless slop
This seems really bad, to be honest. I think we may be fucked. There's just no way all these lines of code are doing anything useful. We're now just burning stuff in desperation and confusion.
Use less AI slop coding
Almost 8 hours of downtime across all core workflows, and the word "sorry" or "apologize" appears nowhere in this post.
"If you were trying to ship software that day, we let you down" is classic corporate non-apology speak.
I’m done.
i am confident that if "sorry" appeared, someone would make a comment about "hollow apologies" or similar.
thats what i liked about it. its fact and action oriented. what does a "sorry" buy you that the "we let you down" doesn't.
It's funny, I bet you could take any software dev and blind AB test a page written by a corporate manager and a page written by an engineer.
Here's something an engineer writes, loaded with facts:
"I got to the office and we had a huge panic going on, I immediately called our IT in US-2West and they reported on cascading box failures, I checked our load balancer via remote admin and indeed it was failing to. I called my IT managers and learned we had hard resetting in progress for the past 20 minutes with minimal impact on recovery."
Totally missing from the article.
im sorry u feel this way
Bye Felicia
[flagged]
[dead]
[dead]
[dead]
[dead]
[dead]
Just add a queue. Now the outage is eventually consistent. /s
[flagged]
[flagged]
> the entitled freeloaders
Now remind me again, who trained a coding-assistant without consent on those "freeloaders" code and sold it for profit?
Using a corporation's free offerings isn't freeloading. Microsoft wants people to put their code on GitHub. They want GitHub to be the place where source code is hosted, it is incredibly valuable. Saying "GitHub is sucking and I might leave" is information that Microsoft wants to know if they want to preserve GitHub's dominance.
Of course some people take it too far. Of course there are reasons that the outages are occurring. But Microsoft wants GitHub to be a core, reliable pillar of the software world. Nobody's making them do that, they do it because it's good for them.
It seems that paid users are equally impacted as free users.
It sounds a little bit unfair to me.
True but paid users are also impacted.
We're still staying on Github at work, but have had backup self hosted git repos as a break glass option when Github is completely broken and leveraged this several times now.
Whos fault is the massive load increase, or their inability to navigate it?
Did I iject copilot into every part of github? In fact not only did I not insert it, I have never even used it.
Did I move their infrastructure to Azure?
Did I sell them to MS?
Did I set all the directives and priorities that MS has set on them like telling everyone they must use openai for everything, and then telling them they must stop doing that and user their own ai instead?
The outage is not due to a natural disaster that no one could anticipate and no one had any input on creating the conditions. They keep the free tier because THEY want what THEY get from the free tier. They could easily have a $1 tier and various totally sensible throttle limits on various services and apis that would have avoided all this, but that would not get them the 100% user coverage that they want. So THEY choose to provide free, swiss cheese service.
It's not some unreasobable burden they labor under that anyone else should be understanding and forgiving about.
This is exactly the entitled attitude I'm talking about. Nobody forced you to use Github for free. Don't like it, use something else. Beggars cannot be choosers
I don't have to use a thing, for free or otherwise, to observe it's nature, and nowhere in that observation is there any claim to any entitlement.
It's curious why anyone outside of GH or MS leadership would even care to try to make such a claim.
"Most people use GitHub and features for free"
Do you have a source for that factoid? (I suspect the vast majority of Github resource usage is paid. And we are upset.)
That sounds unusual for a free platform (not a limited trial but an actual free tier). Isn't it usually the case that only some small percentage can be convinced to pay?
They're arguing that most usage of resources would be by companies, who are presumably paying.
I don't think my employer pays for our use of Github. Edit: but, then, perhaps that's why my view is tainted. Point taken!
Entitlement? Please. It's not like Github is a charity that operates on kindness and goodwill.
It's a service that is owned and operated by Microsoft Corporation, and we're the product of it.
LOL. If it dies, it dies.
People take the weirdest rhetorical hostages.
Agreed - up-time problems have been going on with Github for a while. They need to sort out their problems.
I mean, a lot of us have paid GitHub a lot of money for CI on private repos. And when GitHub themselves encourages the insane behavior of vibe coders and agents instead of just charging or rate limiting access of bots, it's hard to give them sympathy.
Meh, just needs better QoS. Let the free tier shoulder the outages.
well written
Ctrl+F "Sorry"
No results.
Cool
i wanna vibecode a replacement for git and call it jit
No sorry we messed up your work?
> What we have done and what comes next
"You've seen what we've done. The August 21st outage comes next. See you then!"
Github down, no hard drives available, no memory available, thanks AI!
Seems like we are headed for Tech Gridlock.
This stuff is good! This is what a booming economy looks like. There are people out there competing with you for resources because they have cool ideas they want to implement.
Or at least they asked the AI to come up with cool ideas, which is even more interesting. It's exciting watching the world transition away from humanity being in the driver's seat!
Layoffs by the 10s of thousands, food prices out of control, people barely able to afford gas. At least some tech bros can launch their 50th B2B SaaS. I didn't realize a booming economy sucked so much.
The exciting thing about AI is precisely that it'll let software move beyond Yet Another B2B SaaS and into doing useful things in the real world. I regularly ride in driverless cars! That was the stuff of science fiction when I was a kid.
If you're worried about food prices, you should be happy that robots will make agriculture less labor-intensive and bring prices down.
probably worth asking when those prices will come down.
Automation has been making food steadily cheaper for 300+ years, and continues to do so. Sometimes the price of oil goes up so much that fuel and fertilizer price increases negate the wins from automation, but automation is still generating wins and will only accelerate as robots get smarter.
What they can implement is to slowdown the commit rate, rate limt or just queue-up messages not to overburden their downstream service.
I don't think GH has any of those, but just keep scaling, but that scaling failed.
Just bad architectural decisions from the postmortem.
--
It will only get worse due to AIs spawning massive commits, and they don't have unlimited cloud resource.
They can scale but not scalable in terms of effort, resources, and $
How would any of what you're saying help with this?
> The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery.
load balancer failure? rate limit woudl address concurrency limits? throttle or queue up messages. auto-scale failed cause was misconfiguration policy, which i admit cannot be handled by my suggestions. The cascade? it's downstream service degradation, which I mentione should have had been prevented with queues. One of the jobs that queues/kafka solve is to prevent these downstream outages.
If your LB is down, you're just kind of screwed. You can't enqueue things if requests aren't getting through at all. Same deal with authn/authz issues, which they also had. If you can't answer the question "is this message allowed to be added to the queue" you can't enqueue stuff.
GitHub does use queueing for all kinds of stuff internally, though, because they're not morons.
YOu are making an assumption on which LB is down, and so am I.
Where is LB placed? Is that hte LB in front of API GW? or behind GW? GW can have rate limiter or cloudflare FW can handle the rate limiting. The doc does not show which part of LB is down so it's all guessing at this point so i won't go forward with this.
They should rewrite their Ruby code to a performant language.
I think it should be noted that the CTO of GitHub doesn't use his own product. No commits since January 2024: https://github.com/v-fedorov-gh
No side projects? Nothing? Just seems odd.
Maybe he's too busy with leading the engineering side of the company to write code these days?
If even the CEO is vibe coding[1], then the CTO has zero excuse.
[1] - https://www.youtube.com/watch?v=SEZADIErqyw
Probably managing and mentoring. I has a boss that wanted to code and be CTO. Just horrible.
Can you imagine the flack he'd take if it turns out he'd been moonlighting whilst the GitHub ship is sailing though cat 5 with both the mast and the ship whore on fire?
yeah grinding on side projects, while everything is in flames.
this is fine.
Sarcasm? Someone as important as CTO of GitHub is the last person I'd expect to have "side projects". I'm sure his job is project enough.
Reading this port-mortem / plan shocks me, this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started. It seems the solution has been capacity, capacity rather than architectural or data changes.
> Our next milestone is an architecture that scales read capacity linearly with the number of readers, enabling unlimited read operations
How do you not have read-replicas / read caches at this scale yet? Which is what I am reading from this statement. You can of course get really far with sharding and whatnot. But at some point it might become worth it to engineer your data into a model that scales better.
> this doesn't look like a service that has been serving high-throughput services for more than a decade. In fact it is almost like they've barely started.
Well that's because in comparison to the absolute flood of traffic brought on by AI, they really haven't been operating on this scale before.
Here's Vladimir Fedorov's GitHub contribution graph, as linked to as the author of this post:
https://imgur.com/a/zIbT0Gi
It shows zero contributions in the past year, on this account. This is a huge, huge red flag.
I strongly disagree. GitHub, a year ago, acknowledged the fundamental problems and began work on them. We all agree with the diagnosis and strategy: stop building new things, bring stability. Why would whether the CTO codes have any bearing on the correctness of this strategy? GitHub’s problem isn’t that leadership don’t understand the product, or that they don’t know what they should be doing, it’s that they’re battling unprecedented demand. If it was a disconnect between users and leadership on what matters, sure, a CTO who doesn’t use the product would be notable, but that isn’t the problem. And that’s all assuming he doesn’t actually use the product, maybe his privacy settings hide private commits.
Interesting - There's a couple of things that I disagree with here, but I do think this resonates: "We all agree with the diagnosis and strategy: stop building new things, bring stability."
Where I have a problem with the positioning of their GitHub profile is, he's the CTO of GitHub, arguably the defacto standard for open-source version control systems. His GitHub profile is linked to as the author for the post, and his GitHub profile simply tells me: "this guy doesn't code."
I don't care if this guy doesn't work on GitHub itself, I hardly would expect that, but IMO, any CTO of a company like GitHub should eat, breathe, and sleep code. He might, but his profile, which is being published as if it means anything, tells me he doesn't.
Mine would look the same if you didn't have access to the private repositories I contribute to at work.
Don't contributions to private repositories simply show as: "N contributions in private repositories"?
Here's me: https://github.com/ryanisnan
In other words, I think his private contributions should still manifest on the contribution graph. And for being the CTO of an organization like GitHub, with no open-source contributions... Not a great look.
you have to opt in to private contributions being visible like that