Since yesterday a lot of users in Europe found their workflows failing due to Github seemingly randomly throwing HTTP/401 on git clone/git pull when interacting with public repos without authentication.

It was now confirmed by staff member that this indeed is intentional and no further steps are planned at this point.

all 75 comments

sorted by: hot top controversial new old
[–] 57 points 2 months ago (15 children)

Public GitHub repositories remain public and can still be accessed without a GitHub account, including repositories owned by paying customers. However, a subset of unauthenticated clone or fetch requests may now be asked to authenticate as part of GitHub’s protections against abusive traffic. If you receive a 401, update your application or script to use GitHub credentials.

Uh okay.

  • source
  • hideshow 15 child comments
  • [–] 33 points 2 months ago (14 children)

    I wonder why people still keep up with this bullshit and not switch to some better public Git hosting provider.

  • source
  • parent
  • hideshow 14 child comments
  • [–] 5 points 2 months ago* (last edited 2 months ago)

    Can you help me find a solution that:

    1. I don’t have to self host
    2. Provides SSO
    3. Provides local and remote build agents and actions
    4. Allows me to store private proprietary code
    5. Supports static IPs for runners

    Number 4 rules out Codeberg. The only other one that really supports that level is Azure DevOps, and well….

  • source
  • parent
  • [–] [S] 8 points 2 months ago (12 children)

    Could you suggest an alternative?

    I know Codeberg exists, but they had reliability problems recently IIRC?

    sr.ht and Gitlab are paid products.

    Technically one can self-host a forge, but my attempts at setting up CI were unsuccessful (IMO that's way more complicated that setting up the forge itself).

  • source
  • parent
  • hideshow 12 child comments
  • [–] 33 points 2 months ago (4 children)

    Codeberg's couldn't be much worse than github's reliability since 2019

  • source
  • parent
  • hideshow 4 child comments
  • [–] -1 points 2 months ago* (last edited 2 months ago) (2 children)

    The problem with Codeberg now is that there is a spectrum of AI usage between manually coding and AI slop that Codeberg chooses to ignore with their new policy.

    I get it that's their platform and their choice, and at some point I even looked for ways to support them financially if that meant I could migrate all my repos from GitHub - I wouldn't mind paying for my use, "vibe coded" or not - but an intransigent LLM ban creates uncertainty for many of my active projects, and I can't consider them a viable alternative to GitHub anymore.

    So sure, I'll take 3h of GH actions downtime a month over a platform ban. I'm sure I'm not alone there.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 7 points 2 months ago (1 child)

    So, you should be fine with the measures against scraping, then?

  • source
  • parent
  • hideshow 1 child comment
  • [–] -2 points 2 months ago* (last edited 2 months ago)

    That doesn't affect me, but you can't eat your cake and have it too. I'd rather have open source content actually open and easy to access, but GitHub has an availability issue and they're trying to address it, so I don't see this as a hostile move.

    I'm sure if codeberg was nearly as popular as github, they'd be forced to restrict some clones too. Their current load is unprecedented, even for their standards.

  • source
  • parent
  • [–] 13 points 2 months ago (2 children)

    you can self host gitlab too, and its free. yes, you CAN buy a license, but you can run it free forever. you can also use their SaaS free forever too.

    at least right now, I think it's the best alternative, though I completely understand people wanting to favor OSS.

    disclaimer: I have contributed code to gitlab, but I am NOT an employee.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 8 points 2 months ago (1 child)

    Their SaaS is pretty good but of course you are running the same chain trust. You're betting that gitlab doesn't enshittify within the next 5 years which is hardly a guarantee.

    Self-hosting gitlab is very resource-intense and complex from what I tried, though I did only try it two or three times.

    I did set up forgejo which was way easier and less heavy but I haven't tested it much so who knows.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 2 months ago

    I hosted GitLab Community Edition on-prem as a trial for a very small team, but switched to a Microsoft offering. Part of it was it being demanding, part of it was not. The point is you can host CE yourself and get an experience that’s very similar to their service, “for free,” where “free” translates to your hardware requirements and responsibility. I’d even say it’s worth it to go with GitLab, in that context, for the familiarity you can provide your team. “You’ll have to learn gitonator9000” turns into “you’ve used GitLab, right?”

    Personal use? You could run it in a container and periodically backup your data. Is it proportionally more demanding than other things? Probably.

  • source
  • parent
  • [–] 4 points 2 months ago

    Yes, they had reliability problems because so many fucking people are suddenly switching to them because they're so much better overall and not evil. Those are the kind of reliability problems that it is genuinely nice to see someone having. Having a little bit of a bumpy road when scaling due to significant rapid adoption is normal. So what?

    Imagine not wanting to use Linux because it recently had a bunch of security flaws. And it did. But again, so what? Does that mean Linux has always been insecure? Well for those things it was. But is it still insecure? Maybe, who knows, nothing is perfect. Are you going to refuse to use it because you're not sure? Why? Past performance is not an indicator of future success.

    If some minor reliability issues are your foremost concern to the point that the other things codeberg provides for free are not valuable to you because of it, I question the depth of your priorities.

    That said, it is much healthier and better for people to self-host or use smaller less centralized providers if possible. I do not wish Codeberg to become a victim of their own success, and a healthy ecosystem is a diverse one. But it is not for everyone, and if all you need is a minimal fuss alternative to Github, Codeberg is right there.

  • source
  • parent
  • [–] [S] 123 points 2 months ago (16 children)
  • [–] 24 points 2 months ago (14 children)

    The word is “scrapers,” as in to scrape.

  • source
  • parent
  • hideshow 14 child comments
  • [–] 8 points 2 months ago (2 children)

    Well given the waves of "buy book, scan, scrap" scrappers might become an acceptable version as well soon.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 3 points 2 months ago (1 child)

    We’re talking about AI bots scraping GitHub in this post. Nothing to do with books.

  • source
  • parent
  • hideshow 1 child comment
  • Yes, and I was talking about the practice of some AI companies buying books in bulk to feed to their LLMs en masse which sometimes/often destroys the book in the process.

    I was trying to build a bridge between those two related topics to "justify" the mistake and make it an intentional remark instead.

  • source
  • parent
  • [+] -10 points 2 months ago (9 children)
  • [–] 2 points 2 months ago* (8 children)

    Are you claiming that “AI scrappers” is a valid term and not just an (all too common) misspelling? If so, I’ll need to see a source on that, please.

  • source
  • parent
  • hideshow 8 child comments
  • [–] 0 points 2 months ago (7 children)

    Hah. No. I'm claiming there's a joke that went over your head. And apparently over the heads of quite a few more readers. That's OK.

  • source
  • parent
  • hideshow 7 child comments
  • [–] 5 points 2 months ago (4 children)
  • [–] 1 point 2 months ago (3 children)

    That the misspelling is amusing because if you look at it in a certain way it would still apply to AI/LLM companies. Whether it was intended or not.

    Do we always have to be so tight and literal about everything?

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 2 months ago* (2 children)

    Yeah except in the context of the post/article, it seems pretty far fetched for someone to be referring to the “scrapping” of books (assuming that’s what you meant; it’s strange that neither you nor the person who posted “whoosh” was willing to clearly state what the supposed joke was).

    Misspellings annoy me, especially when they’re common like this one. Trying to explain away the misspelling as some kind of “joke” annoys me even more.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 0 points 2 months ago (1 child)

    Misspellings are misspellings. They happen, nothing to get worked up about as long as you can decode what was said. But whatever, if it annoys you, it annoys you.

    It's been a couple of days now, and I'm replying through the response reply so I'm lacking rereading the context. But my memory here, was someone was explaining it as a potential joke in a typical social fashion, where you'd expect people to have a bit of chuckle but know that's not really the case.

    Instead a bunch of people are getting so literal and piling on them for not being correct, I suspect they already knew they were unlikely to be correct.

    It's the kind of response that sucks the fun out the internet. It just teaches people, no-one will appreciate a bit of playing around and everything becomes a little more sterile.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 1 point 2 months ago*

    Misspellings are misspellings. They happen, nothing to get worked up about as long as you can decode what was said. But whatever, if it annoys you, it annoys you.

    In this case the person went through the effort to make an image macro/meme, which still had a misspelling in it. Why would you go through all that effort without double checking the spelling? It reeks of laziness.

    But my memory here, was someone was explaining it as a potential joke in a typical social fashion, where you’d expect people to have a bit of chuckle but know that’s not really the case.

    The person I was replying to just said "woooosh" and then clarified that I was definitely missing a joke. They've since disappeared (probably because they realized they were wrong) but they were not engaging in the banter you describe; they were wrongly asserting that the spelling was intentional and that I missed the joke. The original poster of the meme then clarified that it was indeed a mistake.

    It’s the kind of response that sucks the fun out the internet. It just teaches people, no-one will appreciate a bit of playing around and everything becomes a little more sterile.

    I've been on the internet for over 30 years and conversations like this have been occurring since before my time here. I'm still having fun; if you can't have fun maybe you should log off for a bit.

  • source
  • parent
  • [–] 74 points 2 months ago

    did microsoft just lock up a good chunk of open source behind a (stochastic (for now)) login wall?

    this feels like it should violate the gpl, but i bet it doesn't. truly devious.

  • source
  • [–] 27 points 2 months ago

    Our aim is to make public repositories accessible without authentication as much as possible. However,...

    It's new reddit then. I was still using Github as a shitty backup for my projects, but an alternative may be required faster than expected.

  • source
  • [–] 19 points 2 months ago* (2 children)

    I deleted my repos on github and moved to a mix of codeberg and selfhosted forgejo.

    But I do occasionally fork/clone/reupload random projects from github to github so they have to spend money on storage and the scrapers.

  • source
  • hideshow 2 child comments
  • [–] 17 points 2 months ago

    Embrace. Extend. Extinguish.

  • source
  • [–] 16 points 2 months ago (1 child)

    Time to leave github! Boycott is the only language companies understand!

  • source
  • hideshow 1 child comment
  • [–] 5 points 2 months ago

    Done a long time ago. It would be great if others at least had official mirrors. A mirrors.txt with a list of links in the repo would be very helpful and could be a start.

    But I'm afraid people will either just stay or start hosting on the cursor git forge.

  • source
  • parent
  • [–] 21 points 2 months ago (13 children)

    This might be a problem for many projects. Rust, comes to mind, that pulls everything from Github and is 100% dependent on it. Same as Go. They pull everything from Github.

    It might make sense for them to use something like radicle, a distributed git. Many people can easily pick what they want to distribute. Scripts can be written that make the local node only host the projects you depend on. That would naturally make popular projects more available.

  • source
  • hideshow 13 child comments
  • [–] 8 points 2 months ago (7 children)

    Wait, doesn't rust cargo pull from crates.io?

  • source
  • parent
  • hideshow 7 child comments
  • [–] 3 points 2 months ago* (last edited 2 months ago) (6 children)

    crates.io pulls everything from github.

  • source
  • parent
  • hideshow 6 child comments
  • [–] 16 points 2 months ago (3 children)

    No true at all.

    The index used to be from github but now it pulls from a CDN by default and doesn't do a full clone of the index. Crate downloads have always been pulled from a CDN.

    The index was quite small in the beginning this is why it worked.

  • source
  • parent
  • hideshow 3 child comments
  • [–] 1 point 2 months ago (2 children)

    The service itself pulls from github to create the crates. Cargo pulls from crates.io, that is correct.

  • source
  • parent
  • hideshow 2 child comments
  • [–] 10 points 2 months ago

    Who pulls from Github to create the crates? The crates are pushed by cargo publish to crates.io's API, not to github.

    crates.io does have an index in GitHub, but that's legacy and pretty much unused for reads now. GitHub doesn't store the crates themselves.

  • source
  • parent
  • [–] -4 points 2 months ago (1 child)

    Does all of that access really need to be anonymous access? I get that it's convenient. But let's be practical; they have to manage anonymous bot spam somehow.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 6 points 2 months ago

    Yeah I'm not managing several different accounts for each vm in qubes os.

    The only real way to maintain any semblance of digital privacy is to build information silos.

    That will always be more important than Microsoft's hosting costs. (Though probably not to them.)

  • source
  • parent
  • [–] 25 points 2 months ago*

    Holy crap this is huge!

    Open source is no longer open source on github.

  • source
  • [–] 5 points 2 months ago (1 child)

    First time ever seeing color in a post title

  • source
  • hideshow 1 child comment
  • [–] 5 points 2 months ago (2 children)

    Our aim is to make public repositories accessible without authentication as much as possible. However, like much of the Internet, we continue to see significant increases in the volume of robot traffic recently which has increased the need for verification, for example CAPTCHAs.

    As I expected, it's about combating (excessive) bot traffic.

  • source
  • hideshow 2 child comments
  • [–] 7 points 2 months ago (2 children)

    I believe that it is some fuckup and they don't know where exactly problem is and while they are looking for a way to fix it they've created plausible lie. When they'll fix it or believe that it's fixed there would be a public announcement like 'we heard the community and reversed our decision' and users will be happy. There is a serious need for github mirror, they are becoming less and less stable every year.

  • source
  • hideshow 2 child comments
  • [–] [S] 13 points 2 months ago

    I'm 100% sure this is damage control on their part, they refused to acknowledge the incident and are looking for their way out.

    What users found in this thread is

    • problem is limited to EU
    • problem is limited to subset of git builds (gix version x TLS lib x TLS lib version in place)
    • problem goes away if you switch back to HTTP/1.1 for some reason

    If these are LLM scrapper mitigation steps then apparently fighting LLM scrappers is 7D chess game or something 🤷

  • source
  • parent
  • [–] 7 points 2 months ago (4 children)

    AI bot problem is real and there is no good solution to it. Look, I very much dislike GitHub for various reasons BUT currently there is no good way to throttle AI bots that literally trash web. They are like that geeky classmate who can never hold his liquors: it's nice having them around for some answers, but they ramble a lot and shit/puke in random places of the house making it unlivable.

  • source
  • hideshow 4 child comments
  • [–] 7 points 2 months ago (1 child)

    BUT currently there is no good way to throttle AI bots that literally trash web.

    Sure is. There are plenty of protocols resilient to DDOS.

  • source
  • parent
  • hideshow 1 child comment
  • [–] 4 points 2 months ago

    DDOS is most effective on expensive endpoints. A git clone isn't exactly cheap. Fetching is especially expensive if you vary the revisions you fetch.

    Does GitHub have an alternative here? A Git-compatible protocol with anonymous access that is DDoS-resilient?

  • source
  • parent
  • [–] 2 points 2 months ago (5 children)

    This seems reasonable. Cry all you want but if even smaller sites are getting overwhelmed with bots imagine what it's like for GitHub.

  • source
  • hideshow 5 child comments
  • [–] [S] 13 points 2 months ago (3 children)

    I'm not sure scrappers are so kind to git clone instead of rendering every single file in HTML 🤔

  • source
  • parent
  • hideshow 3 child comments
  • [–] 5 points 2 months ago* (last edited 2 months ago)

    My obscure site for my small business is getting hundreds of thousands of hits a month and its practically all bots.

    When Cloudflare says bots are most traffic nowadays, I believe it based off what I see.

  • source
  • parent