89
Codeberg had a major outage today! About 7 hours of complete down (July 31, 2026) (statusgator.com)
submitted 3 months ago by to c/programming@programming.dev
27 comments fedilink hide all child comments
 

StatusGator last checked the status of Codeberg on July 31, 2026 at 7:57 PM CEST and the service was operational. There have been 83 user-submitted reports of outages in the past 24 hours, but the outage is now resolved.

you are viewing a single comment's thread
view the rest of the comments
[–] 7 points 3 months ago* (last edited 3 months ago) (2 children)

Codeberg runs on Forgejo which is a fork of Gitea.

I'm running Forgejo myself and am running into difficulties with High Availability. Many data files can be stored in S3 compatible interfaces, but the Git repos are stored as Git files on disk. That's hard to distribute across multiple hosts. Something like NFS seems to be prone corruption and Forgejo has to be able to lock.

An open issue on clustering : https://codeberg.org/forgejo/discussions/issues/259

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 3 points 3 months ago* (1 child)

    Reading your comment, I immediately thought there must be an S3 FUSE wrapper out there, and quickly found this:

    https://github.com/s3fs-fuse/s3fs-fuse

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 5 points 3 months ago* (last edited 3 months ago)

    The problem is still concurrency. If the application, in this case the git CLI, is not thread-safe, then you could run into unsafe data mutations causing data corruption.

    Imagine two pushes that come in around the same time. S3 doesn't have the locking mechanics you need to ensure that only one of those writes win and the other fails. You need to upload the individual commits (which can be isolated except in the case of multiple writes with a partial write), but then the branch and HEAD pointers have to be serialized. I've done a lot of work with distributed systems in AWS and my experience has always been to layer something like DynamoDB on-top of S3 to provide this.

    My knowledge of at least one distributed Git system was that they had a fleet of Git data servers and a router on front. The router would fan it out to 3 or more data servers where each data server would store copies. But ultimately there's some application logic that is aware of the underlying filesystem and manages locking.

    GitHub shares some of their architecture here: https://github.blog/engineering/infrastructure/stretching-spokes/

  • permalink
  • fedilink
  • source
  • parent