Update: the original link is now broken. Reported here. New link: https://lemmy.sdf.org/post/52446618/24104545

Thanks! That link is jailed in Cloudflare’s walled garden but this variant is reachable to everyone (in case someone else wants to follow this):

https://overflow.ocus.top/questions/2882620/is-it-possible-to-remote-count-object-and-size-of-git-repository

The fetch operation does not disclose the total before fetching, but it gives a way to monitor progress which is good enough in some situations. I wonder if the percentage can be trusted. If the percentage is accurate, then it must also know the total. But the ratio is not apparently based on size.

Further down in your same article, there is a Github-specific way to get the size, which I tweaked to:

$ curl -sL https://api.github.com/repos/Marijnh/CodeMirror | jq -r .size

Perhaps the best generic answer (independent of Github) is further down the same page. This comment gives a codified way of doing what I was contemplating.

I appreciate the helpful link. Though I’m seeing a huge discrepency. E.g., for this repo, curl gives 34785 (which is apparently in kilobytes) and git count-objects -vH after fetching gives about the same answer. Yet the repo is actually nearly double that size (~61 mb).

Instead of git clone, initialize a new local repository, add the remote origin, and then git fetch:

https://stackoverflow.com/questions/2882620/is-it-possible-to-remote-count-object-and-size-of-git-repository

Good find. Apparently it’s just a matter of waiting for instances to upgrade.

Want that implement in 2015, where you can have a private repo with public access?

While that is the main way how pages are discovered, some scrapers also just try random URLs, or known words from leaks

I’ve often wondered about the reach of crawlers. Don’t crawlers need to be seeded with links? Don’t they rely on links that are published on crawled pages? If there is no navigable path to the repo, how would a crawler find it?

Of course if a friend publishes the link, all bets are off. And I would accept that.

Any content that is not hidden (private) is effectively public since crawlers will eventually find it, even if it's not in the built-in search

If it fits in Git LFS, use Git LFS.

I had to version control large (10+GB) files; too big for Git LFS. I wrote a program that stored the files in S3 with metadata in DynamoDB. There was a Java library, CLI, and a GUI. It worked well. The users liked it. I tried to get my employer to open source it but they refused.

If I had to do it over again, I would have stored the metadata in Git and written some scripts that call the Git CLI to manage the files.

Git Large File Storage is a mature solution.