393
Huggingface have scraped the entirety of Github. (huggingface.co)
submitted 3 months ago by to c/programming@programming.dev
108 comments fedilink hide all child comments
 

Would be a terrible shame if lots of people opted out.

If you are a EU citizen you might also want to write a complaint to privacy@huggingface.co because they are collecting your personally identifiable information in machine-readable form which they are distributing to third parties.

you are viewing a single comment's thread
view the rest of the comments
[–] 6 points 3 months ago*

Noteworthy: They crawled only the default branch HEAD and inlined all source content.

  • The file contents are included inline. The decoded UTF-8 source text is embedded directly in the dataset, so it is fully self-contained — you can start training the moment the download finishes.
  • It reflects the state of GitHub in August 2025. The corpus is a direct crawl of GitHub repositories at their default-branch HEAD, capturing roughly two additional years of open-source code compared to The Stack v2.
  • permalink
  • fedilink
  • source
    •