For instance, the gravity falls episode "society of the blind eye", in the fandom.com wiki, there's a synopsis for the episode. It would be incredibly helpful if I had some kind of generator, where I could paste the URL of said wiki page, and it would output a lorebook text file for me, with what the AI could extrapolate and phrase into said lorebook. Is this possible? have I somehow missed this generator? Please help, I'm sure someone's thought of the need for this by now.

you are viewing a single comment's thread
view the rest of the comments
[–] 1 point 1 year ago (1 child)

What you're asking for is an automation tool called a "web crawler" or "spider" (IT guys love their jokes). Just a heads up, websites have a million reasons why they don't like such tools and will very likely block your ability to do it easily.

Here's one for Wikipedia. No clue if it's baked in for just the main site or if you can use other wikis: https://janus-tg.github.io/Araneae/

If you want to do it right, learn python and have fun! There are scripts to pull web text contents, then you can truncate the parts you don't want like the header, page tabs, and reference footer using string-parsing functions.

  • source
  • hideshow 1 child comment
  • [–] 1 point 1 year ago*

    I'm currently in the process of making a script that gets every available URL from a website. It's been done a hundred times before, but I'm doing it for learning.

    OP, you should look into BeautifulSoup for Python. It would be a starting point for you. That will give you the full HTML of the webpage.

  • source
  • parent