[–] 3 points 1 year ago (3 children)

It is caching guarantees for a day. It shouldn’t do it for first time authentications I guess.

I’ll try to fix this today. At most, you will need to wait for a day. Sorry for the inconvenience.

  • source
  • Searching for it is a valid option, it's basically what the boost/federate not does if I'm not mistaken.
    Searching for it adds the community, but not the content.
    If someone subscribe, we'll receive new posts, but not the existing one.
    Syncing existing content can kinda be done by searching for each original post's permalink.
    I've done a few posts manually, so they'll show for us on SJW.
    Syncing existing comments would probably require having some sort of link for each comment which I'm not sure where to find on wordpress...

  • source
  • parent
  • context
  • [–] 5 points 1 year ago (1 child)

    I just did a manual search and received this:

    subscription pending

    So my instance was able to retrieve it, but we weren't already federated. Since I'm the first from my instance it may be a matter of syncing in the backend of things.

  • source
  • [–] [S] 2 points 1 year ago* (last edited 1 year ago) (1 child)

    The communities w/relationships is a much bigger dataset. I tried grabbing 100 records per fetch in a loop with no sleep or throttle. Page 523 had 100 records and page 524 was an empty file. I restarted with the skip at 523 and got to page 531. It died again, this time leaving a file that ended in the middle of a JSON field.

    Any suggestions? I wonder if I should put a 1 or 2 second delay between pages so the server is not overloaded.

    (update) wow, this is bulkier than I expected. 966mb. Hope that didn’t cause any problems. I guess I won’t do that full fetch again. I don’t suppose there an API parameter to select records with updatedAt newer than a specified date?

    (update 2) is skip the number of pages, or records? I treated it as pages but it’s starting to look like that’s number of records -- which would mean I grabbed a lot of dupes. Sorry! (if that’s the case)

    (update 3) Shit.. looks like skip is the number of records, which makes sense. Sorry for the waste! I’ll fix my script.

  • source
  • parent
  • context
  • [–] [S] 1 point 1 year ago* (last edited 1 year ago) (3 children)

    Thanks! That clears some things up.

    Using bash, I came up with this code:

    for pg in {0..999}
    do
        curl -s --get -o lemmy-federate_nodes_pg"$pg".json\
             --data-urlencode 'input={"search":"","skip":'"$((pg*50))"',"take":50,"enabledOnly":false}' \
             'https://lemmy-federate.com/api/instance.find'
        
        qty_of_items_returned=$(jq '.result.data.instances | length' < lemmy-federate_nodes_pg"$pg".json)
        [[ "$qty_of_items_returned" -gt 0 ]] || break;
    done
    

    When it reached page 80, the array was zero in length so the loop was terminated at that moment. It started off taking 50 nodes per fetch but near the end the amount fetched gradually dropped off. I was expecting it to be 50 per fetch until the last page. I wonder if that’s a throttling feature? Anyway, it’s not problem. It worked.

    update

    updated the script above to not fetch redundant records

  • source
  • parent
  • context
  • [–] 4 points 1 year ago (4 children)

    You need to change input parameters (skip, take) to paginate through pages and merge all communities you've fetched. You'll probably need to use something like Python or NodeJS.

    For example right now you're using skip=0 and take=10, which will take first 10 communities found in DB.

  • source
  • [–] 2 points 1 year ago (1 child)

    Here you go:

    https://lemmy-federate.com/api/community.find?input=%7B%22skip%22%3A0%2C%22take%22%3A10%7D
    https://lemmy-federate.com/api/instance.find?input=%7B%22search%22%3A%22%22%2C%22skip%22%3A0%2C%22take%22%3A10%2C%22enabledOnly%22%3Afalse%7D
    

    You can change the skip and take values to paginate through list.

  • source
  • parent
  • context
  • [–] [S] 2 points 1 year ago* (2 children)

    I don’t have full time Internet access. I pop into a library periodically. So I have found it useful to fetch the lemmyverse dataset and import it. Takes just a few seconds to fetch the whole dataset. Then when I am offline I search the DB to decide where to post, and write my posts offline.

    I’m trying to grasp what you are suggesting. Do you mean I write a program to reach out to each instance and harvest the data? I think there are around 150 lemmy instances. I’m not sure how quickly that can be done, or if it would be welcome. When I visit slrpnk.net using my browser, there is an anti-bot check which takes a few seconds. I don’t imagine cURL or the like would get past that, which I suspect is some javascript code that must be executed.

    In your repo I see routes/community.ts and nodes.ts. But therein I do not see a list of instances or communities to visit. Is ts for typescript? That lang is unknown to me.

  • source
  • parent
  • context
  • [–] [S] 2 points 1 year ago (4 children)

    Thanks for the feedback. Your dataset would still be useful nonetheless. I fetched the lemmyverse dataset and found it to be incomplete. E.g., it only has ⅔ of the slrpnk.net communities. So if I could combine your data it would help fill in some of the missing data. Is that available anywhere?

  • source
  • parent
  • context
  • [–] 4 points 1 year ago* (5 children)

    It is difficult to detect the connection between instances. Because Mbin and Piefed either don't provide federated instance list or the admins can turn it off. That's why I completely stopped checking federation status in Lemmy Federate, I just send an API request and if it fails I assume they are defederated.

    So, I'm sorry, I do not have this data.

  • source
  • [–] 4 points 1 year ago

    Lemmy Federate has 3 types of client types:

    • Lemmy
    • Mbin
    • Generic ActivityPub (with group federation support)

    Lemmy Federate provides first-class support for Lemmy and Mbin. Other types of software can’t follow other instances.

  • source
  • [–] [S] 2 points 1 year ago* (last edited 1 year ago)

    You switched to Piefed? It’s still registered as Lemmy instance in Lemmy Federate. This is an edge case I haven’t thought of.

    I can fix this manually but if you haven't transferred keys, switching on the same domain didn't seem healthy to me.

    Update: fixed it.

  • source
  • parent
  • context
  • view more: next ›