The communities w/relationships is a much bigger dataset. I tried grabbing 100 records per fetch in a loop with no sleep or throttle. Page 523 had 100 records and page 524 was an empty file. I restarted with the skip at 523 and got to page 531. It died again, this time leaving a file that ended in the middle of a JSON field.
Any suggestions? I wonder if I should put a 1 or 2 second delay between pages so the server is not overloaded.
(update) wow, this is bulkier than I expected. 966mb. Hope that didn’t cause any problems. I guess I won’t do that full fetch again. I don’t suppose there an API parameter to select records with updatedAt newer than a specified date?
(update 2) is skip the number of pages, or records? I treated it as pages but it’s starting to look like that’s number of records -- which would mean I grabbed a lot of dupes. Sorry! (if that’s the case)
(update 3) Shit.. looks like skip is the number of records, which makes sense. Sorry for the waste! I’ll fix my script.