171
[PSA] Admins: Watch for the antiyanks troll and consider adjusting your rate limits (lemmy.world)
submitted 1 year ago by to c/fediverse@lemmy.world
47 comments fedilink hide all child comments

top 44 comments

sorted by: hot top controversial new old
[–] 25 points 1 year ago (22 children)

Thanks for the heads up. I don't know what 'Antiyanks' is, but I already had to ban one comment spammer.

The rate limits are indeed a bit confusing. The settings are:

Rate Limit: X Per Second: Y

I understand this to be 'X for every Y seconds'

So, a 'Comments' Rate limit: 10, Per second: 60, means a maximum of 10 comments per minute, correct?

Maybe the reason you see 99999999 is due to troubleshooting. I have increased my instance's limits multiple times while troubleshooting server issues, because the meaning of the settings was not clear to me. These limits are usually not the reason for the sever issue, but I put some high number and did not bring them back down after the issues were resolved.

I have lowered them now to more reasonable numbers. I will also be more strict with new applications for the time being.

  • permalink
  • fedilink
  • source
  • hideshow 22 child comments
    •
  • [–] 15 points 1 year ago* (2 children)

    Hmmm - after changing these settings to what I think are reasonable settings, the server crashed and I am now getting 'Too many requests' messages.... So, perhaps there is something not working so well with these rate limits, or I am still misunderstanding their meaning.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
    • load more comments (1 reply)
  • [+] [S] 7 points 1 year ago* (last edited 1 year ago) (18 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 18 child comments
    •
  • [–] 7 points 1 year ago

    I don’t recall any of them being from mander (unless they were dealt with before I started testing?), but thanks for taking preventative measures :)

    I don’t know what ‘Antiyanks’ is

    It’s the codename for a particular long-term troll and is based off of their original username pattern (which they still use sometimes). I have reason to believe it’s also the same troll that used to spam the racist stuff in Science Memes.

    These are most of today’s batch (minus the JON333 which was just a garden-variety spammer that made it into the last screenshot).

    No, they were not in mander.xyz. But I am generally quite relaxed when it comes to accepting applications. I mostly reject an applicant if it is very clear it is not an actual user, and then actively follow up on recent accounts for a short time. So the possibility of silent spammer accounts accumulating over time is always a concern.

  • permalink
  • fedilink
  • source
  • parent
  • [–] 6 points 1 year ago (16 children)

    So, a ‘Comments’ Rate limit: 10, Per second: 60, means a maximum of 10 comments per minute, correct?

    Correct, per client IP.

    Setting the limits to more reasonable values, like '20 posts per minute', causes the server to stop serving posts. My front page goes blank.

    So, I am starting to think that '20 pots per minute' means 'requesting 20 posts per minute' and not 'creating 20 posts per minute'.

    I am still having doubts about what these limits mean, but setting reasonable numbers seems to break things, unfortunately.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 16 child comments
    •
  • [+] [S] 2 points 1 year ago (15 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 15 child comments
    •
  • [–] 5 points 1 year ago (14 children)

    Thanks! Yes, I saw both messages and I am now going through the NGINX config and trying to understand what could be going on. To be honest, Lemmy is the hobby that taught me what a 'reverse proxy' and a 'vps' are. Answering a question such as 'Are you sending the client IP in the X-Forwarded-For header?' is probably straight forward for a professional but for me it involves quite a bit of learning 😅

    At location /, my nginx config includes:

          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;

    So, I think that the answer to your question is probably 'yes'. If you did have these rate limits and they were stable, the more likely explanation is that something about my configuration is sub-optimal. I will look into it and continue learning, but I will need to keep my limits a bit high for the time being and stay alert.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 14 child comments
    •
  • [+] [S] 0 points 1 year ago* (13 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 13 child comments
    •
  • [–] 4 points 1 year ago (12 children)

    Yes, I see this there. Most of the nginx config is from the 'default' nginx config in the Lemmy repo from a few years ago. My understanding is somewhat superficial - I don't actually know where the variable '$proxy_add_x_forwarded_for' gets populated, for example. I did not know that this contained the client's IP.

        # backend
    location ~ ^/(api|pictrs|feeds|nodeinfo|.well-known) {
      proxy_pass http://0.0.0.0:8536/;
      proxy_http_version 1.1;
      proxy_set_header Upgrade $http_upgrade;
      proxy_set_header Connection "upgrade";

      # Rate limit
      limit_req zone=mander_ratelimit burst=30000 nodelay;

      # Add IP forwarding headers
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header Host $host;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    }

    I need to do some reading 😁

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 12 child comments
    •
  • [+] [S] 1 point 1 year ago (11 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 11 child comments
    •
  • [–] 4 points 1 year ago (10 children)

    Thanks!

    I was able to crash the instance for a few minutes, but I think I have a better idea of where the problem is. Ths $emote_addr variable seems to work just the same.

    In the rate limit options there is a limit for ''Message''. Common sense tells me that this means 'direct message', but setting this to a low number is quite bad. While testing I eventually set it to '1 per minute' and the instance became unresponsive until I modified the settings in the database manually. If I give a high number to this setting then I can adjust the other settings without problem.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 10 child comments
    •
  • [+] [S] 0 points 1 year ago (9 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • parent
  • hideshow 9 child comments
    •
  • [–] 4 points 1 year ago (8 children)

    So, ultimately my problem was that I was trying to set all of the limits to what I thought were "reasonable" values simultaneously, and misunderstood what 'Message' meant, and so I ended up breaking things with my changes without the reason being obvious to me. I looked into the source code and I can see now that indeed 'Messages' refer to API calls and not direct messages, and that there is no 'Direct Message' rate limit.

    If I let 'Messages' stay high I can adjust the other values to reasonable values and everything works fine.

    Thanks a lot for your help!! I am surprised and happy it actually worked out and I understand a little more 😁

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 8 child comments
    •
  • [–] 1 point 1 year ago (5 children)

    Hi I think I set the messages too low as well and now no.lastname.nz is down, pointers on how to fix with no frontend?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 5 child comments
    •
  • [–] 1 point 1 year ago (2 children)

    Sorry, I went to sleep. Glad you were able to sort it out 😄

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 2 child comments
    •
  • [–] 2 points 1 year ago (1 child)

    All good, taught me again not to rely on chatGPT. I even said I needed to find the right BD table and field and it lead me down a rabbit hole of editing config/env files

    Fuck AI

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 1 child comment
    •
  • [–] 1 point 1 year ago

    Haha, yeah, trusting ChatGPT with how to manipulate the database and change config files is a risky move 😆 I did use it myself to remind me of the postgresql syntax to find and alter the field.

  • permalink
  • fedilink
  • source
  • parent
    • [+] 15 points 1 year ago (5 children)
    [deleted]
  • permalink
  • fedilink
  • source
  • hideshow 5 child comments
    •
  • [–] 7 points 1 year ago (4 children)

    What is a shitposter?

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 4 child comments
    •
    • load more comments (1 reply)
    • [–] 11 points 1 year ago (4 children)

    Really bad thread-breaking comment spam under this post: https://lemmy.world/post/34824537

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
  • [–] 9 points 1 year ago (3 children)

    I was wondering about that. The same comment got spammed like 800 times.

  • permalink
  • fedilink
  • source
  • parent
  • hideshow 3 child comments
    •
    • [–] 10 points 1 year ago

    Thanks for sharing

  • permalink
  • fedilink
  • source
    •
    [–] 10 points 1 year ago (1 child)

    ok, I fucked up and now my instance is erroring :(

  • permalink
  • fedilink
  • source
  • hideshow 1 child comment
    •
  • [–] 11 points 1 year ago

    If you have DB access, the values are in the local_site_rate_limit table. You’ll probably have to restart Lemmy’s API container to pick up any changes if you edit the values in the DB.

    100 per second is what I had in my configuration, but you may bump that up to 250 or more if your instance is larger.

    Fixed:

    UPDATE local_site_rate_limit SET message = 999, message_per_second = 999 WHERE local_site_id = 1;

  • permalink
  • fedilink
  • source
  • parent
    • [–] 10 points 1 year ago (4 children)

    I don’t operate a fed server, but just curious. Couldn’t/shouldn’t rate limits be on a per-user/per-session-token basis to avoid the vpn issue you mentioned?

  • permalink
  • fedilink
  • source
  • hideshow 4 child comments
    •
    • load more comments (1 reply)
    [–] 7 points 1 year ago* (3 children)

    I'm not an admin or really THAT technically knowledgeable when it comes to underlying infrastructure of these things, but because you mention VPN users in reference to shared IPs - would it be worth considering and mentioning mobile users or users otherwise on CGNAT networks?

    For example, TMobile Home Internet would result in multiple users being represented by a shared public IP. Maybe these exit nodes don't have nearly the number of users under one IP in comparison to users behind a popular VPN services assigned IPs? I don't know, but thought it might be relevant! I understand it's also a tool geared toward combatting this spam and only so much can be considered against the improvement.

  • permalink
  • fedilink
  • source
  • hideshow 3 child comments
    •
    • load more comments (1 reply)
    [–] 3 points 1 year ago

    Seems some others (or maybe alts? I don’t have the server IP logs) are getting in on the action. Saw this a minute ago

  • permalink
  • fedilink
  • source
    •