Voting is now open for the new sh.itjust.works sidebar banner.

There are eight entries below. To vote, leave a top-level reply containing the number of your choice (1–8). Please vote only once. If you change your mind, edit your original reply instead of posting another vote.

The original submission thread is here: https://sh.itjust.works/post/65782570

Rules for this vote

  • Only sh.itjust.works accounts may cast a vote
  • The votes will be tallied on Friday, September 18th.
  • Any further discussion should be posted in the thread linked above

1 — Gullible

Entry 1 — Gullible

View the original submission

2 — cantstopthesignal

Entry 2 — cantstopthesignal

View the original submission

3 — Sinopa

Entry 3 — Sinopa

View the original submission

4 — Eyekaytee

Entry 4 — Eyekaytee

View the original submission

5 — Susaga

Entry 5 — Susaga

View the original submission

6 — loam

Entry 6 — loam

View the original submission

7 — Sinopa

Entry 7 — Sinopa

View the original submission

8 -- Current

Entry 8 - Current

Reply with the number of your choice. For example: 3

[–] [S] 9 points 2 months ago

Great question and I’m happy to provide some clarity.

When the server comes under significant load from this type of coordinated scraping activity, it becomes extremely difficult to investigate the issue while it is happening. Even with out-of-band access, the server can become so overloaded that it is nearly unresponsive.

To relieve some of that pressure, I temporarily enable Cloudflare’s Under Attack Mode. This helps reduce the amount of traffic reaching the server, but it also interfere with certain third-party applications, such as Voyager and others.

I enable Under Attack Mode to reduce the load and collect access logs showing what the filtered traffic looks like. I then disable it temporarily to capture additional traffic, including requests that were previously being blocked. This is why you may intermittently regain access during these incidents.

By comparing these two datasets, I can more quickly distinguish legitimate traffic from malicious or unwanted traffic patterns. From there, I can adjust the security rules to make it more difficult and costly for that bad traffic to reach the server in the first place.

Sorry if that got a little technical.

  • source
  • parent
  • context
  •  

    cross-posted from: https://sh.itjust.works/post/66524166

    Good day SJW community,

    I wanted to provide everyone with an update regarding the stability issues we experienced over the past 24 hours.

    Beginning Wednesday, September 9, we observed a significant spike in incoming requests originating from a large number of IP addresses associated with networks operated by Baidu, Tencent and OneCable Network LLC.

    The traffic patterns pointed to coordinated web scraping activity distributed across large IP ranges. This approach allowed the scrapers to circumvent the per-IP rate limits we have in place by spreading requests across many different addresses. The requests were primarily targeting comments and post content, suggesting the activity was focused on large-scale data collection, potentially for AI model-related purposes.

    We identified this activity by analyzing traffic patterns at the Autonomous System Number (ASN) level and reviewing the average number of requests per minute generated by each network.

    At the same time, we identified a separate source of abnormal traffic that appeared to be coming from another actor. These requests were performing expensive search queries that placed significant load on the server and affected its ability to respond to legitimate requests in a timely manner.

    This traffic was using the following user agent:

    NexusGlobalBot/2.0 (SupabaseEdgeRuntime/1.76.0)

    After blocking this agent and its associated request patterns, we observed a significant reduction in server load, bringing utilization much closer to normal operating levels.

    This is not the first time we have encountered this type of activity over the past several months and unfortunately, it is unlikely to be the last. We will continue improving our monitoring, rate limiting, and mitigation measures to reduce the impact of abusive or excessive automated traffic while maintaining reliable access for legitimate users.

    Thank you for your continued support and trust.

     

    Good day SJW community,

    I wanted to provide everyone with an update regarding the stability issues we experienced over the past 24 hours.

    Beginning Wednesday, September 9, we observed a significant spike in incoming requests originating from a large number of IP addresses associated with networks operated by Baidu, Tencent and OneCable Network LLC.

    The traffic patterns pointed to coordinated web scraping activity distributed across large IP ranges. This approach allowed the scrapers to circumvent the per-IP rate limits we have in place by spreading requests across many different addresses. The requests were primarily targeting comments and post content, suggesting the activity was focused on large-scale data collection, potentially for AI model-related purposes.

    We identified this activity by analyzing traffic patterns at the Autonomous System Number (ASN) level and reviewing the average number of requests per minute generated by each network.

    At the same time, we identified a separate source of abnormal traffic that appeared to be coming from another actor. These requests were performing expensive search queries that placed significant load on the server and affected its ability to respond to legitimate requests in a timely manner.

    This traffic was using the following user agent:

    NexusGlobalBot/2.0 (SupabaseEdgeRuntime/1.76.0)

    After blocking this agent and its associated request patterns, we observed a significant reduction in server load, bringing utilization much closer to normal operating levels.

    This is not the first time we have encountered this type of activity over the past several months and unfortunately, it is unlikely to be the last. We will continue improving our monitoring, rate limiting, and mitigation measures to reduce the impact of abusive or excessive automated traffic while maintaining reliable access for legitimate users.

    Thank you for your continued support and trust.

    [–] 59 points 2 months ago* (last edited 2 months ago) (9 children)

    We are experiencing some kind of bot attack at the moment. Voyager and other mobile apps are unavailable while we attempt to mitigate this. Only the web-ui is available. We’ll have more to share soon. Thanks for your patience

    Edit: Fix typo, grammar

  • source
  • [–] 50 points 3 months ago* (last edited 2 months ago) (6 children)

    Hey @westingham

    Thanks for starting a post about this.

    Our traffic today jumped 4x our previous daily high. I had to enable some extra protection to give the server some breathing room to recover. One of the side effects is the mobile apps are temporarily unavailable until I get everything tweaked. It will be back in a few days. The good news is that I already got my hands on a newer more powerful server that will be going into production in the next few weeks.

    For the next few days if you want to scroll the site you will need to use your mobile browser. I understand its totally not the same experience as voyager but I promise you i’ll get everything back up as it was as soon as I can.

    Thanks for your trust and patience!

    Edit: Access via the mobile app Voyager has been reinstated.

  • source
  •  

    cross-posted from: https://sh.itjust.works/post/64240582

    To the members of SJW, our friends across the Fediverse, and the curious passersby, this update is for you.

    Now that we are well into 2026, I wanted to look back at how 2025 unfolded and share some insight into SJW’s finances, infrastructure, traffic trends, and priorities for the year ahead.

    2025 Finances

    One of the biggest changes in 2025 was the introduction of member-supported funding.

    In January of 2025, I opened donations to help cover the cost of operating SJW. The goal was to begin moving away from a model funded entirely by me and toward a community-supported model that can remain financially sustainable over the long term.

    Throughout 2025, we received $3,239.41 in net donations and spent $1,396.14 on operational expenses.

    During the year, we also experienced a few hardware failures, most notably a failed power supply and hard drive. Thanks to the redundancy built into our servers, neither failure resulted in a service disruption.

    I was able to repurpose a compatible power supply from a decommissioned server at no cost. I also personally covered the replacement hard drive before prices went through the roof. While donation funds could have covered these expenses, I wanted to preserve as much of the balance as possible for a larger hardware upgrade.

    All amounts below are in Canadian dollars.

    Category Amount Additional information
    Net donations received $3,239.41 January through December 2025
    S3-compatible storage $69.62 Paid through July 2026
    Email service $70.60 Paid through July 2026
    Colocation $1,200.00 2025 infrastructure hosting
    Domain name $55.92 Domain registration
    Total expenses $1,396.14
    Remaining balance $1,843.27 Reserved for operations and future infrastructure

    This year, I hope we can raise enough to help purchase newer hardware. Unfortunately, as many of you are probably aware, the rapid growth of AI infrastructure has significantly increased the price and demand for many server components.

    Traffic Trends

    This is where things become particularly interesting and, at first glance, a little puzzling.

    Based on our traffic logs and Cloudflare data, overall traffic approximately doubled during 2025, despite SJW adding only around 4,000 active accounts during the same period.

    As of January 2026, when I first began preparing this update, Cloudflare was reporting:

    • More than 300 million pages served per month
    • Approximately 6 million unique visitors over a 30-day period

    These are enormous numbers, but they almost certainly do not represent six million individual people.

    I suspect that at least 50 to 60 percent of this activity, and possibly more, comes from bots, scrapers, automated crawlers, vulnerability scanners, archival services, and other automated systems. Some of this activity may also come from services gathering public information for traditional search engines and newer AI-powered search tools.

    I recently implemented Anubis to help reduce abusive automated traffic. While it cannot prevent every bot or scraper from accessing the site, it makes large-scale automated collection more difficult and has helped us begin addressing the problem.

    Hardware and Infrastructure

    Given these traffic trends and some of the performance issues we have encountered, our current hardware is reaching the point where I need to seriously consider replacing or upgrading it.

    I recently changed our Lemmy deployment to use horizontal scaling. In simpler terms, we now run multiple Lemmy service instances in parallel and distribute incoming traffic between them. This has helped improve the instance’s ability to handle periods of heavier activity.

    However, software improvements can only take us so far. Newer hardware would give us more capacity, better performance, and additional room to build resilience into the platform.

    I would also like to explore bringing our primary S3-compatible object storage in-house while maintaining an appropriate off-site backup strategy. This would reduce our reliance on third-party storage providers while giving us more control over privacy, performance, and redundancy.

    2026 Priorities

    Our main infrastructure priorities for 2026 are:

    • Upgrade or replace aging server hardware
    • Add further redundancy to critical services
    • Continue improving horizontal scaling
    • Reduce the impact of abusive automated traffic
    • Explore self-hosted object storage
    • Deepen our collaboration with Fedecan

    Fedecan

    As some of you may remember, we previously shared our intention to join Fedecan, an organization with which we share common values and a similar vision for the Canadian Fediverse.

    That transition has now been finalized. SJW is associated with a registered Canadian nonprofit organization and can benefit from its banking, administrative, and nonprofit structure.

    The initial banking and legal integration is complete, but our collaboration will continue to deepen as we work toward a larger shared vision and develop common guidelines.

    Donations are now processed through Fedecan, but funds donated specifically to SJW are tracked separately and remain reserved for SJW’s operating costs, infrastructure improvements, and future service expansion.

    Donations

    As outlined above, one of our main goals this year is to upgrade our existing hardware and improve the resilience of the services supporting SJW.

    I am still evaluating the available hardware options and will share more specific information once the upgrade plan and expected costs have been finalized.

    If you have the means and would like to support SJW, donations of any size are greatly appreciated. Every contribution helps us maintain the instance, prepare for future growth, and reduce how much of the operating cost must be covered personally.

    Regardless of how much we raise, I'm committed to keeping SJW running. I will continue looking for suitable replacement hardware and am prepared to personally contribute toward any shortfall required to support the next generation of our infrastructure.

    If you'd like you can contribute through either of the following platforms:

    Thank You

    In closing, I want to thank our admins and mods, who continue to volunteer their time to help make SJW a safe and welcoming place.

    I also want to thank our members who have chosen to make the Fediverse, and SJW in particular, their home. Whether you donate, contribute to discussions, report problems, help other members, or simply spend time here, you are part of what keeps this community alive.

    Thank you for being you.

    And if nobody has reminded you lately: you are great, you are loved, and most importantly, you are enough.

    TheDude ✌️

     

    To the members of SJW, our friends across the Fediverse, and the curious passersby, this update is for you.

    Now that we are well into 2026, I wanted to look back at how 2025 unfolded and share some insight into SJW’s finances, infrastructure, traffic trends, and priorities for the year ahead.

    2025 Finances

    One of the biggest changes in 2025 was the introduction of member-supported funding.

    In January of 2025, I opened donations to help cover the cost of operating SJW. The goal was to begin moving away from a model funded entirely by me and toward a community-supported model that can remain financially sustainable over the long term.

    Throughout 2025, we received $3,239.41 in net donations and spent $1,396.14 on operational expenses.

    During the year, we also experienced a few hardware failures, most notably a failed power supply and hard drive. Thanks to the redundancy built into our servers, neither failure resulted in a service disruption.

    I was able to repurpose a compatible power supply from a decommissioned server at no cost. I also personally covered the replacement hard drive before prices went through the roof. While donation funds could have covered these expenses, I wanted to preserve as much of the balance as possible for a larger hardware upgrade.

    All amounts below are in Canadian dollars.

    Category Amount Additional information
    Net donations received $3,239.41 January through December 2025
    S3-compatible storage $69.62 Paid through July 2026
    Email service $70.60 Paid through July 2026
    Colocation $1,200.00 2025 infrastructure hosting
    Domain name $55.92 Domain registration
    Total expenses $1,396.14
    Remaining balance $1,843.27 Reserved for operations and future infrastructure

    This year, I hope we can raise enough to help purchase newer hardware. Unfortunately, as many of you are probably aware, the rapid growth of AI infrastructure has significantly increased the price and demand for many server components.

    Traffic Trends

    This is where things become particularly interesting and, at first glance, a little puzzling.

    Based on our traffic logs and Cloudflare data, overall traffic approximately doubled during 2025, despite SJW adding only around 4,000 active accounts during the same period.

    As of January 2026, when I first began preparing this update, Cloudflare was reporting:

    • More than 300 million pages served per month
    • Approximately 6 million unique visitors over a 30-day period

    These are enormous numbers, but they almost certainly do not represent six million individual people.

    I suspect that at least 50 to 60 percent of this activity, and possibly more, comes from bots, scrapers, automated crawlers, vulnerability scanners, archival services, and other automated systems. Some of this activity may also come from services gathering public information for traditional search engines and newer AI-powered search tools.

    I recently implemented Anubis to help reduce abusive automated traffic. While it cannot prevent every bot or scraper from accessing the site, it makes large-scale automated collection more difficult and has helped us begin addressing the problem.

    Hardware and Infrastructure

    Given these traffic trends and some of the performance issues we have encountered, our current hardware is reaching the point where I need to seriously consider replacing or upgrading it.

    I recently changed our Lemmy deployment to use horizontal scaling. In simpler terms, we now run multiple Lemmy service instances in parallel and distribute incoming traffic between them. This has helped improve the instance’s ability to handle periods of heavier activity.

    However, software improvements can only take us so far. Newer hardware would give us more capacity, better performance, and additional room to build resilience into the platform.

    I would also like to explore bringing our primary S3-compatible object storage in-house while maintaining an appropriate off-site backup strategy. This would reduce our reliance on third-party storage providers while giving us more control over privacy, performance, and redundancy.

    2026 Priorities

    Our main infrastructure priorities for 2026 are:

    • Upgrade or replace aging server hardware
    • Add further redundancy to critical services
    • Continue improving horizontal scaling
    • Reduce the impact of abusive automated traffic
    • Explore self-hosted object storage
    • Deepen our collaboration with Fedecan

    Fedecan

    As some of you may remember, we previously shared our intention to join Fedecan, an organization with which we share common values and a similar vision for the Canadian Fediverse.

    That transition has now been finalized. SJW is associated with a registered Canadian nonprofit organization and can benefit from its banking, administrative, and nonprofit structure.

    The initial banking and legal integration is complete, but our collaboration will continue to deepen as we work toward a larger shared vision and develop common guidelines.

    Donations are now processed through Fedecan, but funds donated specifically to SJW are tracked separately and remain reserved for SJW’s operating costs, infrastructure improvements, and future service expansion.

    Donations

    As outlined above, one of our main goals this year is to upgrade our existing hardware and improve the resilience of the services supporting SJW.

    I am still evaluating the available hardware options and will share more specific information once the upgrade plan and expected costs have been finalized.

    If you have the means and would like to support SJW, donations of any size are greatly appreciated. Every contribution helps us maintain the instance, prepare for future growth, and reduce how much of the operating cost must be covered personally.

    Regardless of how much we raise, I'm committed to keeping SJW running. I will continue looking for suitable replacement hardware and am prepared to personally contribute toward any shortfall required to support the next generation of our infrastructure.

    If you'd like you can contribute through either of the following platforms:

    Thank You

    In closing, I want to thank our admins and mods, who continue to volunteer their time to help make SJW a safe and welcoming place.

    I also want to thank our members who have chosen to make the Fediverse, and SJW in particular, their home. Whether you donate, contribute to discussions, report problems, help other members, or simply spend time here, you are part of what keeps this community alive.

    Thank you for being you.

    And if nobody has reminded you lately: you are great, you are loved, and most importantly, you are enough.

    TheDude ✌️

     

    Since there has not yet been an official announcement regarding this matter, I wanted to provide some clarity to the community.

    Last week, it was discovered that the Tesseract frontend included an undisclosed and unsanctioned blocklist that prevented users from seeing certain people and communities across the fedivese. This led to significant controversy across the Fediverse, after which the original developer withdrew the project and left the Fediverse.

    Everyone is entitled to their own views and to make their own choices about the content they wish to see. However, when those choices are built into an open-source project, they must be disclosed publicly and presented transparently to users. Quietly imposing a blocklist without their knowledge is a serious breach of trust and, in my view, raises legitimate concerns about the security and integrity of the software.

    As a result, we have disabled the Tesseract frontend on sh.itjust.works indefinitely.

    We are open to offering other alternative frontends. If there is one you would like us to consider, please share your suggestion in !main@sh.itjust.works or !agora@sh.itjust.works.

    Thank you

     

    This is the first big step in the process to develop comprehensive guidelines for the Fedecan non-profit and the various platforms.

    While this will mostly involve converting tacit knowledge and experience into an explicit written form, we expect that this process will inevitably bring up some points of disagreement on the best way to deal with different issues. We ask everyone participating in these discussions to please contribute constructively and in good faith. We encourage you to bring up any concerns or issues you have with the proposed structure and drafted guidelines, so that we can work together to fix them early on. However, in order to keep a productive environment for those discussions, we will be pruning any comment chains that devolve into personal attacks, slap fights, etc.

    To help ground your feedback, consider these thought experiments when evaluating a potential guideline:

    • Veil of ignorance: Would it still feel fair to you if you switched places with someone else on the platform (ex. a new user, a moderator, an admin, a member of a vulnerable group, etc.)?
    • Equal Applicability: These rules will be enforced uniformly on everyone. A poorly written rule that helps "your side" today, can easily harm "your side" in the future as circumstances change.

    The full guidelines, including governance details like the annual review cycle, can be found on the website: https://fedecan.ca/en/guidelines/

    We plan to structure the guidelines as follows:

    diagram of the tiers that are described below

    Tier 1: Fedecan Rules

    Internal Conduct

    These rules apply to Fedecan team members (directors, officers, admins, and anyone with elevated access). They set expectations for how team members should act.

    Universal Rules

    These are the baseline rules that apply to every user on every Fedecan platform. They cover the things that are prohibited by Canadian law (threats, hate speech, CSAM, non-consensual intimate imagery) as well as universal policy rules (privacy/doxxing, harassment, fraud, content that could cause harm, labelling of sensitive content, etc.).

    Tier 2: Platform-Specific

    Each platform has different functionality and norms, so this is where we can be more specific with the rules. The threadiverse platforms (lemmy.ca, piefed.ca, sh.itjust.works) share similar rules around community creation, moderation, vote manipulation, and content labelling. Pixelfed has its own rules tailored to its platform.

    Tier 3: Community-Level Rule Templates

    These are optional templates that communities can link to, or use as a starting point for their own rules. The idea is that moderators can point users to a clearly written explanation of why a rule exists, and any relevant exceptions, rather than trying to fit everything into the sidebar. Additionally, if many communities are enforcing a particular rule in the same way, then users will have an easier time understanding and following them.

    The post title standards template has been drafted, and we plan to add more as the need arises. I have a few others that are in the works, but they have some overlap with the other sections, so I thought that it would be better to let people discuss first.

    [–] 4 points 3 months ago (1 child)

    alright, turns out it had nothing to do with cloudflare and more related to my nginx settings. I made some changes and it appears to have fixed the issue. Let me know if this has fixed it on your side as well.

    Thank you

  • source
  • parent
  • context
  • [–] 5 points 3 months ago (2 children)

    I believe this might be related to the fact that we are behind cloudflare. I notice other instances that are behind cloudflare appear to have the same issues while those that are not behind cloudflare appear to be ok. Lemmy.world appears to strip the embedded video function completely. See https://lemmy.world/post/49302693 as an example

    I also noticed someone did open a github issue on the lemmy-ui repo https://github.com/LemmyNet/lemmy-ui/issues/4001

    I'll take a look and see if there's something in the nginx settings that could fix this

  • source
  • [–] 31 points 4 months ago (2 children)

    Hey all, most of today's outage was caused by me. I implemented Anubis in attempt to differ some bot/scrapping traffic. We are presently serving over 350 million requests per month and its starting to show in the stability in the instance. I will be switching to the horizontal scaling approach in hopes to stabilize things more.

    I'll also be providing an long overdue update once I get everything back under control.

    Sorry for the stability issues as of late, you all deserve better.

  • source
  •  

    Hello all,

    I hope everyone has been enjoying their summer and has had the chance to check a few new items off their bucket list!

    I will be upgrading this lemmy instance to version 0.19.12 tomorrow at 8PM EDT. This should be a quick update.

    As always I will provide an update once the upgrade has been completed.

    You may experience up to 30 minutes of downtime as the upgrade completes.

    I apologize for the short notice.

    Thank you for the continuing to support to SJW and the Fediverse

    UPDATE: The upgrade was successful. Took about 10 minutes to complete. Thank you

    view more: next ›