Update: Federation and community creation are now back online!

Hey all, there’s a hack floating around which spreads via federated comments and steals users’ Lemmy auth tokens. Lemmy.world and other large instances have been hacked, so we’re taking some precautions until this is fixed:

  • We’re logging everyone out so that auth tokens reset
  • We’re closing off federation and community creation until this is patched

FYI, there are no indications that anyone on our instance has been hacked. We did find ten comments with the code injection attack, which we’ve now scrubbed. But it’s very unlikely that this will cause harm at this stage. There are several steps between this and hacking the entire instance. (Also FYI for nontechnical users, the hack affected Lemmy logins and nothing else. Web browsers run all websites in a kind of “jail”)

Sorry for the inconvenience – growing pains. Updates to come as we learn more!

  • zen@monyet.cc
    1·
    1 year ago

    edit-mmm still can’t see the posts here from my instance, sadge

    this is why I wanted allowlist-federation to friendly instances (like yours), instead of turning federation off completely.

    when I saw that it was off completely, I had a gut feeling that it would not be easy to bring back up, as I have not seen any other instance doing it this way (turning federation off completely).

    so, now troubleshooting time, curl test commands from the docs no longer work (they are all returning 404 for me), even webfinger, (which afaik, is an important enabler of federation with mastodon) doesn’t work.

    I hope this can be fixed by rebooting the server, and it’s not something more serious like turning federation off causing all the activitypub private keys to be deleted from the db.

    hate to bother you so early in the morning @dcx@monyet.cc, but I don’t know who else is part of the tech team.

    reminder: please do not copy the nginx config that is above the curl commands on the join-lemmy.org page. they contain a subtle bug. use what is in the lemmy-ansible repo, which has the bug fixed.

    anyway, I don’t think you need to touch anything in the nginx config if you’re following lemmy-ansible completely without any change. if the curl commands return html, then something is wrong with the config, but here they return 404.

    • dcx@monyet.ccOPM
      1·
      1 year ago

      Thanks for the heads-up! We’ve given it a reboot and things seem to be working as intended now. In general I have a weekly crunch period which happens around Tue-Thu so I go into low availability, and catch up on the weekend – always welcome to try other admins! Naomi is on technical too :)

      Re: Defederation / allowlist: I’m quite sure a lot of instances defederated actually! I believe I read a note on this on a github discussion, or on a lemmy.ml post. IMO if there’s an active security hole which appears to be spreadable to other instances via federation feeds, there is no reasonable basis on which we can decide to trust external instances. Without a patch, they can become infected at any time. I could be missing an angle, but I still feel full defederation was the right call. Happy to hear the arguments against!

      • zen@monyet.cc
        2·
        1 year ago

        don’t think I’ll argue against that, but full defederation requires a restart to bring back the http routes required for federation (or they’ll go 404 instead like what happened this morning), so you’ll have to remember to do that (disable federation while awaiting the security fix, upgrade the server when that fix is out, then enable federation and restart it again to really bring back federation once you’re satisfied that the upgrade is secure).

        maybe if there wasn’t a restart in between the disable and enable federation steps, the http routes wouldn’t need to be re-initialized, but in the case of a security problem, you’ll always need a restart to patch it…