Before the scaled sort was introduced, the hope was that it would provide a solution to surface posts from smaller communities, without being overrun by memes and political posts from larger communities. However, the scaled sort has been ineffective so far, as most posts appear with a single vote, making it practically the same as the “New” sort.
The developers have closed all issues related to the scaled sort, even though it fails to address the issues raised in several discussions:
- Rework “Hot” sorting to show posts from more varied communities
- The rank of a post in the aggregated feed should be inversely proportional to the size of the community
- Is there any way to reverse degrowth of the niche communities on Lemmy?
- I hate to say it but I haven’t been very active on lemmy, but I want to be
Personally, I believe the best way to address this issue is through the implementation of tags and custom feeds. With post tags and custom feeds, users could create separate feeds tailored to their preferences by subscribing to a few communities and blocking specific tags or keywords. However, this would require an incentive system similar to imageboards like Safebooru, with a leaderboard to encourage accurate post tagging by users, as also mentioned in The Great Monkey Tagging Army: How Fake Internet Points Can Save Us All!
Although I’ve blocked the largest communities, I still want to see some of that content occasionally.
Do you have any ideas or suggestions on how Lemmy could better surface content from smaller communities?
Edit:
Potential Solutions
Several potential solutions were discussed:
-
Tagging System and Custom Feeds Implementing a tagging system could allow users to create custom feeds by subscribing and blocking specific tags across communities. This could surface niche content by filtering for relevant tags. An incentive system like leaderboards could encourage accurate user tagging.
-
Community Grouping Similar to Reddit’s “Multireddits”, allowing users to group multiple smaller communities together into a single custom feed could boost visibility for those niche communities when browsing that grouped feed.
-
API for Client-Side Sorting Providing an API endpoint that shares metadata for recent posts like post ID, post votes, and comments would allow third-party clients and plugins to experiment with custom sorting algorithms on the front-end tailored to user preferences.
-
“Unanswered” View Having a view that surfaces posts with little or no engagement yet, specifically from smaller communities, could help discover underrepresented niche topics that may need more attention.
-
Server Plugin Architecture If sorting algorithms must be implemented server-side for performance, having a plugin architecture where different instance owners can test out new sorting implementations and formulas could allow faster iteration.
Ideally, a combination of tagging, custom feeds, and improving sort algorithms to factor in community size could provide a multifaceted approach to better surface content from niche communities on Lemmy. Encouraging open discussion around desirable features is valuable to guide development efforts when resources do become available.
I just want the ability for people to deploy their own sorting systems. Then we could have opensource projects and everyones happy (ok maybe not, but thats as close as we get to it)
Idk how that would work, I’m not tech literate but it’s the obvious solution to me
I just want a superintelligent AI that always talks to me about the things I like to hear. That’s a bit far-fetched, but I also thought it was possible before: Algorithm Marketplace
This is not possible because sorting is done in the database, so adding a new sort option requires a database migration with new indexes, columns and updated queries. Not something that can be done with a simple plugin.
Ah that’s a shame
Actually scores are regularly calcuated from a scheduled task which runs in Rust. Yes the score caculation is currently implemented in SQL, but it could also be changed to a Rust implementation or a plugin. This would probably need some optimization so that the plugin only calculates scores for recent posts, not every single known post. In any case it would need someone with the time and motivation to implement it.
It could probably be done with a dedicated local client
I agree that scaled sort is not working ideally right now. However the issues you linked were opened before the feature was released, so they are not relevant anymore. You are definitely welcome to open a new issue about improving the implementation. The scaled sort logic is here, if you have any concrete suggestions how to improve it thats even better. And I would rather improve this existing functionality to make it work as expected, before tacking on an entirely new feature which may or may not work.
After giving it some more thought, I believe the best way to solve this would be through an API call to fetch metadata from today’s posts. This would include post ID, votes, and comments. By doing so, anyone can experiment on their client with a custom JavaScript Greasemonkey plugin to determine the best way to sort the posts. When a better sorting method is discovered, it could then be implemented in the backend.
There is already such an API endpoint which is available for mods and admins.
Slowly but surely, people are starting to realize that one of the main benefits of decentralization is that it gives power back to the users, and that we don’t need to be tied by the work done on the backend.
(Yeah, I was writing then about Mastodon, but the logic applies to all fediverse applications)
Personally, I believe the best way to address this issue is through the implementation of tags and custom feeds.
I expressed my enthusiasm for tags over in The Great Monkey Tagging Army discussion and have had a chance to ponder this some more - I still like the idea:
- It provides context and meaning to posts - just think of all the disambiguation pages on Wikipedia. Are you looking for Predator the film, the UAV, etc? Really only tags can help there - multitags or disambiguation? Or both? There are discussions to be had.
- It allows more depth to things like communities - I was just talking to someone who had been failing to find a place to talk about a specific British comic until they found !britishcomics@feddit.uk. With tags, I could have tagged the community with #dandy, #beano, #2000AD, #bunty, etc.
- It also provides a way for Lemmy to integrate better with the Fediverse as it doesn’t currently play well with services outside the Threadiverse.
My current thinking is leaning towards FediTags - a separate, federated plug-in service. So you could run it with Lemmy, Pixelfed, etc and it would bring together all the tagged content. You look up stonehenge + photographs and get pictures on Pixelfed, Mastodon and kbin. As a plug-in the changes in code needed to, for example, Lemmy would be minimal and, hopefully, a one-time thing. Development of FediTag could then take place completely separately.
Tagging would make sense to categorize posts within a single community. But you seem to suggest tags which are shared across communities. I dont really see the point of this, as communities themselves are already used for a global “tagging” of posts. So it would only duplicate that functionality.
Having shared tags across communities allows for better content discovery and curation based on specific interests or preferences. Users could also easily filter out or avoid posts related to triggering content like their phobias, traumatic events, or other sensitive topics by specifying certain tags they wish to exclude.
It certainly doesn’t help that Lemmy had and still has absolutely no sensible way to actually surface niche communities to its subscribers. Unlike Reddit, it doesn’t weigh posts by their relative popularity within the community but only by total popularity/popularity within the instance. There’s also zero form of community grouping (like Reddit’s multireddits) - all of which effectively eliminates all niche communities from any sensible main view mode and floods those with shitty memes and even shittier politics only. This pretty much suffocated the initially enthusiastic niche tech communities I had subscribed to. They stood no chance to thrive and their untimely death was inevitable.
There are some very tepid attempts to remedy this in upcoming Lemmy builds, but I fear it’s too little too late.
I fear that Lemmy was simply nowhere near mature enough when it mattered and it has been slowly bleeding users and content ever since. I sincerely hope I’m wrong, though.
@PurpleTentacle@sh.itjust.works https://sh.itjust.works/comment/4451602
Reddit isn’t going to magically get better though. Reddit is dying and will only get worse with time, continually shedding users as time goes by.
So making noise about lemmy, and making suggestions on how to prove it is a good thing, because unlike reddit, lemmy might actually get better with time.
If your issue is that you are only seeing popular stuff from “lowest-common-denominator” communities, then maybe stop browsing by all and only subscribe to the communities you are interested in?
stop browsing by all and only subscribe to the communities you are interested in
The issue with this is that subscribing to a large community results in seeing predominantly content from that community, overshadowing the smaller communities. All the communities I subscribe to would have to be about the same size.
I see. I think this technical issue doesn’t bother me as much as the fact that the communities I’d like to have are small in the first place. The Reddit mirror bots were solving this for me, but apparently I lost this battle.
Out of curiosity: how many communities would you have in your “subscribed” list?
Reddit was like this when it started. I was there when Dig was a competitor. It took a decade of constant posts by people who really loved their communities to really take off. And reddit, when it started, didnt have subreddits, it was closer to hackernews.
Not saying your arguments are wrong per-say. But it will take some time to get started. The good news is federation means different instances will come and go but the protocol means we dont need to stay on any one kind of server. Kinda like email. I use mastodon to talk with peertube creators and use lemmy to talk to other lemmy instances all the time.
multireddits
I think you are spot on here. We need this kind of functionality in order to keep up with instances. For example, lemmy.world/technology and lemmy.ml/technology, etc… could all bu there own multicommunities. Something like /mc/technologies or something saved on a per user perspective. It would transform lemmy overnight I think. multicommunities would make those niche topics have a great deal more action.
Mbin has that. They released a new version 2 weeks ago I’ll have a closer look soon, seems very promising
I can see the draw of subscribing to and blocking different keywords — adding another taxonomy to the existing communities. Mastodon and similar microblog projects are ahead in that regard, and not only in terms of muting or following hashtags. I would definitely welcome more user settings on Lemmy so we could curate feeds granularly on an individual level.
The freedom from advanced algorithms really demands that users be the wardens of whatever other sorting comes next, rather than Lemmy devs leaving things be or Meta-like algorithms creep in because of familiarity.
Any metric that ultimately depends on frequency and variety of posts is going to bury niche communities more or less by design.
Dunno what we need, honesdly, but I would venture that one thing that could help would be like, for exxample, StackOverflow’s “Unanswered” view, where you can check which topics have not garnered conversation attention. It would probably have to be tuned so that it specifically ignores 0-comment posts that are also links, because those tend to be reposts, news or stuff like that that’s easy to spam.
