In an age of LLMs, is it time to reconsider human-edited web directories?
Back in the early-to-mid '90s, one of the main ways of finding anything on the web was to browse through a web directory.
Lycos, Excite, and of course Yahoo all were originally web directories of this sort.
These directories generally had a list of categories on their front page. News/Sport/Entertainment/Arts/Technology/etc.
Each of those categories had subcategories, and sub-subcategories that you clicked through until you got to a list of websites. These lists were maintained by actual humans.
Typically, these websites also had a limited web search that would crawl through the pages of websites listed in the directory.
By the late '90s, the standard narrative goes, the web got too big to index websites manually.
Google promised the world its algorithms would weed out the spam automatically.
And for a time, it worked.
But then SEO and SEM became a multi-billion-dollar industry. The spambots proliferated. Google itself began promoting its own content and advertisers above search results.
And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.
My question is, if a lot of the web is turning to crap, do we even want to search the entire web anymore?
At some point, does it become more desirable to go back to search engines that only crawl pages on human-curated lists of websites?
And is it time to begin considering what a modern version of those early web directories might look like?
I used them and contributed to links as well - it was quite a rush to see a contribution accepted because it felt like you were adding to the great summary of the Internet. At least until the size of the Internet made it impossible to create a user-submitted, centrally-approved index of the Net. And so that all went away.
What seemed like a better approach was social bookmarking, like del.icio.us, where everyone added, tagged and shared bookmarks. The tagging basically crowd-sourced the categorisation and meant you could browse, search and follow links by tags or by the users. It created a folksonomy (thanks for the reminder Wikipedia) and, crucially, provided context to Web content (I think we’re still talking about the Semantic Web to some degree but perhaps AI is doing this better). Then after a long series of takeovers, it all went away. The spirit lives on in Pinterest and Flipboard to some degree but as this was all about links it was getting at the raw bones of the Internet.
I’ve been using Postmarks a single user social bookmarking tool but it isn’t really the same as del.icio.us because part of what made it work was the easy discoverablity and sharing of other people’s links. So what we need is, as I named my implementation of Postmarks, Relicious - pretty much del.icio.us but done Fediverse style so you sign up to instances with other people (possibly run on shared interests or region, so you could have a body modification instance or a German one, for example) and get bookmarking. If it works and people find it useful a FOSS Fediverse implementation would be very difficult to make go away.
@Emperor
This this this! Some kind of service that would sit alongside a fedi instance and serve as a community directory.
@ajsadauskas
@Emperor @ajsadauskas I’ve been thinking about this myself lately - but I had wondered how a curated directory might scale, I hadn’t considered federated social bookmarking and honestly that sounds like a brilliant solution. I’d love to see something like that happen, maybe even contribute
@Wren @Emperor @ajsadauskas Back in the day people’s web sites had a links page and if their site was good it was always worth looking at what they listed as worthy links. I still have one but it’s out of habit rather than being useful. Might rethink now tho.
As the links show, Relicious/Fedilicious has been on my mind a while and I have been mourning the loss of Delicious for a long time. However, the above got me jotting down some notes.
It should be doable. I haven’t had a root through PostMark’s code but it might be they have done the bulk of the work already and it just needs a multiuser interface bolting on top of it.
@Emperor @ajsadauskas that’s Lemmy?
Although Lemmy is called a link aggregator it is really just a kind of web forum and nothing like a social bookmarking service.
Pinboard and TinyGem come to mind.
Oh indeed there are services out there that do something similar to Delicious, but I put a lot into that site only for it all to disappear due to the whims of some corporate overlord and I am not doing that again. What I am looking for is an easy Fediverse solution so my data is never lost again. Postmarks is definitely getting there but as a single-user service it isn’t quite what I am looking for.