In an age of LLMs, is it time to reconsider human-edited web directories?
Back in the early-to-mid '90s, one of the main ways of finding anything on the web was to browse through a web directory.
Lycos, Excite, and of course Yahoo all were originally web directories of this sort.
These directories generally had a list of categories on their front page. News/Sport/Entertainment/Arts/Technology/etc.
Each of those categories had subcategories, and sub-subcategories that you clicked through until you got to a list of websites. These lists were maintained by actual humans.
Typically, these websites also had a limited web search that would crawl through the pages of websites listed in the directory.
By the late '90s, the standard narrative goes, the web got too big to index websites manually.
Google promised the world its algorithms would weed out the spam automatically.
And for a time, it worked.
But then SEO and SEM became a multi-billion-dollar industry. The spambots proliferated. Google itself began promoting its own content and advertisers above search results.
And now with LLMs, the industrial-scale spamming of the web is likely to grow exponentially.
My question is, if a lot of the web is turning to crap, do we even want to search the entire web anymore?
At some point, does it become more desirable to go back to search engines that only crawl pages on human-curated lists of websites?
And is it time to begin considering what a modern version of those early web directories might look like?
Sounds like you may enjoy https://en.m.wikipedia.org/wiki/Gemini_(protocol) if you haven’t installed a browser and tried it.
@ajsadauskas @degoogle definitely something o be thinking about. More and more I’m using my followed hashtags, mastodon lists, and links to resources other people provide rather than just finding useful things in search results. But the big gap is still when I want to find quality info on a new topic. Cannot trust any of the damn results searching for how and how often to clean my kid’s new aquarium, for example. So much LLM and SEO crap info.
@ajsadauskas @degoogle I guess the problem though is how you make sure they are actually maintained by a human acting in good faith. The way community Facebook groups meant to be for this kinda thing get spammed by likely fake businesses doesn’t give me hope
I suppose any measures at all would cut out a massive number of spam pages already.
@joannaholman @degoogle Good point.
If it were run as a private company, I think the solution might be just to pay actual humans as employees.
If it’s a community-run project, the challenge would be to come up with a robust moderation system…
@ajsadauskas @joannaholman @degoogle maybe a mix of wikipedia and search engine would be nice. WikiSearch?
@ajsadauskas @degoogle DMOZ was once an important part of the internet, but it too suffered from abuse and manipulation for traffic.
For many DMOZ was the entry point to the web. Whatever you were looking for, you started there.
Google changed that, first for the better, then for the worse.
@ajsadauskas @degoogle Webrings! Bring back Webrings!
I had this exact thought earlier today. Either curated directories, or a ground-up, vetted search engine that only pulls from pre-screened sources.
Reddit and Lemmy are supposed to be what you want: link aggregators.
We’re supposed to link to sites and pages and people vote on how good they are in the context of the sub community topic.
Of course, then Ron Paul happened, and now it’s just memes and Yank politics so… maybe deploy Lemmy and turn off comments.
I think you are mostly right, except Lemmy and reddit are not organized.
I used them and contributed to links as well - it was quite a rush to see a contribution accepted because it felt like you were adding to the great summary of the Internet. At least until the size of the Internet made it impossible to create a user-submitted, centrally-approved index of the Net. And so that all went away.
What seemed like a better approach was social bookmarking, like del.icio.us, where everyone added, tagged and shared bookmarks. The tagging basically crowd-sourced the categorisation and meant you could browse, search and follow links by tags or by the users. It created a folksonomy (thanks for the reminder Wikipedia) and, crucially, provided context to Web content (I think we’re still talking about the Semantic Web to some degree but perhaps AI is doing this better). Then after a long series of takeovers, it all went away. The spirit lives on in Pinterest and Flipboard to some degree but as this was all about links it was getting at the raw bones of the Internet.
I’ve been using Postmarks a single user social bookmarking tool but it isn’t really the same as del.icio.us because part of what made it work was the easy discoverablity and sharing of other people’s links. So what we need is, as I named my implementation of Postmarks, Relicious - pretty much del.icio.us but done Fediverse style so you sign up to instances with other people (possibly run on shared interests or region, so you could have a body modification instance or a German one, for example) and get bookmarking. If it works and people find it useful a FOSS Fediverse implementation would be very difficult to make go away.
@Emperor
This this this! Some kind of service that would sit alongside a fedi instance and serve as a community directory.
@ajsadauskas@Emperor @ajsadauskas that’s Lemmy?
Although Lemmy is called a link aggregator it is really just a kind of web forum and nothing like a social bookmarking service.
@Emperor @ajsadauskas I’ve been thinking about this myself lately - but I had wondered how a curated directory might scale, I hadn’t considered federated social bookmarking and honestly that sounds like a brilliant solution. I’d love to see something like that happen, maybe even contribute
As the links show, Relicious/Fedilicious has been on my mind a while and I have been mourning the loss of Delicious for a long time. However, the above got me jotting down some notes.
It should be doable. I haven’t had a root through PostMark’s code but it might be they have done the bulk of the work already and it just needs a multiuser interface bolting on top of it.
@Wren @Emperor @ajsadauskas Back in the day people’s web sites had a links page and if their site was good it was always worth looking at what they listed as worthy links. I still have one but it’s out of habit rather than being useful. Might rethink now tho.
Oh indeed there are services out there that do something similar to Delicious, but I put a lot into that site only for it all to disappear due to the whims of some corporate overlord and I am not doing that again. What I am looking for is an easy Fediverse solution so my data is never lost again. Postmarks is definitely getting there but as a single-user service it isn’t quite what I am looking for.
The tale of the internet has been curation, and I would describe it a little differently.
First we had hand made lists of website (Yahoo directory, or we had a list of websites literally written in pen in a notebook saying “yahoo.com” and “disney.com”).
Then it was bot-assisted search engines like Google.
Then there was so much content we didn’t even know where to start with Google, so we had web rings, then forums, then social media to recommend where to go. Then substack style email newsletters from your chosen taste makers are a half-step further curated from there.
If that is all getting spammed out of existence, I think the next step is an AI filter, you tell the AI what you like and it sifts through the garbage for you.
The reasons we moved past each step are still there, we can’t go back, but we can fight fire with fire.
@ajsadauskas @degoogle i love this idea, i’m going to start my own web directory.
Do it!
Then federate it.
@ajsadauskas @degoogle I used to be one of those human editors. I was the editor of Scotland.org from about 1994 to about 1997, back in the days when it was exactly one of those hierarchical web directories – with the intention of indexing every website based in Scotland.
@ajsadauskas @degoogle having said that, the patents on Google’s PageRank algorithm have now all expired, and a distributed, co-op operated search engine would now be possible. Yes, there would be trust issues, and you’d need to build fairly sophisticated filters to identify and exclude crap sites, but it might nevertheless be interesting and useful.
Main problems are:
-
Link rot
-
Sneakily inserted sponsored links
@Moonrise2473 @ajsadauskas
3. Infinitely growing list of categories.
4. Mis-categorisationi remember learning HTML (4.0) and reading that you should put info in a <meta> tag about the categories your page fits in, and that would help search engines. Did it also help web directories?
-
@ajsadauskas @degoogle it sounds a bit like Kagi‘s Small Web initiative and search. have you seen it? https://blog.kagi.com/small-web
I don’t know if this is the intent of their small web effort, but the first impression I got just now when I clicked through and saw this was, “gross”.
Scummy looking/feeling marketing hustle is a huge turn off.
@ajsadauskas @degoogle What we need to do is re-visit the GnuPG philosophy of building rings of trust. If one emerges with enough people proven to provide quality aggregators/summarizers then we can start to depend on that, or those.
What’s to say we won’t have AI-curated lists and directories? That way we don’t have to deal with link rot and the like. I think the issue is the algorithms used for search. We need better ones, better AI, not more frivolous human labor.
