May 18, 2024 - 9:15am
No company is quite so inseparable from the World Wide Web as Google, which made searching the internet an eponymous verb. The web made Google rich, too, but this week Google relegated it to a submenu. In a design of its next-generation home page that the company showed at its annual I/O developer conference this week, the demotion is quite clear.
Searching now returns a blancmange of content in special pull-out boxes, apps and features, some of it artificially generated. The days of lists of links are over — if you want to see web pages the tech giant now offers a “new ‘Web’ filter” that will refine your search to only see web pages. This may startle Generation Xers whose first taste of the online world was via a web browser, but the web has become a legacy format like the DVD.
The high hopes of the web really began three decades ago. In the spring of 1994, online communities consisted of services such as Compuserve, or bulletin boards, which were either expensive or difficult to use, as were the first internet searching tools like Gopher. But in the mid-Nineties, users launching the new Mosaic browser could soon discover how simple it was to produce these early mixed-media pages: now, anyone could publish anything.
This caused particular excitement among progressives. Here was a mechanism to bypass “big media” and the false consciousness it purportedly created for the masses. Radio had also started as a two-way communication system, but now the web offered publishing at a radically lower cost. It gave a pamphleteer the same profile as Condé Nast. The promise of the web explains the subsequent determination to keep the internet “free” or “open”.
The media’s fetishisation of the web was not always shared by the public. For example, in 2002 the BBC conducted a public poll to nominate the greatest living Britons, and Corporation staff insisted that the protocol’s co-inventor Sir Tim Berners-Lee should be one of the 100 nominees. To their dismay, he came 99th.
Today’s teenagers — and I’ve polled a random sample — neither know nor care what “the web” is. They were born into mobiles and social media, and see no interest in reviving it as a semi-ironic cottagecore medium, like the cassette tape. Web utopianism is strictly a Gen-X media phenomenon.
Join the discussion
Join like minded readers that support our journalism by becoming a paid subscriber
To join the discussion in the comments, become a paid subscriber.
Join like minded readers that support our journalism, read unlimited articles and enjoy other subscriber-only benefits.
SubscribeI have a web-server that has been running since 2008. It started out as a vanity website, written back in the days when a large number of websites belonging to individuals, before the monetization of the web. As that process developed, privacy issues began to rear up and my site went through a series of contractions, to the point were I almost shut it down. But it remained useful to me for several reasons. First, I had written a number of web applications that were very useful to me, personally. Writing a computer application for particular platforms takes a lot of work and must be constantly monitored for compatibility with continually evolving operating system changes. Web applications put that burden on browsers, who provide an applications programming interface which works (pretty much) across various platforms. My applications work on my native Linux machines, my wife’s Mac, our phones and tablets, etc. They enable me to interact with my own records, resources, references, stream my own music, transfer large files to and from wherever I am in the world – without Google, Amazon, Facebook, or any other corporate or government entity looking over my shoulder.
As security became an issue, I changed the site to require authorization to access most of it. The existence of most of it became invisible without authorization. But I left a small number of publicly accessible pages on the site. I had a pretty decent weather station I’d built and had online since I started the site. And I published some codes I’d written that a few people found interesting and led to some email discussions (and one exchange wherein a Chinese student tried to get me to solve his take-home Lisp programming final exam problem for him).
I regularly monitored my server logs – a record of the request traffic it receives. As the monetization of search took off my server traffic exploded. A large amount of it was the big search engines – both foreign and domestic. It got to be ridiculous. In any given period, only a very small amount of the traffic was from “real people” (me, my family and friends, and an occasional stranger steered there mysteriously by search); all the rest was search engines scraping the site – which changed rarely. There are methods one uses which are supposed to control them somewhat, and the big commercial domestic ones seem to obey them. Most of the foreign ones just ignore them.
But the biggest growth in traffic I saw from about the mid-twenty-teens was from hackers and from commercial operations looking for ways to exploit my site or data and sell it. I spent a lot of time learning how to track and classify these and the hackers. The emergence of geolocation techniques (which use multiple world-wide servers to triangulate actual latitude, longitude locations of IP sources based on transit delays) helped tremendously in this endeavor. China, Russia, the UK, and, curiously, locations around Washington, D.C., turn out to be the largest single sources of attacks on my U.S. located site. But there are waves of attack origin that temporarily roll through (lately, Ukraine, Hanoi, Tehran, Sweden and Hamburg, Germany have been prominent).
How my comment here ties in with this article is this: a while ago I relaxed my constraints on the big search engines. And what I discovered was that, while they came back around and sniffed at it, they just moved on without bothering to index it. They simply don’t care about individual sites like mine anymore. They would if I was posting ads on it. Or cross-linking to sites that posted ads. Or selling something. Or buying something. But just to put up information about various topics without any of that? Sorry, not interesting anymore. I am a non-entity to Goggle (in more ways than one). They just are not interested in the content anymore – not if will not be useful to generate clicks to their advertisers.
And I realized that this is what I’ve been noticing with search for quite sometime. It is very difficult to find non-monetized websites with search. There was a time when you could if you dove deep – meaning kept going through page after page of links. Eventually you’d get past the heavy advertising and find a few real, interesting topical pages. Now, after several pages the search engines simply say you’ve reached the end of their search results. The end of the Internet! It used to be a joke. Now it’s a reality. At a time when there has never been more websites online it has never been shallower.
I’ve seen very similar experiences. Well-established, long-standing websites with lots of useful content get blanked by Google – despite also having a lot of good back links – and it’s getting harder and harder to do deep searches which might mean scanning through the first 1000 results.
It’s not helped by Google trying to force website creators to jump through more and more technical hoops as it battles SEO-spammers. The result being that only the SEO companies are able to comply, and all the original content writers who specialise in their content and aren’t SEO-hacks get progressively demoted. And, as a result, it feels like Google ends up biased towards professional SEO sites which are monetised with lots of ads on them – which also happens to be very good for Google’s business.
It means there’s no point building a good-content website if you can’t also do the SEO. So authors and hobbyists end up migrating to Substack or Medium or a blogging platform, and small business then feel there is no point in having much more than a one-page company website because no-one will find it unless the business buys Google ads, and social selling works better anyway.
Google’s problem will be that with fewer authors, and with more content moving behind paywalls, AI ends up sampling itself constantly. It should provide an opportunity for someone to reinvent search (eg an index only of sites without advertising on them), but the resources needed for that are so huge now it’s unlikely to happen.
Try Perplexity Exceptional
AI for search misses the point that common uses for search include looking for very specific details or data, and searching for personal experiences and viewpoints, not just pseudo-expert answers.
It’s why it’s become increasingly common to add ‘reddit’ to Google searches to force it to find opinion and discussion not salesy answers. Or, when trying to understand detail and nuance, to search with time filters to see views before and after an event. Or to need deep search rooting around to find out complex details like trends in children learning the violin.
One current Google issue is that it think it knows what we want to find and so simplifies and constraints the results to make it ‘easier’. It’s probably effective for mobile phone use where search is simpler and the questions aren’t too deep and a quick top 10-20 answers are good enough. For anything detailed though, deep-searchers are in a constant battle with ‘the algorithm’ to get more info.
For all its faults, Reddit is still an absolutely incredible resource.
Go to any niche subreddit, and more than likely there will be an oracle of information that is helping others with their questions.
The upvote/downvote mechanism is still pretty robust, if you ask a question that has been answered many times they will just downvote you and expect you to find it in previous posts.
Anonymity is a double edged sword, but it does mean that experts can be active in this kind of forum without their institution or employer sanitising what they say!
Absolutely agree here. WWW is a pay-to-play environment and has been for a while.
With the advancement in LLM, It will become even more restricted, exclusive, and more pay-to-play.
Imagine a WWW where you only get the good information if you pay. Or the end of the Internet is you don’t. That is one of the reason the big three are spending billions.
It’s about those who spent billions to control it. getting paid for it.
Doesn’t matter if you are a boomer, gen-x, millennial, gen-z, centennial, alpha, gama beta, gen- next.
WWW is being taken over. It not a generational war, over who owns the web now. It’s a war over the have and have nots. Matter or matter nots. Message is clear. You don’t have millions or billions. You will be excluded.
In fairness, by 2008 Google was ten years old and Facebook had existed for four. Smartphones were launched around that year. The web was absolutely commercial by 2008.
Actually the idea there was ever a non-commercial time on the web is a bit of rose tinted glasses. The first banner ad supported sites appeared about a year after the ability to display images was added to browsers.
Final paragraph is particularly spot on.
Agree. I don’t have the answer either . in the case of Google I suppose you can’t blame a dog for barking. Even if other search engines tried to deliver deeper content -a là Berners-Lee how would it be funded? Ultimately we -the consumer- don’t want to pay- although we used to be okay paying a modest sum to access a library. Thinking it through..even if some public funded “Content Web” were created as soon as it started to draw meaningful traffic Google would kill it
Actually, Google (and search engines in general) are what made it un web-like in the first place. Before then, you could go to a site with a URL. Most websites had links to other interesting websites, often with similar or related themes. It was web-like because you would “surf the web” by jumping to pages strung together with no overall structure. You could even buy books such as “The Internet Yellow Pages” to have a directory of websites.
Adding a search engine means that it’s the opposite of web-like. You have a hierarchical structure and an index. You get a list of specific links. And most websites are dead ends in the sense that you can’t surf by following links to unrelated websites. That’s especially true of commercial websites.
I also don’t see what this has to do with Gen-X. They used the same Web as prior generations. The Web didn’t take off until millennials were being born, so they likely never surfed the web in the first place since there were search engines by then.
I am not a tech guru. I am a lay man. I have discovered that many times “progress” is not what it is, not what it claims. There’s always the hidden, unheard side. Good luck to humanity!
WWW is struggling, perhaps, due to the reliance on advertising to pay for it.
I use search engines a lot and it has become increasingly difficult to get to “knowledge” because of the need to wade through pages of advertising.
No idea if there is any solution.
A bit like Unherd these days then, eh?
What the what… How do you define the web… Hyperbole.
It’s defined. Go to, for example, w3 dot org. (I guess Unherd doesn’t allow links?)
They are talking about the www bit, obviously, rather than clicking a curated button. Rather like what happened to the extremely diverting habit of looking up something in an encyclopaedia to be replaced by a google search for something, a wikipedia page and then thats it. Whereas with an encyclopaedia there was the facing page and over, other topics on the same page, all without having to follow a link AND, most importantly, no screen flicker to stop your eyes unconsciously reading above, below and to the side of the subject you had opened the book for.
This is just patently not true. No, a Google UI redesign does not mean the WWW is over.
Hmm, I wonder if they are making a ‘metaverse’ scale mistake here, after all Zuckerberg was convinced that’s where we want to exist.
I prefer the autonomy and freedom of a web browser over an app, I don’t want my content served up by an algorithm that may or may not know what I want, but of course these companies want to control the end user and this move looks quite sinister in that respect.
But it’s quite simple, if I don’t like it I will use something else, and there will always be an alternative.
Google is not the web. Rather than the web being dead, I would say Google is just becoming a walled garden like AOL.
I rarely use Google to search anymore and prefer other methods of finding the information I want, although keywords are still important.
I also find that non US websites are generally better at not focusing on the FANG tech titans as being the only holders of data.
We are now having to use all those old skills we outsourced to Google for a decade or two and do the work ourselves again.
This is SO interesting. I did have a “feeling” that something was changing but could not identify why. It looks like going to the reference library to look into actual books might become a thing again.
I only use Google to find out opening times of places I want to visit,it’s useful for that. I’ve noticed the “changing narrative”,it used to be…the web is a thing of the air,it’s above and beyond earthly powers to restrain or control,it bestows Godlike freedom on its users….now I’m hearing….it’s just cables under the sea (except starlink,expensive), any competent military diver could cut a cable and have us all moiling around like a disturbed ant heap.
The Internet as we know it is going to get chopped up,portioned off,privatized,monetized. It will be like every human object where you are willing mirror your social status. Some of you will be in the 16th arr,and a lot of us will be up in the Goutte d’Or with the shouty Africans.
Some great, detailed comments here already, thanks guys and gals.
As a non-techie, but with a basic understanding of what’s involved – and at stake – here, i wonder how much the focus by the voracious giants on monetisation via ads and data is such a waste of their, and our, time? Obviously, clicks and revenues speak for themselves, but i suspect an awful lot of people just ignore, or block, ads.
Confession here: i’m pretty much allergic to ads. To give an example, i haven’t watched a terrestrial tv programme (apart from sport) in real-time for probably a decade now. I record everything of interest and just skip the ads x6/x12 so those ads have wasted their time, not mine.
Online, are we as humans just wasting an awful lot of our time and energy with all this monetisation? I know trade, in its broadest sense, is one of the foundation stones of civilisation but in the past, that meant face-to-face cultural exchange and to some extent, tolerance, for mutual benefit. Online, all of that’s lost. So a question: what are we losing that made us who we are, or were?
I’ve described some of the steps i take to avoid being drawn into the commercial sphere, and there must be plenty of others “resisting” in similar fashion. It’s the human element that interests me, but is this a losing battle in terms of overall humanity?
You are spot on about avoiding adverts. We also record everything we want to watch and then we can fast forward thru the adverts. I do pay a Youtube sub so I don’ t get adverts, $16/mth worth every cent as I use it a lot. I even place the screenview to one side so I can screen out the adverts happening along the left side of the page.
God moring, can someone give alternatives to Google as a search machine, please? Myself I am no expert in these things, I just dislike all the ads, bu need good and deep information.
Brave Browser is an excellent choice
DuckDuckGo, Startpage, Odysee. Just Goggle your question. Irony.
Try Perplexity. You ask it questions. It gives you answers
so surely the answer is to ditch Google and use other web browsers.
Undoubtedly. Plenty of good resources still out there on the www to de-google your life.
Exactly. Google is not a monopoly, they just want us to think they are. If you don’t use Google, Facebook, X, Instagram or TikTok your life won’t be over. You might even discover that there’s a real world with real people out there. Imagine.
“Google is currently erecting a wall between the searcher and the information they seek, using Generative AI, which the company believes creates more useful results such as summaries.”
And which, by way of how Generative AI is structured and taught, will give Google even more ability to censor, control and manage what users are able to access on the web.
The era of the open web is ending because the Left discovered that it was the ordinary masses on the Right who have used it to subvert its gatekeeping, and that simply will not do.
I guess that’s important rather say we needed controlled web two decades ago, when google techies we planning for hot pool work spaces
This sounds to me to be the truly important point. “They” (the OverClass, Our Betters, Those With Good Intentions, whatever) are most furious that so many of “Us” refuse to line up and cooperate (for the vaccines, to buy an EV or a heat pump, a sex change, whatever). They’ve tried moving the goal posts, then changing the rules; now they’re going to try switching us all to a new game. The first rule of the new game will be “Rules may change without notice”.
I think at this point we’re supposed to say “Thank You” for relieving us of the burden of having to make our own decisions.
Why did this have so many down votes?
Google is reimagining it’s results so as to not return links, just “the answer”. They regularly change the way that their search works so as to reduce major site traffic by 90% overnight. They have been shown to inflate the price of their paid content.
Look at the ideology baked in to google to Gemini, often to hilarious results. Is that not concerning gatekeeping?
It’s not really about the web, or not – it’s about open systems v closed proprietary ones, and perhaps the death of a vision of the future. Remember “information wants to be free”. Also it’s a change in the role the internet plays in our lives – from the most amazing information source the world has seen to tailored pap served up to the masses.
Looked at now, the great innovation of the internet was getting users to create value for providers completely free of charge, either in the form of content, or sellable information about themselves and their friends. And of course the money to be made by hooking into human pleasure systems to addict them to low level trash – much of it churned out by themselves.
DARPA, NSA and CIA heavily bankrolled Google. Not just bankrolled, but guided and directed its development.
And now we are “Shocked! Shocked to find gambling is going on here!”
Too much negativity, especially in the comments. And Googlophobia. I use Google immensely on a daily basis. Wikipedia is often on top, or stanford.edu and sites like that, but also (with precise querying your topic of interest) the internet archive, and sites that host academic articles. Then there is searching for all the people / authors that have my interest. Google brings them all up, with the places where they write and their social media (the latter regularly mistaken, unless you add to the name search the proper title, like Prof., or a serious subject s/he writes about). I think I discovered all Unherd’s writers of my interest that way.
If you read multiple languages, searching specific with French or German words for instance may deliver additional exquisite results. (You don’t even need to know other languages if you installed the Google translate extension to your Chrome or Edge browser). And oh my! I didn’t even mention YouTube (also Google, right?) where you can find so many pearls among the pulp. Besides, if you search directly in youtube, after a while it presents you all your favorites, especially when you subscribed to channels. That is not to say you should only listen to what you like – here again: use it wisely.
If you can still use Google that way in the future then I’m ok with their change. I understand they need to do something for the spoiled brats. They are still an information giant, for real info. Maybe one has to be a good and not too naïve liberal to use Google in a way that makes it deliver what you are looking for. Most of the commentary sounds to me more like half-baked conspiraphilia than reason. But, then again, it may just be that some people don’t know how to query that thing in the best possible way.
The article in itself was interesting.
Google has manipulated its search engine results to drive left wing ideology. Their new AI will be worse.
Google just pushes the coterie of monopoly-media that is a source of income for google. Alternative views are either buried or dismissed entirely. The top 20 articles on a search of “Floyd Fentanyl” is dominated by the usual suspects, BBC, CBC, Reuters, CNN, NBC, etc. all reporting in unison the established narrative. They even deny that Fentanyl arrests the pulmonary muscles and can lead to asphyxiation, while this is well known to medical science.