Roadworks on the open web - who builds the way forward?
The mobile web got walled off, but as regulation brings a reset publishers have a chance to find a new path
In 2004 in an interview with Playboy, Co-CEO Larry Page said of an emerging Google;
“Most portals show their own content above content elsewhere on the web. Google conscientiously tries to stay away from that. We want to get you out of Google and to the right place as fast as possible. With news, we’re not buying information and then pointing users to information we own. We…point the user to other websites” 1
This month at the Twipe Digital Growth Summit, Varun Shetty as the Head of Media Partnerships for OpenAI said of their relationship with publishers;
“We think there’s an opportunity here to drive significant incremental traffic from new audiences…we’ll have to see if that is valuable enough to partners to remain opted-in” 2
These positions may seem the same, that publishers will benefit from platforms through traffic and new market penetration. In truth the context in which the quotes are made has changed beyond recognition.
In the twenty years since Page was in Playboy the ecosystem within which platforms and publishers engage has been steadily constructed to favour the tech side. Through the skilful application of market power and competitive advantage, three players have built the rules of digital media that we play by today.
Between them Google, Apple and Meta have built hugely successful businesses as technologists, as aggregators and as gatekeepers. As a result, publishers across the world of news and information are facing a commercial endgame.
Looking back over the twenty years there are a handful of killer strategic moves that have created that dominance. The relationship with content creators that started out as symbiotic (you give access to content, we deliver traffic to brands) is now one of commoditised servitude. How did it happen?
Firstly Apple and Google delivered the core tech for a digital and latterly mobile world.
They built iOS and Android operating systems which dictate how software for smartphones is built. They developed Safari and Chrome browsers that control how websites are shown.
Google developed the dominant search engine and then did deals to make sure it was also the default on Apple phones, and then bought YouTube and helped it to become the dominant video player.
All of this foundational technology operated at the most granular level of content; indexing a single article or short video. The traditional publisher or broadcaster models of consolidated editions and scheduled programming respectively were replaced.
A search engine with global aggregation across all content created a tremendous monopoly position for Google. The same playbook helped Meta (nee Facebook) capture discovery and engagement in social media, with links and images linked by a social graph that developed a scale of recommendation engine and daily usage that no individual publisher could match.
Originally, the decision to distribute professional content on a platform that publishers did not own themselves was in return for traffic that returned to the individual articles on publisher sites. But quickly the habit of visiting publisher sites in order to find their content was replaced - why visit many when you could get all you needed from the aggregator site?
Shareholders expected hyper-growth in audiences to translate into growing monetisation. In 2007 Google bought DoubleClick, creating a model where they controlled both demand and supply of ad inventory and all the information in between. Facebook in turn tagged and pixelled the internet to build the Facebook Audience Network, where information from any website could be used to hone the targeting and drive up the yield of Facebook ads.
Ad operations teams knew that they needed Google to maximise ad revenue because of their grip of the ad exchanges. Editorial, marketing and product teams knew they needed Facebook pixels for their analytics and optimisation. As a consequence publisher control of selling advertising against their own content continued to be eroded.
There needed to be consent from users to be traded in this way, and more data to feed growth, so there was expansion into email and messaging (Gmail, WhatsApp, Messenger) and maps. Data permissions expanded to cover the whole network – one of the benefits of the Meta rebrand was a global permission for all products.
These data assets have now provided the stepping stone to the next wave of innovation in AI. Google acquired the UK innovator DeepMind in 2014, Facebook and Apple have steadily built and acquired technologies for the development of artificial intelligence. All three have the baseline of massive data assets and computing power to apply AI to the worlds information, which they have by virtue of the ecosystem they worked to build.
All of this was executed at breathless pace with innovation and delivery skills, in rational service of the goals of their respective shareholders.
Bit by bit though, publisher access to the Open Web was closed. In 2004 most traffic was desktop and direct, and search engines were incremental. In 2024 to succeed a publisher must play by the rules of the dominant gatekeepers in search (Google, via Search and YouTube), social (Meta via Facebook and Instagram) or even the smartphone device itself (Apple and Google via OS and AppStore rules).
In twenty years publishers were outflanked and as a result utterly lost control of distribution for the digital and device era. Without a self determined route to market there is a future as content farms for the platforms, or extinction.
Fortunately this endgame has not escaped the attention of government and their regulators. They have finally started to push back hard on dominant market power in many concurrent legal cases. I highlighted this acceleration in a previous article here
Most recently, the latest filing from the Department of Justice in the Google case 3 laid out the structural remedies required to meaningfully address the stranglehold that has been put in place.
That DOJ filing requires remedies for default search dominance and tying that access to tools like Chrome and Android.
It requires remedies for data sharing. It requires optouts for publishers to prevent Google building AI products using the index information it gathers from search indexing.
It requires solutions to tackle Google’s incredible market power and competitive harms in digital advertising. And it wants those answers by March 2025.
In short, it recognises the world as it has become and seeks to restore balance to the market for digital media across discovery, ad ops and monetisation.
I firmly believe this period of reassessment will provide a tremendous strategic opportunity for publishers, well over and above a temporary bump on ad yields and traffic. It raises questions like:
If the mechanics of Google’s framework for dominance is are to be unbundled, what replaces it?
If the precedents of that case create legal protections preventing the extraction of content from the open web without publisher’s consent, what does that consent allow?
If the regulators in the UK, Australia and beyond take this to its logical conclusion in all major markets, what other structural positions are in play and how does that shake out?
The battle between the platforms and the regulators will play out, but might take five to ten years to move from consultations to seeing the results of market correction in a number of interconnected multi billion dollar industries.
The work for publishers now is in identifying what the future can look like and how this might best be shaped by their influence, industry and innovation. The opportunity is immense.
https://storage.courtlistener.com/recap/gov.uscourts.dcd.223205/gov.uscourts.dcd.223205.1052.0_1.pdf