Search engines

If this is the first time you’re hearing about it, you’re not alone. Despite the significance of the changes, these latest rules are the result of industry codes, which differs to regular legislation. These codes don’t go through parliament. Instead, they’re developed by the tech industry and registered by the eSafety commissioner in a process called co-regulation. On one hand, this can be good: it can allow for more flexibility or technology-specific detail that is less appropriate in legislation. On the other: it creates risk of industry co-option, and by bypassing parliamentary process, can give an enormous amount of power to an unelected official (in this case, the eSafety commissioner).

Greens senator David Shoebridge has called the implications of age verification for search engines “staggering” and noted that “these proposals don’t have to go through an elected parliament and we can’t vote them down no matter how significant concerns are. That combined with lack of public input is a serious issue.”

The age verification policy development process has been littered with blunders that make a mockery of meaningful consultation and evidence-based policy development. It is particularly striking that these codes were drafted before the completion of the government’s $6.5m trial into the efficacy of age assurance. Later, the trial’s preliminary findings conceded the technology is not guaranteed to be effective, and noted “concerning evidence” that some technology providers were seeking to collect too much personal information.

While a government-commissioned survey on the teen social media ban found overwhelming support in theory, it also found most people have no idea what that means in practice, with many uncomfortable with the methods it might entail – such as biometric face scanning or handing over your credit card details. And while there was much fanfare around the social media ban, it’s not clear there is a social licence to extend this approach to search engines and beyond. It seems many people may be unpleasantly surprised.

Many users of web search engines have been complaining in recent years about the supposedly decreasing quality of search results. This is often attributed to an increasing amount of search-engine-optimized but low-quality content. Evidence for this has always been anecdotal, yet it’s not unreasonable to think that popular online marketing strategies such as affiliate marketing incentivize the mass production of such content to maximize clicks. Since neither this complaint nor affiliate marketing as such have received much attention from the IR community, we hereby lay the groundwork by conducting an in-depth exploratory study of how affiliate content affects today’s search engines. We monitored Google, Bing and DuckDuckGo for a year on 7,392 product review queries. Our findings suggest that all search engines have significant problems with highly optimized (affiliate) content—more than is representative for the entire web according to a baseline retrieval system on the ClueWeb22. Focussing on the product review genre, we find that only a small portion of product reviews on the web uses affiliate marketing, but the majority of all search results do. Of all affiliate networks, Amazon Associates is by far the most popular. We further observe an inverse relationship between affiliate marketing use and content complexity, and that all search engines fall victim to large-scale affiliate link spam campaigns. However, we also notice that the line between benign content and spam in the form of content and link farms becomes increasingly blurry—a situation that will surely worsen in the wake of generative AI. We conclude that dynamic adversarial spam in the form of low-quality, mass-produced commercial content deserves more attention.

 The ancient Library of Alexandria aimed to create a universal collection of scholarship, indexed using the first library catalog, the Pinakes. We're working toward that same goal, but making it completely open:

• Our data is free and reusable, available via bulk download or API,
• our code is fully open-source, and
• we're governed by a sustainable and transparent nonprofit.

We believe the global research system is one of humankind's most beautiful creations. OpenAlex aims to make that whole beautiful creation available to everyone, everywhere.