The business of search engines : understanding how Web advertising, partnerships, and the race for market dominance affect search tools and search results – 1

The business of search engines : understanding how Web advertising, partnerships, and the race for market dominance affect search tools and search results – 1 – Cover Story

Rita Vine

Introduction

Now more than ever, users rely on the Internet for information and news. With well over 8 million distinct websites (2) and billions of individual Web pages, finding high-quality information is increasingly challenging. Providers of information and services know that their website is a key component of their business and that, in a crowded information marketplace, searchers must be able to find it using search engines.

Search engine advertising has grown tremendously in the past two years, and prospects for continued growth are strong. (3) Information providers and marketers know that Web users seek information on the Web prior to making a major purchase or information decision, (4) and that users rely heavily on commercial search engines for most of their searches.

Why Is Search So Hot?

Web search now represents a significant portion of Web activity. Google searches average 250 million searches per day, and the total daily number of Web searches is estimated at well over 600 million. (5)

At least a portion of searching is for products or services that the searcher will eventually purchase. Research has shown that higher-income users spend more time on the Internet and buy more online. (6) This marketplace of high-income earners is intensely attractive to marketers and much harder to isolate in traditional media such as TV or magazines.

Technological advances have enabled advertisers to track the success of their Web-based ad placements. The availability of this technology–along with specialized ad-buying programs where payment is made only if a link is clicked–has enabled advertisers to ensure greater return on investment of their Internet ad purchases in ways not offered by traditional media.

Research has shown that brand advertising works on the Web. (7) Initially it was used as an alternative advertising medium by a few early adopters who placed banner ads on search engine pages. Now many more advertisers, including many small businesses, have embarked on advertising purchases after several major players-and probably their competitors–adopted the medium as a main marketing stream.

Advertisers know that they fare better with search engine ads than without them. In a survey conducted during fall 2003 with 20 online advertisers, half said that paid advertising in search engine results listings made visible, sometimes critical differences to their traffic. (8) In the advertising industry, where return on investment is often elusive, that kind of claim can make companies race to bid for search engine keywords.

What Business Are Search Engines In?

Searchers believe that search engines are in the search business. They count on free, commercial search tools to help them find the relevant information they want quickly and easily, regardless of type of question, complexity, or language. Search engine companies focus their own marketing initiatives heavily on perpetuating this idea among searchers, asserting that they deliver the most relevant or comprehensive information and ensuring that the searcher returns to conduct additional searches. (9)

In reality, commercial search engines are in the advertising business. They earn the vast majority of their revenue by delivering context-sensitive advertising using a variety of means, but principally by leasing search keywords to purchasers. Virtually all commercial search engines (Google, Alltheweb, AltaVista, Teoma, MSN, Lycos, and WiseNut-Gigablast is the current exception) and major portals (About.com, Yahoo!) rely on this model for most of their revenue. (10) And of those, Google remains the only search engine that still keeps paid results out of its main listings.

The Big Three: Traffic, Relevance and Monetization

Commercial search engines require three key elements to ensure ad placement success. Traffic represents the flow of Web users to a search site. The search site must attract as much traffic as possible in order to maximize the possibility that some of that traffic will turn into a revenue-generating activity. Relevance represents the capacity of the search engine to deliver meaningful results to satisfy the user’s keyword query. Relevance is made possible by the creation of mathematical algorithms. When executed, these algorithms define how search results are ranked for presentation to the user. Relevance algorithms vary across different search engines and are regularly tweaked in order to improve the user experience. For example, Google uses PageRank[TM], a set of algorithms that rank order results by the number of links to those pages. (11)

To improve Google’s relevance and to combat the negative effects on relevance of a robust search engine optimization industry, some tweaking to the PageRank[TM] algorithm is used in addition to the raw link analysis. For example, some linking pages receive greater weight than others in the algorithm. Other search engines use a variation of link analysis methods and may combine link analysis with other ranking methods, including frequency of occurrence and proximity of words to each other. (12)

Monetization refers to the act of converting the all-important traffic into revenue for the search engine. Monetization can occur in many different ways. When search engines deliver ads to search results pages, advertisers pay fees to the search engine or their designated ad-feed partner for every ad impression that is delivered. If the searcher clicks on the ad link (“clickthrough”), additional revenue may accrue to the search engine.

The Basics: Keyword Buying

All search engine advertising purchases start with keyword buying. The advertiser purchases–leases, really–one or more keywords or key phrases that the advertiser believes searchers will use when searching for specific products or services. For example, an online video store may purchase the keyword movie or film or actual film titles such as harry potter, or a combination of any of these. Keyword buying enables the ad buyer to display a URL link when the searcher enters one or more of the leased keywords into the search engine. Some contracts may enable the purchaser to pay only for click-through. Contracts typically stipulate a time period, but they may also stipulate the number of impressions that will be delivered. An impression represents a single display of the advertiser’s URL or banner on a user’s screen.

Paid Search Deliverables

In paid inclusion programs, search engines and their ad-feed partners guarantee that their search engine will list pages from the advertiser’s website in its index. However, paid inclusion typically does not guarantee that the advertiser’s pages will rank high. At this writing, search engines Inktomi, AlltheWeb, and AltaVista offer paid inclusion programs.

Paid placement programs, by contrast, generally guarantee that a link to the advertiser’s URL will be delivered in the search results on a matched keyword or keywords. Location of the delivered link generally governs the fees, so advertisers will pay more to be placed higher up the page in the search results.

Pay-per Options

After the advertiser’s link is delivered to the page, additional gradations of monetization are possible based on whether the link is clicked on or otherwise processed by the searcher. Advertisers have a variety of pay-per options, which escalate in price as the deliverable moves closer to an actual sale. Pay-per-impression enables advertisers to pay based on how many users were served their ads. In this model, users do not have to click on the ad for monetization to occur. In a pay-per-click model, advertisers pay agencies and/or media companies based on how many users clicked on an online ad. Pay-per-lead enables advertisers to pay only for each “sales lead” generated. For example, an advertiser might pay for every visitor who clicked on an ad or site and proceeded to complete a form. In pay-per-sale, advertisers pay based on how many sales transactions were generated as a direct result of the ad.

Keyword Bidding

Because so many advertisers want to lease the same keywords, a mechanism of keyword bidding has developed. Keyword bidding is the process of bidding on keywords for specific rankings in search engines. The more you bid, the higher your site will rank in the search engine you have selected. Google uses keyword bidding as a partial determinant of the placement of ads in its right sidebar. In a sponsored links list, higher-bidding advertisers get improved position at the top of the list, although the ranking formula also includes a popularity component.

Contextual Search

As the search engine marketing arena becomes more crowded, advertisers are seeking ways to improve relevancy. Contextual search is a process that drives selected paid search results by user behavior and perceived relevance as opposed to strict keyword matching. Google’s AdSense (13) program is an example of contextual search: the program places ads on pages of the websites that sign up for the program, and the ad selection is contextually based on what Google believes the page to be about. Overture’s ContentMatch (14) is a similar program.

How Paid Listings Affect Search Results

With the exception of Google, which separates ads completely from its main search results, all the major search engines have, to a greater or lesser extent, embedded paid listings in their main search results page. Some paid placement results are separated from algorithmically generated results and are accompanied by headers such as “Partner Sites” or “Sponsored Links.” Paid listings may also be embedded in the actual Web search results: practices vary from search engine to search engine.

The presence of paid listings in search results may also vary depending on the searcher’s keywords. If a searcher uses keywords that are likely to attract advertisers who lease them, paid listings are more likely to appear in search results. As a rule of thumb, the more commercial the search keywords, the more likely the search is to produce paid listings. A search for the keywords “network printers” is far more likely to produce paid results than a search for the keywords health “canada aboriginal.” Keyword searches in technology, entertainment, and publishing topics are particularly prone to paid placements, (15) but the search engine industry is doing its best to extend the reach of paid placement, particularly in the area of local search (e.g., pizza in Toronto, hairdressers in Philadelphia).

Over time, it is arguable that paid listings, particularly paid placements (because they appear most prominently on the results page), can have great impact even on unpaid-for (or “pure”) search results. Much like traditional advertising, (where brand awareness and eventually brand allegiance emerges from a combination of advertising and peer adoption), persistent viewing of paid listings inevitably creates greater awareness of those paid listings and their brands. With greater awareness comes the likelihood that those who create Web pages will link to those paid listings simply because they have seen them many times and can remember them. Many excellent but unpromoted search tools never appear on these link lists. Sites that have been part of a search engine marketing campaign are linked simply because they are well known.

As linkage to popular, well-known sites grows, even Google’s PageRank,[TM] which ranks pure search hits by the number of other pages that link to them, has the cumulative effect of preferring what is popular. Advertising is an important factor in creating popularity, and the reach of paid placement extends even to pure search tools like Google that rely on a link analysis algorithm for ranking. As the persistent presence of paid listings creates greater awareness of particular websites, Web page creators link to these popular sites. As a result, the sites’ PageRank[TM] increases, and the sites eventually rise higher in Google’s search results. Moreover, as a larger number of popular sites climb higher in search results, many excellent informational resources (that languish in relative obscurity because they lack the funds for paid listings) crawl even further down the list of search results and off the searcher’s radar entirely.

The Problem with Meta-search

Many Web searchers use meta-search engines–tools that send a query to several different search tools, retrieve the first several search results from each tool, remove the duplicates, and present the results in a rank-ordered list. Meta-search tools are problematic because they can capture paid listings served up by their search tool partners, but in the aggregation and display process they may strip these results of any indicators that they are paid links. (16)

The Role of the Federal Trade Commission

Many Web users wonder about the capacity of regulatory agencies to mandate the enforced disclosure on websites that some searched-for content is paid for. Since all the major search engines are produced by companies owned and operated in the United States, it is the U.S. Federal Trade Commission (FTC) that attempts to regulate the activities of search properties. The FTC treads a fine line between watchdog and cheerleader, attempting to protect consumers while not unduly restricting company growth and profitability.

Its practices with search engines illustrate its challenge. After becoming aware that search engines were inserting paid results into their main search listings, the FTC sent a warning letter to all the major search engine properties (17) (except Google, because it didn’t insert ads into search results), instructing them to disclose the insertion of paid listings or risk being in violation of key sections of the FTC Act.

This letter prompted the search engines to disclose information on how paid listings are included in search results, but just enough to avoid legal action. The resulting disclosure practices meet the FTC’s criteria but are strikingly unclear to most Web searchers. (18) The onus is clearly on the user to learn about search engine practices.

The Major Players in Web Search

At the end of 2003, the first tier of search engine advertising consisted of two large players–Google and Yahoo! Both companies control huge amounts of traffic to their websites and both control their own monetization mechanisms. Google controls its own paid placement services through its AdWords and AdSense programs, which appear not only on the Google website and major partners like AOL, but on thousands of other websites that serve up Google ads through the AdSense algorithm. (19) Yahoo!, by contrast, built its powerhouse capacity principally through acquisitions. In late 2002, it acquired Inktomi, a provider of spidered Web databases and the originator of pay-for-inclusion and pay-for-spidering options. Yahoo!’s recently completed acquisition of paid-placement search tool Overture gave Yahoo! the capacity to compete directly with Google on monetization and further enabled Yahoo! to directly own its monetization properties. Overture already owned search properties FAST Search and Transfer (creator of the Alltheweb search engine) and Altavista, which provided the additional enhanced traffic that Yahoo! required. (20)

[ILLUSTRATION OMITTED]

In the area of relevance, Google currently handles more than 75 percent of all Web searches, (21) but that number may change when Yahoo! converts its search index from Google to Inktomi. Yahoo! is working to increase relevance of more commercial searches that are shopping and commerce related, indicating that it may seek a different–and possibly more commercial-market niche than Google. (22)

The third major player in search engine advertising is–or will be–Microsoft. Although it has, at this writing, only the search engine MSN.com, a modest traffic generator, and lacks ownership of monetization properties, this situation is expected to change dramatically in the coming months as Microsoft starts to compete directly with Google and Yahoo! Journalists and analysts covering the Microsoft move into search engine advertising generally agree that Microsoft will capitalize on its dominant position in operating systems and will embed searching into the operating system, quite possibly without the need for a browser. (23) During the past year, Microsoft has been building relationships with paid content providers such as the Gale Group (24) and LexisNexis (25) to build pay-per-view content download options into new versions of Microsoft Office. Designed to enable users to quickly identify and purchase paid content based on a contextual search of currently open applications, it is only reasonable to assume that this contextual search could easily extend to include a Microsoft-owned search property filled with paid listings.

Google’s Split Personality: Search Savior, Ad-monger

In making an ethical issue out of paid listings in search results, Google brilliantly established itself as a trusted search tool, the lone savior of Web searching. Google plays both relevance and monetization sides of Web search in an inspired way. It draws users to its search tool through finely tuned relevance and the promise of pure search results, yet it is one of the largest ad agencies on the Web. (26) In addition to its large partners (like AOL), Google serves up ads to thousands of smaller websites through its AdSense program, which compensates host sites when Google’s context-sensitive ads are clicked on.

The Future of Search

What would happen to searching if Google either were bought or went public, and when could that happen? In such a rapidly growing and changing search landscape, it is hard to predict. The topic has made headlines in the technology press for several months–which is itself often an indicator that a company and its investors are testing the waters to ensure that the timing is right for a public offering. There have also been rumors–largely unsubstantiated, it appears–of Microsoft being a possible suitor.

Companies go public for two reasons–to enable the principals to maximize their investment (essentially “cashing out” when the going is as good as they think it’s going to get) or as a means of raising capital to enable additional growth. There is no particular reason to think that Google is short of cash and with no burning capital investment requirement going unfunded. But with Microsoft clearly planning a major entry into the search marketplace, Google has good reason to be worried, particularly because Microsoft can use its dominant position in the operating system market to fuel its use of search within the operating system. Many of Google’s 1,000 or so employees will become instant multimillionaires, even billionaires, if Google launches an initial public offering and if that offering carries anticipated valuations. Moreover, the company has grown so large that the Securities and Exchange Commission may require Google to begin publishing its financial statements even if the company remains private. So why not capitalize on the loss of secrecy, cash out when the going is good, and raise billions of dollars in the process? (27)

Even if Google maintains its position as the only search engine without paid listings in search results, the search engine landscape will change dramatically in the coming months. As consolidation of Web search between three major players–Microsoft, Yahoo!, and Google–is completed, the next step will be competition for market dominance. The quest for users’ eyeballs will become ever greater as the quest for market share increases.

The Quest for Authentic Search Results

Serious searchers need to understand and accept that there is not now and will never be just one starter that will be suitable for finding everything anywhere. Popping a few keywords into Google is easy: research is hard. The increasing commercialization of search will require all serious searchers to have a “search toolbox”–a list of starter sites that they can return to when they don’t already know the best starting points for their information search.

There has been considerable interest among searchers in emerging schemes for improving relevance algorithms to enhance the user experience. (28) Unless relevance algorithms are radically altered and the pressure of commercial search somehow removed from the information-searching equation (which is hard to imagine), it is likely that serious searchers will be required to depend more on independently produced, quality-filtered link lists, catalogs, and directories of resources.

That is an enormous step away from our current dependence on commercial search tools and represents a sea change for most searchers. Methodically seeking quality information sources from excellent starter sites will be key in the future. Doing so will mitigate the ever-increasing effects of paid search, help searchers feel confident that they have fully explored the Web, and–perhaps most important–provide a clue as to when to stop searching and move on to other fee- and print-based information tools.

End Notes

(1) Every effort was made to ensure that information was accurate at the time of writing, but it is possible that some of the information in this article may have changed. The author wishes to thank Sharon Virtue of the optimization firm Virsha.com for her comments.

(2) Web Characterization Project, http://wcp.oclc.org. Data drawn from 2002 estimates.

(3) The search market represented $1.4 billion in 2002 and was projected to grow to $7.0 billion worldwide by 2007, according to Safa Rashtchy and Jason M. Avilio, Golden Search: Dynamics of the Online Search Market and Scope of Opportunity. US Bancorp Piper Jaffray, March 2003, p. 7.

(4) In 2002, 84 percent of all Americans expected to find information online related to news, health, business, and government services. In Counting on the Internet. Pew Internet and American Life Project, December 29, 2002. Available online at http://www.pewinternet.org/reports/pdfs/PIP_Expectations.pdf. Last viewed November 30, 2003.

(5) Danny Sullivan of Search Engine Watch estimates that eight major search services serve up over 625 million search requests per day. See his “Searches Per Day” at http://searchenginewatch.com/reports/article.php/2156461. Last viewed December 1, 2003.

(6) Robin Greenspan, “More money = more surfing.” October 14, 2003. http://www.internetnews.com/stats/article.php/3091091. Last viewed November 30, 2003.

(7) See, for example, “Advertising: Attitudes, Preferences and Engagements,” In E-Marketer.com, An Elephant in the Room: the Online At-Work Audience. February 2003. http://www.emarketer.com/products/report.php?atwork_feb03. Viewed December 1, 2003.

(8) “Web Searches: The Fix Is In.” Business Week Online, October 6, 2003. http://www.businessweek.com/magazine/content/03_40/b3852098_mz063.htm. Viewed November 30, 2003.

(9) See, for example, Google’s statements at http://www.google.com/technology/index.html, Alltheweb.com’s at http://www.alltheweb.com/help/faqs/web_search, and Altavista’s at http://www.altavista.com/about. Viewed December 1, 2003.

(10) According to analysts at US Bancorp Piper Jaffray, Google generated an estimated $294 million in 2002, Yahoo! $140 million, and MSN $138 million. Safa Rashtchy and Jason M. Avilio, Golden Search: Dynamics of the Online Search Market and the Scope of Opportunity, US Bancorp Piper Jaffray, March 2003, p. 6.

(11) “Our Search: Google Technology” at http://www.google.com/technology/index.html. Viewed December 1, 2003.

(12) For an overview of ranking mechanisms of the major search engines, see Greg Notess, “Search Engine Features Chart,” Search Engine Showdown, http://searchengineshowdown.com/features. Viewed December 1, 2003.

(13) https://www.google.com/adsense. Viewed November 26, 2003.

(14) http://www.content.overture.com/d/USm/ac/ba/cm.jhtml. Viewed November 26 2003.

(15) For a fuller discussion of the competition for links on the web, see D. M. Pennock, G. W. Flake, S. Lawrence, E. J. Glover, and C. L. Giles. “Winners don’t take all: Characterizing the competition for links on the web.” Proceedings of the National Academy of Sciences, 99(8): 5207-5211, April 2002. http://www.pnas.org/cgi/content/full/99/8/5207. Viewed December 1, 2003.

(16) Angela Gunn, “In Search of Disclosure: Meta-search Sites Compile Results But Don’t Always Tell You Who’s Paying for Placement.” April 17, 2003. http://www.consumerwebwatch.org/news/gunn/metasearch030416.htm. Viewed December 1, 2003.

(17) The text of the letter is available at http://www.ftc.gov/os/closings/staff/commercialalertat-tatch.htm.

(18) Leslie Marable, “False Oracles: Consumer Reaction to Learning the Truth About How Search Engines Work.” June 30, 2003. http://www.consumerwebwatch.org/news/searchengines/index.html. Viewed December 1, 2003.

(19) See, for example, how context-sensitive ads are served up by Google at sites as diverse as the almanac Infoplease (http://www.infoplease.com) and NationMaster (http://www.nationmaster.com/). Ad delivery will vary in an attempt to deliver ads contextually relevant to the search performed.

(20) An excellent online visualization chart of the major U.S. search property relationships is produced by search optimizer Bruce Clay and is available at http://www.bruceclay.com/searchenginerelationshipchart.htm. Viewed December 1. 2003.

(21) Danny Sullivan, “comScore Media Metrix Search Engine Ratings.” Search Engine Watch, October 28, 2003. http://www.searchenginewatch.com/reports/article.php/2156431. Viewed December 1, 2003.

(22) Stephanie Olsen, “Yahoo! takes a leaf out of Google’s book.” Silicon.com, November 4, 2003. http://www.silicon.com/networks/webwatch/0,39024667,39116742,00.htm. Viewed December 1, 2003.

(23) Bill Gates provided some clues to the idea of “seamless computing” during remarks made at the 2003 Comdex Conference in Las Vegas. A transcript is available at http:/www.microsoft.com/billgates/speeches/2003/11-16comdex2003.asp. See also Michael Kanellos, “Microsoft Aims for Search On Its Own Terms,” Cnet News, November 24, 2003. http://news.com.com/2100-1008_3-5110910.html?tag=nefd_lede. Viewed December 1, 2003.

(24) “Gale Group to offer online business information to Microsoft Office 2003 customers.” Press release, March 10, 2003. http://www.galegroup.com/servlet/PressArchiveDetailServlet?articleID=200303_office. Viewed December 1, 2003.

(25) “Microsoft Office System to Include LexisNexis Legal Research.” Press release, October 21, 2003. http://www.lexisnexis.com/about/releases/0626.asp. Viewed December 1, 2003.

(26) As a private company, Google’s exact revenues are not known, but could be as high as $1 billion, according to Eric J. Savitz, “Waiting for the Big One.” Barron’s Online, October 13, 2003. http://online.wsj.com/article_barrons_email/0,,SB106583046445910900-H9jeoNplaB2m5yvZX6HbKaEm4,00.html. Viewed October 13, 2003.

(27) Fred Vogelstein, “Can Google Grow Up?” Fortune148(12) (December 8, 2003) :102-113. http://www.fortune.com/fortune/technology/articles/0,15114,548765,00.html. Viewed December 1, 2003.

(28) Paula McKinnon, “It’s in the algorithms: a glimpse into the future of mapping the web.” Information Highways, 11(1) (November-December 2003): 12-14.

Rita Vine is a professional librarian and co-founder of Workingfoster.com (http://www.workingfaster.com), which helps professionals break through the clutter of the Internet and access information that matters. Her article “Real People Don’t Do Boolean: How to Teach End Users to Find High-Quality Information on the Internet” appeared in the March 2001 issue of Information Outlook. Rita teaches Web searching to clients across North America and serves on the selection team of the Search Portfolio (http://www.searchportfolio.com), an enterprise product of the 100 top starting points for searching the free Web. A “lite” version of the Search Portfolio, with links to about 10 percent of its resources, is available at http://www.searchportfolio.com/searchlite.html.

COPYRIGHT 2004 Special Libraries Association

COPYRIGHT 2004 Gale Group