Paid Inclusion Going the Way of the Dodo

Late last week, Ask Jeeves announced it was going to cease its paid submission program and allow webmasters to submit sites to its database for free. The announcement was quickly followed by the release of MSN’s new search tool that also does not charge fees for inclusion. Of the four major search engines, only Yahoo continues to charge webmasters fees to submit their sites, however even Yahoo is reconsidering its paid-inclusion plans. (Google has never charged for inclusion in its database).

At first, submitting to search engines was absolutely free of cost for webmasters. Search Engines such as AltaVista, Infoseek and Lycos assumed that large-scale advertisers would cover costs by purchasing banner advertising. As a revenue model for search engines, banner advertising simply did not raise enough money to cover the costs of running a large IT operation. During this period of the mid to late 90’s, the Internet economy was booming but heading for the bust which hit in April 2000. Money was literally flying from the pockets of venture capital firms eager to invest early in emerging technologies. Revenue generation was important but took second stage as the proponents of the new-economy believed that the seemingly exponential growth of the IT sector had magically reversed the most ancient law of business, make money or die. This was the time when small IT firms were suddenly valued as highly or higher than solid old-economy businesses such as TimeWarner or Proctor and Gamble. “Be Big or Be Bust” was a common quote, one the various search engines took to heart. This was the time of convergence, portals and rapid, unhindered growth. This was the time when AOL was able to purchase TimeWarner for what are now nearly worthless stocks and options. Suffice it to say that anyone who did not live through that era would never believe it actually happened. It did happen, and like every bubble before it, the Dot-Com bubble burst, removing the seemingly unlimited flow of cash from the sector. Along with the burst came the sudden realization that running a search engine is much like running any other business with one notable exception, most businesses start small. Search engines have always been among the most important tools on the Internet and by the time the tech-bubble burst, consumers were already relying on their use to find products and information. Clearly, a revenue generation model that actually boosted the bottom line was essential to remaining in business. Hence the concept of having advertisers pay to be included in the databases of the major search engines.

Paid Inclusion was at one time the most popular revenue model for search engines, emerging shortly after the dot-com crash of 2000. Under paid inclusion programs, webmasters were forced to pay a nominal fee to have their websites included in the databases of search engines such as Alta Vista, Lycos, All the Web, Inktomi, Yahoo and LookSmart. Payment for inclusion did not guarantee Top10 listings however it did guarantee the site would be found by search engine spiders and would receive more frequent spider visits. As a revenue model, paid-inclusion was a double-edged sword. The search engines were suddenly able to charge for listings that had previously been offered for free, much like the local phone company suddenly charging you to list your home number in the white pages. At this point, it is important to remember that there were already hundreds of millions of websites in most search databases, the vast majority of which had never paid a cent for inclusion. By charging new listings for a service that hundreds of millions were already receiving for free, the paid inclusion model set up a potentially contradictory position in which a paid listing and a free listing would compete for Top placements.

It was naturally assumed that paid listings would receive a rankings boost from the search engines however it was also assumed that search engine users would not necessarily know which ads had been paid for and which were organic. The search engines themselves did not want to point out which were paid as search engine users tended to favour sites they figured had gotten to the top by merit as opposed to money. The situation became so confusing that Ralph Nader’s Public Interest Resource Group (PIRG) petitioned the Federal Communications Commission (FCC) to investigate and potentially regulate paid inclusion programs. The threat of federal regulation, combined with users’ perceived distaste for paid-advertising led the search engines to universally mix sites that had paid for inclusion with the standard non-paid sites. The only real benefit to paying was the increased frequency of spidering enjoyed by paid-inclusion sites. As it turned out, many SEO firms including StepForth realized that our clients were getting into the search engines without paying for inclusion in their databases and were continuing to get the Top10 placements. Needless to say, most players in the SEO industry simply stopped charging their clients the extra fees for inclusion. Paid-inclusion turned out to be a rather clunky revenue model, especially when compared to the far more lucrative contextual advertising model found in Pay-Per-Click tools such as Overture and Google AdWords.

Yahoo is the only search tool that continues to ask webmasters to pay for their sites to be included in the Yahoo search database however the vast majority of our clients is getting Top10 placements at Yahoo without paying. This happens because, like all algorithmic search tools, Yahoo uses a spider to find websites by following links from other sites. Like with Google, if you have a link on your site, chances are Yahoo will find and list it. The same can be said for Ask Jeeves, Teoma and MSN. With the costs of keeping a billing department running for a paid-inclusion program most have learned they can bypass with very little effort was likely too much for search engines to support. Today, the dominant revenue model is Pay Per Click as offered by Google and Overture.