mediajunk

Search engine watchers aren’t just trainspotter types (honest!). Search has become the single hottest sector on the internet, with the advent of search-based marketing and advertising, and the global appeal of search brands. Google, more than any other engine, has made search sexy. The search sector has seen rapid conglomeration during the last year, as competition intensifies. Microsoft is threatening to enter the fray and, as anticipated, Yahoo finally dumped Google as its search technology provider this week. No longer will a search on Yahoo produced the same results as a search on Google. What few expected, was that Yahoo would build an entirely new search algorithm. After all, Yahoo owns Inktomi, a company that provides search technology for MSN, AltaVista and others. It is too early to say whether Yahoo’s new search results are as high-quality – that is, as relevant to search queries – as Google’s. Relevancy is not the only factor that will influence the battle of these two search giants. Another is the non-trivial matter of the pay-for-inclusion (PFI) service that Yahoo will soon offer. PFI is not the same as paying for a Yahoo sponsored listing; nor is it the same as paying for a listing in the Yahoo directory. Rather, PFI allows a customer to pay for a site to be quickly submitted to by crawled by the Yahoo spider. PFI does not guarantee that the site will appear highly – or even at all – for particular searches in Yahoo’s results. For example, let’s say I noticed that Mediajunk.com was not appearing in Yahoo’s results and, by looking through my site visitor statistics, I confirmed that Yahoo’s spider (which, incidentally, was recently renamed “Slurp”) was not crawling my site. I may then decide to pay for inclusion. This would guarantee that Slurp would visit my site, and return regularly to check for updates. If I chose not to pay for inclusion in Yahoo, on the other hand, I could choose simply to submit Mediajunk.com to Yahoo for free. My site may still appear top of the regular (non-sponsored) listings in Yahoo – but not before Slurp has crawled it. If I were to update Mediajunk (as I do regularly), I may have to wait – who knows how long? – for Slurp to return, so that my updates are reflected in Yahoo’s listings. By contrast, Google crawls as many URLs at it can on a regular, almost daily basis – and for free. Google does not offer a PFI service. Why do I think that resisting PFI will ultimately work out to Google’s advantage? Because Google’s search results will be fresher than Yahoo’s. After all, the web now contains several billion pages. Only a tiny fraction of those (let’s say 0.05%) will be submitted to Yahoo via its PFI service. Thus, only 0.05% of web pages will be regularly crawled by Yahoo; the remainder will be crawled less frequently. The result: search results that look out-of-date. After all, if the majority of results aren’t stale, what advantage does PFI give? It will certainly be interesting to see how Yahoo’s results look several months from now.