What the Moreover, Weblogs.com, Verisign deal means.

Posted by | October 18, 2005 | search engines | No Comments

This is my personal opinion and does not reflect any company policy.

Most web content is published and then indexed when a search engine finds it, taking up to 30 days. In the past submitting your site to a search engine was the done thing – now its coming back, only better.

Search engines have completely different indexes for news and weblog search, because the indexes need to be updated more quickly, to be able to do this they cannot search the entire web every few minutes but need to be alerted – or pinged. Currently, ‘pings’ to sites like weblogs.com or ping-o-matic or blo.gs say that SOMETHING has been updated on a weblog or news site. Specs such as RSSPing change this to a ping that says WHAT has been updated. If all pages being published on the web did this (and there is no technical reason why they couldn’t), then search engines would not need to crawl websites and search engines would be updated instantly.

Search engines are measured on how much, how relevant and how fresh. Pings are the answer to the fresh bit.

Mike Graves points out that Verisign plan to build value add services on top of pings, but acknowledges that pings themselves should be free:

“Ping services are not a profitable business, in and of themselves. Pings are free by tradition and by necessity. Attempts to introduce cost or latency into the ping layer would be self-defeating; the network simply routes around such problems. A free, open, scalable service fabric for pings is a powerful base for us to build value-added services on, however.”

This is good news because a single vendor ‘owning’ pings would really mess things up for publishers and Internet users in the long term.

In order to maintain innovation and development in weblog style publishing, RSS syndication and possibly even, in the long run, search, publishers such as SixApart should now bake the default ping to a (soon to be) non-profit service such as Ping-O-Matic (unless Feedmesh gets its act together) who would then pass it on to weblogs.com, blo.gs etc.

Alternatively, Verisign could keep Weblogs.com in a non-profit entity and develop premium services within Verisgn itself. This was pretty much how Dave Winer had things, separating church and state between his own publishing engine and Weblogs.com, so people trusted him to keep it neutral. The benefit with this option would be that there needs to be money from somewhere to make pings reliable and filter out the spam. The amount of money or infrastructure needed is not that great. I would argue that despamming, if it is by authentication, isn’t part of the value add but that custom subscriptions to alerts on topics are, but that’s debatable. In addition, despamming pings doesn’t need heavyweight authentication like certificates because the publisher to ping server ratio is not many to many. The problem with a Verisgn controlled root ping server (even non-profit) is that there are other large companies with ping server aspirations, such as Yahoo, who own blo.gs. There may need to be a truly neutral ping service for there to be a central one.

If this does not happen, ‘pinging’ will disappear as it either: Balkanizes, with companies who have both publishing and search products such as Google or Yahoo refusing to ping Verisign, or each other; stagnates with a single vendor having a lock on the whole thing, stopping competition and therefore, evolution.

The Internet works because nobody owns the roads. Keeping the infrastructure free and making money at the edges is what preserves the marketplace.