Comments on: QDF - WTF…? http://www.silverspike.co.uk/2007/06/15/qdf-wtf/ The Official SilverDisc Blog Fri, 21 Nov 2008 03:08:25 +0000 http://wordpress.org/?v=2.6.1 By: alan http://www.silverspike.co.uk/2007/06/15/qdf-wtf/#comment-4 alan Fri, 13 Jul 2007 11:38:47 +0000 http://www.silverspike.co.uk/2007/06/15/qdf-wtf/#comment-4 > Given that some content doesn’t change, I would argue that it is prudent to not bother downloading such pages very often True, but that's why the "If-Modified-Since" HTTP request header exists. The SERPs for any query consist of fresh results, i.e. results that are likely to still exist and be what the robot saw. That's not to say that the results need to be recently created - just recently verified. > Maybe, rather than freshness, it is better thought of as ... “document accuracy”, i.e. the likelihood that the data stored for a document is accurate, That's exactly what's at issue. > Given that some content doesn’t change, I would argue that it is prudent to not bother downloading such pages very often

True, but that’s why the “If-Modified-Since” HTTP request header exists. The SERPs for any query consist of fresh results, i.e. results that are likely to still exist and be what the robot saw. That’s not to say that the results need to be recently created - just recently verified.

> Maybe, rather than freshness, it is better thought of as … “document accuracy”, i.e. the likelihood that the data stored for a document is accurate,

That’s exactly what’s at issue.

]]> By: mike http://www.silverspike.co.uk/2007/06/15/qdf-wtf/#comment-3 mike Sat, 16 Jun 2007 03:33:42 +0000 http://www.silverspike.co.uk/2007/06/15/qdf-wtf/#comment-3 “Query deserves freshness”? I thought that all queries deserved freshness! A stale index results in poor quality SERPs that could lead to lots of pages that no longer exist or have been changed since they were indexed. A fresh index solves this problem to a great extent. I disagree that all queries deserve the "Last crawled date" freshness, because it pits two goals (fresh index and effective utilisation of crawl resources) against each other. Some content will not to change much. Take the W3C HTML stanadards: http://www.w3.org/TR/html401/ Given that some content doesn't change, I would argue that it is prudent to not bother downloading such pages very often. In terms of bandwidth and useful utilisation of a parser (which, with ever increasing elements to uncover, is probably taking more CPU cycles than ever), the resources spent grabbing an unchanging page are better spent grabbing either new content for the index, or refreshing content that changes regularly, like the front page of Digg or news sites. Reconciling an algorithm that optimises the utilisation of crawl resources, that likely discriminates against primary sources that never change, with a usage of "last crawled date" in rankings would be problematic, especially when the search clearly wants the primary source. Sure, you could make the last crawl date for such pages constantly NOW, but that makes the ability to optimise crawling schedules (which would be as close to 1 second after publishing as possible :)) potentially harmful to results. Optimising in two direction (crawl effieciency and relevance) is terribly difficult. Maybe, rather than freshness, it is better thought of as "staleness" or "document accuracy", i.e. the likelihood that the data stored for a document is accurate, with staleness the moment at which trust in the accuracy of the indexed copy moves from accurate reflection to not an accurate reflection. Freshness, per se, doesn't really matter, but accuracy does. Such a measure would aslo alow different documents to loss the trust in teh acuracy of the indexed copy at different rates (a news site's home page, for example, would likely be innaccurate after 10-15 minutes; a side with every legal judgement likely has a lot longer decay cycle). “Query deserves freshness”? I thought that all queries deserved freshness! A stale index results in poor quality SERPs that could lead to lots of pages that no longer exist or have been changed since they were indexed. A fresh index solves this problem to a great extent.
I disagree that all queries deserve the “Last crawled date” freshness, because it pits two goals (fresh index and effective utilisation of crawl resources) against each other.

Some content will not to change much. Take the W3C HTML stanadards:
http://www.w3.org/TR/html401/

Given that some content doesn’t change, I would argue that it is prudent to not bother downloading such pages very often. In terms of bandwidth and useful utilisation of a parser (which, with ever increasing elements to uncover, is probably taking more CPU cycles than ever), the resources spent grabbing an unchanging page are better spent grabbing either new content for the index, or refreshing content that changes regularly, like the front page of Digg or news sites.

Reconciling an algorithm that optimises the utilisation of crawl resources, that likely discriminates against primary sources that never change, with a usage of “last crawled date” in rankings would be problematic, especially when the search clearly wants the primary source.

Sure, you could make the last crawl date for such pages constantly NOW, but that makes the ability to optimise crawling schedules (which would be as close to 1 second after publishing as possible :)) potentially harmful to results. Optimising in two direction (crawl effieciency and relevance) is terribly difficult.

Maybe, rather than freshness, it is better thought of as “staleness” or “document accuracy”, i.e. the likelihood that the data stored for a document is accurate, with staleness the moment at which trust in the accuracy of the indexed copy moves from accurate reflection to not an accurate reflection.

Freshness, per se, doesn’t really matter, but accuracy does. Such a measure would aslo alow different documents to loss the trust in teh acuracy of the indexed copy at different rates (a news site’s home page, for example, would likely be innaccurate after 10-15 minutes; a side with every legal judgement likely has a lot longer decay cycle).

]]>