There’s nothing really new here, but it’s good to get all this info into one place and to see it confirmed by Matt.

Topics covered:

• crawl budget/indexation cap – the use of Pagerank and host load to control crawl depth and frequency
• the effect of duplicate content on Pagerank
• session IDs and affiliate IDs in links/URLs
• faceted navigation – good to see Matt confirming that the advice I gave at SES London, and will be giving next week at SMX Munich, is all correct.
• Different ideas for use of the rel=canonical tag
• 301 redirects and how they differ from 302 redirects
• Google Webmaster Tools (WMT) ignore parameters
• Pagerank Sculpting and its effectiveness in the modern world
• Javascript, IFRAME and PDF handling
• Paid links and nofollow

Overall, the article strongly reinforces the fact that a successful site architecture is essential to SEO success.

John doesn’t mention that the same thinking applies even if you are targeting a single country. A ccTLD is the best way to indicate the location of your target market to search engines – and to that market itself, of course.

A URL gives you at least five places to target a country: domain (ccTLD), subdomain (de.domain.com), directory(www.domain.com/de/), path parameters (www.domain.com/;domain=de) and query parameters(www.domain.com/?domain=de). However, there are lots more axes for the content to be split along:

• Category – Web, Enterprise, Social, Real Time
• Context – Intranet, Library, Personal
• Topic – Health, Travel, Jobs, etc.
• Vertical – Finance, Education, Government, etc.
• Platform – Desktop, Mobile, Television, Kiosk
• Format – Text, Image, Audio, Video, Map

(Note: the above is slightly modified from a table provided by Search Patterns, an excellent read)

Given this number of ways of organising content, and the fact that the location and language of your target audience are major considerations (worthy of a major axis), in all but the most trivial cases a ccTLD is the obvious choice for geo-targeting. It’s good to see official written confirmation of this from Google.

My first impressions were that the ad is too negative. It doesn’t show what Bing can do for you. It’s at risk of associating Bing with information overload and distressed searchers. I’m also not convinced the phrase “decision engine” is a good one – too techie, too nebulous. Who’s making the decisions – me, or Bing?

Compare it with Google’s Superbowl ad:

This has its own potential problems – I’m not sure I would have been brave enough to use no voiceover whatsoever on a TV ad running in a £60,000 per second timeslot – but in general it’s a much more upbeat ad showing someone achieving something – lots of things – using Google Search.

In Microsoft’s position, I think I’d accept the fact that lots of people use Google and get good results lots of the time, and show that Bing is an alternative that often succeeds when Google fails. I’d challenge the notion that Google always delivers the right result, every time, and that if Google doesn’t deliver it it can’t be on the Web. I’d get people to try Bing – that’s all you can ask of the ad. An idea would be to use something based on the famous “Pepsi Challenge”, but bring it right up to date.

Having seen the interview with Ashley Highfield, I’m looking forward to more ads in the series. It would be great to see Bing achieve the double digit market share that he desires, but I think this was a bad start to the campaign.

Here at SilverDisc we’re often having to install and test new conversion tracking code for our PPC clients. Usually this involves searching for one of our client’s keywords on each search engine, clicking on it (thus incurring cost for the client) then going through the client’s site, making a test purchase and, later, checking that all the analytics has worked.

A cool feature that the search engines could add to improve efficiency would be a dummy campaign/ad-group/keyword that was automatically created by the engine itself within the PPC account specifically to test conversion tracking.

The keyword could be assigned by the engine itself, and could be very long, cryptic and unique to each client account, e.g. g54fr89fdcdjasdoe84.

• Searching for this keyword would always trigger the client’s ad
• Clicking this ad would not incur any real charges (although it may simulate a charge). Alternatively, a very low charge could be applied, e.g. £0.01.
• Conversion tracking could work much faster for this one keyword, e.g. near-real-time, to allow better, faster testing

This would save loads of time within agencies and mean that client accounts were up and running sooner, making more money for both clients and search engines.

I’m still very troubled by this paid links issue after all these years!

I agree it’s Google’s right to penalise or promote any page/site in its natural listings, which represent Google’s subjective opinion of relevancy.

However, the idea that all paid links are bad/”evil” is wrong in so many ways:

• Paid links pre-date Google.
• There is no machine-readable standard for labelling a paid link. I’ll repeat that – there is no machine-readable standard for labelling a paid link.
• Labelling paid links fails the “Does this makes sense in the absence of search engines?” ethical test. The answer may well be “Yes”. (Where the answer is “No”, I agree paid links are spam).
• Labelling paid links fails the “Would I do this if search engines did not exist?” test. In fact, you have to know that Google exists, and that they mind about paid links, in order to label those paid links in the non-standard way that Google asks you to label them. This is perhaps my biggest beef with Google’s approach to paid links – they actually violate one of Google’s published Webmaster principles.
• What does “paid” mean anyway? An actual exchange of cash? If you look at the top results for any hugely commercial field, say “car insurance”, it’s hard to believe that there is no commercial influence in the results! When all that a company does is commercial, then every link (positive or negative) to that company’s site is commercial in nature.

I understand that a market in paid links arose because of Google’s algorithm.

However, the irony is that in responding to that market by asking all publishers to label paid links in a non-standard way, Google violated its own principles. It started to ask publishers to adapt what they published to suit Google (because Google existed), and called them spammers if they didn’t. That’s the wrong way around. It’s the spammers that do stuff purely because Google exists!

If you haven’t a clue what I’m talking about, the Wayback Machine has the original Google Webmaster help on this topic, which I’ll paste here verbatim in order to make it searchable (Wayback Machine pages aren’t indexed by search engines):

Results Prefetching Questions

1. What is “results prefetching,” and how does it impact my site?

On some searches, Google uses a special <link> tag supported by Firefox and Mozilla to instruct the browser to download the top search result before the user clicks on the result. When the user clicks on the top result, the destination page will load faster than before. This tag is only inserted when it is likely that the user will click on the first link.

For example, when a Firefox user searches for [stanford], Google includes the following tag in the results HTML:

<link rel="prefetch" href="http://www.stanford.edu/">

The official Mozilla Link Prefetching FAQ describes the behavior of this tag in detail.

Prefetching may impact your site because the prefetch request will happen whether or not the user clicks on the result, so it may result in additional traffic to your web server. Google only inserts this tag when there is a high likelihood that the user will click on the top result, but clearly this heuristic is not right 100% of the time.

2. Can I distinguish prefetch requests from normal requests?

Yes, as described in the Mozilla Link Prefetching FAQ, prefetch requests include the additional HTTP header

X-moz: prefetch

3. I want to block/ignore prefetch requests. What should I do?

To block or ignore prefetch requests (from Google and other web sites), you should configure your web server to return a 404 HTTP response code for requests that contain the “X-moz: prefetch” header.

What else do you need to know about results prefetching?

If you run Google Analytics or another JavaScript-based analytics package, you won’t see these prefetched pages in your analytics. That’s because only the HTML is prefetched, not the images, JavaScript, etc. referenced by that HTML, which means that the Analytics JavaScript is never even fetched, let alone executed. You need to look at raw log files to see prefetched pages.

Google only issues the prefetch code when they are very confident that searchers will click on the #1 result (as in their example, a search for stanford). Most times, particularly for more “normal” sites (i.e. not Stanford), Google won’t issue the code. So you may never see this on your own site.

However, it’s worth being aware of this issue because if you do see a prefetch in your raw logs you’ll want to know why; and because, depending on how you calculate conversions, the fact that a page is prefetched but never viewed by a searcher may significantly affect your conversion tracking and monetisation on that page. I’m surprised that Google removed this info from their Webmaster help.

Ever the fast worker, Nicola gave birth yesterday to Caitlin Rose Richards at 10.29am, weighing 7lb 1oz.

Congratulations to Nicola and Sam from all at SilverDisc!

This builds on the recent series of posts on this topic:

• URL Canonicalisation and Normalisation
• rel=canonical tag
• a robots.txt equivalent to rel=canonical

The rel=canonical calculator will go on general release in the next couple of weeks, and we will be making some PHP code available to insert the rel=canonical tag on your own pages. That’s right – FREE CODE. Register using the instructions provided on the rel=canonical calculator page.

Many of the alternatives have existed for some time – the use of redirects and cookies, for example. However, the introduction of a rel=canonical tag was an opportunity for search engines to also introduce other, more efficient, standards. These are the alternatives I would like to consider – alternatives that don’t exist yet, which the search engines could have introduced this time around and may introduce in future.

I see the rel=canonical tag as analogous to the meta robots tag, and therefore suffering from many of the same limitations:

• The rel=canonical tag is located in a HTML file, and that HTML therefore needs to be fetched and parsed in order for the tag to be seen and acted upon. Therefore, the tag does not save bandwidth or CPU for the Web site or search engine.
• The rel=canonical tag is located in a HTML file and gives instructions about that file. Therefore, it cannot be used to solve canonical issues for non-HTML files such as images, PDF files or Flash movies.
• The rel=canonical tag acts at a micro-level rather than a macro-level. Therefore it is difficult to review that a site-wide policy has been correctly implemented using rel=canonical; Every possible file has to be inspected. Also, code changes have to be made in order to write the rel=canonical tag. This may slow its implementation.

Where the above issues apply to rel=canonical, and similar issues apply to the meta robots tag, it struck me that an opportunity has been missed to also solve canonical issues through the robots.txt file. Any fix applied through robots.txt would not suffer from the above problems.

Extensions to robots.txt could be made in a number of ways. For example, a mod_rewrite-type syntax could be introduced. However, I’m not sure anything so advanced is needed. Most canonical issues arise from three things:

1. the use of query parameters in dynamic URLs.
2. www versus non-www versions of a site (and other subdomains).
3. inconsistent use of default index page URLs.

Some simple robots.txt fields to control these issues would fix most problems without the pain and errors that a mod_rewrite implementation would create.

Query Parameters

Google Analytics and Yahoo Site Explorer are two examples of tools that allow simple manipulation of URL query parameters. Yahoo’s Dynamic URL Help lists some of the crawling, indexing and ranking benefits of this approach.

Yahoo Site Explorer allows you to remove a query parameter or set a query parameter to a default value within a URL. Using this, a URL such as

• http://www.example.com/page.php?refby=affiliate&sid=abc123

could be crawled and indexed as

• http://www.example.com/page.php?refby=yhoo_srch

The session id has been dropped and the referrer has been overwritten as yhoo_srch, meaning all traffic sent by Yahoo Search could be attributed to Yahoo Search rather than the affiliate. This functionality could be implemented in robots.txt using a new syntax something like the following:

User-Agent: Slurp
Disallow:
QueryParam: -sid
QueryParam: refby=yhoo_srch


meaning that the sid query parameter is to be dropped (as it is preceded by ‘-’) and the refby query parameter is to be overwritten with a default value (as a default value is provided). The same effect could be achieved with a single line:

User-Agent: Slurp
Disallow:
QueryParam: -sid, refby=yhoo_srch


One problem with both Google Analytics and Yahoo Site Explorer is that you must list the query parameters you wish to drop from URLs – not the ones you wish to keep. Because third parties can link to your site, you’re not in control of the links they create and the query parameters they use. Therefore, canonical issues can only truly be solved by specifying the query parameters you wish to keep, rather than those you wish to drop. To solve this, wildcards could specify the default action to be applied to all non-listed query parameters. Therefore I propose the following syntax:


QueryParam: retainParam[=defaultValue]
QueryParam: -dropParam
QueryParam: [-]*


where…

• retainParam[=value]: specfies a query parameter you definitely want to keep, and an optional default value you want it set to
• -dropParam: specifies a query parameter you definitely want to drop
• *: means keep all query parameters not specified (default)
• -*: means drop all query parameters not specified

Default domain and Index Pages

Two further, much simpler additions to robots.txt could clear up the majority of other canonical problems. These are Domain and IndexPage:


Domain: defaultDomain
IndexPage: defaultIndexPage


defaultDomain specfies the default domain for this robots.txt file. For example, if the search engine retrieves http://www.example.com/robots.txt and finds …


Domain: http://example.com/


…it would know to index all URLs under the non-www domain. This would allow multiple parked domains to share the same content and robots.txt file without needing redirects or causing canonical issues, which is currently a common problem.

The IndexPage field specifies a default index page for the domain, i.e. a page for which the following two URLs are considered equivalent:

http://www.example.com/path/

http://www.example.com/path/defaultIndexPage

Conclusion

In this post I’ve proposed three new fields to add to robots.txt to provide an alternative to the rel=canonical tag, just as the current robots.txt fields are themselves alternatives to the meta robots tag, with their own advantages and disadvantages. The chief advantages I see of canonicalising through robots.txt are:

• Acting through robots.txt means that a resource does not have to be fetched and parsed in order for the canonicalisation instructions to be followed. Therefore, bandwidth and CPU is saved for both the Web site and search engine.
• Acting through robots.txt means that canonical issues can be solved for non-HTML files such as images, PDF files or Flash movies.
• Acting through robots.txt means large scale changes can be made very quickly and easily without the need for any code changes. It’s also much easier to review the changes that have been made.

The Domain, IndexPage and QueryParam fields would all be optional and independent of each other. It would be great if the search engines could introduce some or all of these ideas into robots.txt.

Much has already been written about this tag and its purpose: to help prevent duplicate content issues. Probably the best summary is this Matt Cutts video:

This tag is a welcome addition to the armoury in the fight against duplicate content issues. In addition to Matt’s comments, I would make the following points:

Copyright Protection

Scrapers are forever copying content and publishing it on their own sites/splogs. Sometimes they are exceptionally lazy or stupid, even to the extent that they copy Adsense code onto their own sites. If they copy your rel=canonical tag onto their site, that would give a strong “hint” to the search engine that you were the original owner of the content:

<link rel="canonical" href="href="http://www.mysite.com/my/content/" />

Microsoft Platforms

Matt made reference to the Microsoft platform in his video, but I would emphasise the point. Microsoft’s implementation of RFC 2396 is flawed. The path component of a URL is supposed to be case sensitive, but Microsoft makes it case insensitive. If there are n alphabetic characters in the path, then a Microsoft implementation gives 2ⁿ possible variations of that path, where there should be only one. For example, if n=1 and the path is “/a/”. Microsoft would allow “/a/” and “/A/”; if n=2 and the path is “/ab/”. Microsoft would allow “/ab/”, “/aB”, “/Ab” and “/AB/”; and so on. 2ⁿ variations gives vast potential for duplicate content and it is a big issue with sites built on the Microsoft platform. The rel=canonical tag makes it very easy to specify the correct, case-sensitive path on a Microsoft platform:

<link rel="canonical" href="http://www.mysite.com/my/case/sensitive/path/" />

Static Web Content

Static web content is content that is stored in the format in which it is delivered. Typically, static content is served under a static URL (a URL that does not contain a question mark). However, it is possible to link to static content and append query parameters, even though these query parameters will have no impact on the content that is served. One example of when this might happen is when a referrer parameter is passed to a JavaScript function within the static content:

<a href="http://www.mysite.com/?referrer=myAffiliate0001">Affiliate Link</a>

Thousands of links can be created to a single, static URL, each with a different referrer query parameter attached. For sites built on static content, trying to manage such links has been difficult in the past. Now, it’s relatively easy. Each page of static content simply needs to contain a rel=canonical tag:

<link rel="canonical" href="http://www.mysite.com/my/static/url.html/" />

Conclusions: rel=canonical

For the reasons stated above, I would recommend the use of a rel=canonical tag in all static content. In fact, I would recommend its use in all content, static or dynamic – with appropriate care of course. It’s a powerful tag and using it wrongly could have dire consequences.

In the next post I’ll look at some of the limitations of the rel=canonical tag and consider some alternatives.