Is Your Content Hidden in Search Results?

As web sites become increasingly dynamic in nature and as the content they publish becomes ever more dynamic in tandem, the risk that site owners are inadvertently hiding their content behind un-indexable site search tools increases. Whilst this might not sound like much of a problem, if you are trying to gain traction with search engines and rank in competitive markets, effectively blocking off a large portion of your content could seriously inhibit your efforts.

In the vast majority of cases the problem arises when well-meaning developers implement functionality on a site that should in theory assist users with their goals by making the process of searching or navigating a site more efficient, but in reality cause problems due to the way in which the tools are constructed and their acting as a block to search engines. If the information architecture of a site isn’t perfect or doesn’t provide a browse-able route to all of a site’s content, then content can be blocked or hidden away in the bowels of a site, never to be seen again.

An Example

Take, for example, the Association of British Insurers, the body charged with representing insurers in the UK. Given the high winds we’re currently experiencing in Scotland and the troublesome ash cloud drifting over from Iceland, they probably have their work cut out for them at the moment and are probably experiencing more than their usual levels of attention, so they make a good example for a brief case study.

Supposing we’re interested in looking for Scottish insurance companies, so we might type ‘Scottish Friendly’ into the search box. We would see this page here:

http://www.abi.org.uk/MemberSearchResults.aspx?searchQuery=SCOTTISH%20FRIENDLY

That’s a normal search results page and not one you’d usually expect to be published to the web, unless search engines have gone to the trouble of using the search tool on the site.

However, if we look a little more closely at the main members’ database, we see a few interesting ‘features’:

  • Filter results by: allows you to filter the results down by the first letter of the name of the company, but it does so in an entirely search engine-unfriendly manner (it uses a script) that means the content can’t be indexed.
  • Next >> allows you to move to the next page of search results but again the subsequent pages are accessed via a script that can’t be indexed.

Neither of these are a particular problem if we are looking solely at links off a rarely used search results page, but given the main members database relies on exactly the same filtering technology, in effect it means almost none of the content is visible to search engines.

This problem could be solved with some technical changes but as always, it’s better for the problem never to arise in the first place, and that comes from thorough SEO planning and strategy from the outset.