Pages: [1]
Author Topic: Site indexing part 2: Meta tags.  (Read 1257 times)
« on: September 28, 2010, 05:15:00 pm »

'In part 1 of this series about how you can control site indexing I've spoken about the robots.txt. This article will deal with using metatags which allow a little bit more control about how your website is being indexed.

Unlike the robots.txt which is a single file meta tags are part of the page and provide extra information about it. The robots meta tag tells the web spider two things: whether it should index the page and whether it should follow the links on the page.

When to use meta tags
You can use the robot meta tag in every page you want. Because the page has to be loaded before the meta tag is read it's better to use a robots.txt if you want to save traffic or to prevent large portions of the your site from being indexed.

The basic syntax is fairly simple again. The meta tag should be placed in the head of the html page. The tag is opened with <META. It has the attribute NAME which should be set to "robots" and the attribute CONTENT which takes the argument follow, nofollow, index, noindex or a combination.

Follow/Nofollow tells the web spider whether it should visit the links that are provided and index/noindex tells the web spider whether it should index the page. For example <META NAME = "robots" CONTENT = "follow,noindex"> tells the web spider to visit the links on the page but it shouldn't store the page itself in the database.

Just like the robots.txt some web spiders support an additonal set of meta tags giving you more control over how they index your site.

Using meta tags in dynamic pages
In CMS's which use one file to publish all the pages it would be required to add metatags specific for each type of page. To do this you'll have to capture the type of pages the same way the cms decides which pagetype to load and with this information decide which metatag should be printed. In an ideal cms this would've been done allready and you're able to configure this yourself. I wouldn't been surprised however if this functionality is lacking in most CMS's around.'
Pages: [1]
Jump to: