November 17, 2007
Bump! Bang! Does Your Site Block Spiders?
Search engine spiders crawl pages following hyperlinks on other pages. But certain things may serve as hindrance to crawling of search engine spiders.It is important to know and understand them so that one can modify he website accordingly for giving better access to spiders.
A link that is too complex to follow or is situated in quite depth from the hyperlink is called a bump.
This serves as speed breaker for the spider in a similar fashion to traffic on the road.
Website may contain sections of data that cannot be accessed at all by the spiders is called wall.
Both of these would block indexing of certain sections of your website.
There may be some parts of your site that you do not want the search engines to know.
But if you are in business, you would want to expose maximum of your content and indexed.
These are few bumps and walls that you must avoid to do that.
Bumps
- Dynamic URLs with >2 dynamic parameters would be a bump. These kind of URLs are reluctantly crawled as complex URLs often result in errors. An exmple of complex URL would be http://www.yoursite.com/page.php?id=3&PF=567&user=premium&name=%Arun%
- Hundreds of outgoing links from a page. In such cases spiders may not follow each link.
- Pages buried in depth of linking structure. Generally, the spiders will not visit a website link that requires more than 3 clicks from home page untill that page has high number of incoming links.
- Frames are a bump to crwaling.
Walls
- Pages requiring a login. This is generally case untill you put special request for robots to visit.
- Documents blocked by robot.txt file
- Links having nofollow tag.
- Pages that require drop down menu to access them.
- Documents that can only be accessed by a search box.
- Pages that can be accessed only after filling and submiting the form. For example a sign up form.
Fewer bumps and walls your site has, smoother would be its indexing.
A rule of thumb to avoid unwanted bumps and walls is to ensure that the content of your site is easily accessible. Most of the spiders would start from your home page. To get indexed, the page must be accessible from home page.
Popularity: 7% [?]
Got A Tip, Idea or Suggestion. Share It Here