The Only World-Standard SEO Software

SEO and Internet Marketing Forum
Welcome, Guest ( Login | Register )

All times are UTC
need some information about Spider.txt
New Topic Add Reply
Page 1 of 1 [ 3 posts ]
Author Message
ali raza


Posts: 6
Posted: Wed Jun 09, 2010 4:52 am

Hi ,
Everyone ,i am confused how to write Spider.txt file ,
i have a quite big website,a online store website,consists of lot of products,
and i include shopping cart pages ,some flash ,
How to write spider.txt file for my website
can anybody help me
i will be thankful


User avatar
Site Admin
Posts: 5621
Posted: Fri Aug 13, 2010 1:58 pm

Hi ,
Everyone ,i am confused how to write Spider.txt file
Follow the instructions, please:

1. name it robots.txt *not* spider.txt

2. make a list of everything on your site you DON'T want robots/spiders to visit, and list in in robots.txt like so:


User-agent: *
Disallow: /directory1
Disallow: /directory2/file1.htm


The * after disallow means NO spider is supposed to visit the files & directories in this section.

If you only want to ban specific robots from certain files, add a second section like so (replace the * with the user-agent of the spider you want to ban):


User-agent: Googlebot/2.1
Disallow: /don't_want_google_here

User-agent: FAST-WebCrawler/2.2
Disallow: /don't_want_FAST_here




Posts: 2
Posted: Tue Aug 17, 2010 3:14 pm

You can sign up to Google webmaster tools and create your robots text file there. Download it from the webmaster tools site and then upload to your own site. This is handy if you use DNN for example where page names can be either human friendly or gibberish or somewhere in between so you can have 3 or more versions of the same page name :)
If you leave the site like this you can be penalised for duplicate content.

Display topics from previous:    Sort by    
New Topic Add Reply
Page 1 of 1 [ 3 posts ]
All times are UTC

Jump to: