The Only World-Standard SEO Software

SEO and Internet Marketing Forum
Welcome, Guest ( Login | Register )

robot error Disallowed but its not
New Topic Add Reply
Page 1 of 1 [ 2 posts ]
Author Message


Posts: 3
Posted: Fri Jul 14, 2017 5:18 am

Hi i cant scan my site to see past home page
your program says a robot texts says disallow
I download the robot texts and it says allowed?

soooo where is the error please?

im on WP and it clearly says User-agent: *
where can this fandom live? or did i just spend a lot of money on a crap software? :shock:


User avatar
Site Admin
Posts: 5612
Posted: Fri Jul 14, 2017 2:05 pm

The thing is that using this type of command:

User-agent: *

you actually disallow your website from crawling if you do not specify the exact folder you wish to allow. In other words, you need either to set it as follows:

User-agent: *
Allow: /

Or remove this line completely because it makes no sense to use Allow command since you want to allow the whole website to be available for robots.

Display topics from previous:    Sort by    
New Topic Add Reply
Page 1 of 1 [ 2 posts ]
All times are UTC

Jump to: