The Only World-Standard SEO Software

SEO and Internet Marketing Forum
Welcome, Guest ( Login | Register )

Filter Problem - Website Auditor
New Topic Add Reply
Page 1 of 1 [ 3 posts ]
Author Message


Posts: 2
Posted: Wed Jun 27, 2018 11:28 am

Hey there,

We have been using Website Auditor for quite a while now and we are quite happy with the way it works.
Since a couple of months we have trouble refreshing and rebuilding the website with Website Auditor.
We use filters on some of the product pages. These filters will add the words ''shopby'' in the url.
This ''shopby'' tag is basically on every webpage.

If I select a brand in the filter then it would change the url into this:


We want these pages to be indexed but there is no need to optimize these filter options. Therefore, we don't want them in Website Auditor.
I've added the words ''shopby'' in the excluded section, yet, Website Auditor takes the ''shopby'' into account when it is scanning the webshop. This takes over 10 hours and it crashes after 4-5 hours. This also makes our webshop very slow and sometimes inaccessible.
Is there any way to not take the shopby into account when rebuilding the website. We just want the individual products and other pages, not the filters.

We tried to filter them out.
Since I'm not allowed to post links imgur[DOT]com/a/772u58P


User avatar
Site Admin
Posts: 5620
Posted: Thu Jun 28, 2018 11:22 am

It is true that when you apply a filter in Website Auditor the software will still scan the pages, but it will not them to the project. This has been done to let the software find all pages of your website, since the directories that you may want to filter out may still contain links to other pages of your website. So the software first scans the whole website and only then adds the pages that match the filter criteria.

However, if you want to avoid scanning some directories, you may close them in robots.txt file for SEOpowersuite-bot and the in the Website Auditor settings you specify to follow robots.txt rules for seopowersuite bot. In this case the software will not visit those pages.



Posts: 2
Posted: Fri Aug 17, 2018 9:48 am

Hey there,

We've tried both of these options but neither of them have worked.
Our webdeveloper added the Robots tag but has removed it already since it didn't work out.
Any other options to exclude the /shopby/ from the crawler or reduce the audit time?

Thanks in advance!

Display topics from previous:    Sort by    
New Topic Add Reply
Page 1 of 1 [ 3 posts ]
All times are UTC

Jump to: