The Only World-Standard SEO Software

SEO and Internet Marketing Forum
Welcome, Guest ( Login | Register )

partial crawl?
New Topic Add Reply
Page 1 of 1 [ 3 posts ]
Author Message


Posts: 1
Posted: Thu Jun 14, 2018 12:09 am

Is there any way to crawl just a portion of your site?

I'm more familiar with Screaming Frog where I enter a URL, and it starts crawling from that URL, visiting all the links it finds and analyzing them, until it runs out of URLs, or I stop it.

When I try this with Website Auditor, and just enter a single URL, it only crawls that single page.

Do I need to just generate lists of a few thousand URLs at a time, and add them to the list of pages manually, and "Update pages", or is there a better way?

Alternatively, if there's a way to stop the crawl after a certain time and have it continue the rest of the analysis on the pages it has crawled so far, that would be another possible workaround.



User avatar
Site Admin
Posts: 5612
Posted: Tue Jun 26, 2018 8:43 am

WebSite Auditor works practically the same way: you can specify the exact page URL and the software will start collecting URLs from that page and will be adding them to the project, but only those URLs that fit into the scope of your project URL. For example, if you specify this page:

then the software will be adding to the project only those pages that fit into this scope, such as:



The URL, that are found during the scan and that do not fit into the specified scope are added to the Resources section in the project.

So, if you want to scan only particular directory, you can simply specify that directory URL as your project URL and WebSite Auditor will pick up only those paages that fit into that directory.



Posts: 1
Posted: Sat Sep 29, 2018 7:43 am

The Fetch as Google feature in the Google Search Console rocks, it shows you how GoogleBot sees your web site or app. It can visually give you a clue into crawling issues. But often webmasters get a notice that GoogleBot was only able to do a partial crawl and that can be scary.

Display topics from previous:    Sort by    
New Topic Add Reply
Page 1 of 1 [ 3 posts ]
All times are UTC

Jump to: