The Only World-Standard SEO Software

SEO and Internet Marketing Forum
Welcome, Guest ( Login | Register )

Website Auditor: do not scan pages with the following words
New Topic Add Reply
Page 1 of 1 [ 1 post ]
Author Message


Posts: 1
Posted: Thu Aug 31, 2017 11:23 pm


I recently started using GTranslate to perform language translation on our site. As part of it's process, I set it up to use subdirectories.
Examples: (Spanish) (Swedish)

I have support for about 15 languages.

I've added the following to my "do not scan pages with the following words" setting:
/es/ /ar/ /ch-ZN/ /nl/ /fi/ /fr/ /de/ /iw/ /is/ /ja/ /ko/ /ms/ /no/ /ps/ /pt/ /ru/ /sv/

The subpages are still being scanned, which makes the crawl for the site take forever. Before we installed the langage translator, the crawl took a long time already (we have about 1300 pages), now, with the langauge translator installed, it takes many hours to finish a crawl 1300x15 = 19500 pages).

Is it possible to make it so that the crawler doesn't follow each of these links, and then exclude them, but instead excludes them from being scanned at all?

Thank You in advance!

Display topics from previous:    Sort by    
New Topic Add Reply
Page 1 of 1 [ 1 post ]
All times are UTC

Jump to: