Best practice with the TextUnited Website Crawler

In this article we are going to outline the best practice and limitations of the TextUnited Crawler


The size of your website matters...

Our crawler is meant specifically for small to medium websites, without 100s of subpages. Although our crawler is powerful, scanning hundreds of pages takes time and due to the complexity and security mechanism put in place by certain websites, may come up with an error. 

Therefore, its best to only user the crawler for small and medium websites. Contact us if you need us to run a specific test on your website or websites. See that chat icon in the bottom right of your screen? Click there to get in touch with us quickly!


TLS and encryptions...

If your website is using TLS 1.3 in the HTTPS encryption, the scanner will most likely run into an error. We are working on a way to make this work. 

If you have used the service "Let's Encrypt", there is a chance our crawler will also run into errors here.