Robots.txt checker

Check if a website has a robots.txt file and review its sitemap and disallow rules.

Similar tools

Check the status code returned by a website for a generated missing page.

Checks if a webpage is set to noindex, preventing it from appearing in search engines.

Get all the HTTP headers that an URL returns for a typical GET request.

609

Website status checker.

7,597,369

Check for 301 & 302 redirects of a specific URL. It will check for up to 10 redirects.

1,306

Get the web-host of a given website.

1,258

Fetch and check every meta tag on any web page.

1,167

Check whether a website is using the new HTTP/2 protocol or not.

1,141

Check whether a website is using the Brotli Compression algorithm or not.

1,088