Expanded robots.txt monitoring with more granular change tracking

ContentKing has always kept an eye on your website’s most important file: robots.txt.

Now we’re taking it a step further with our expanded robots.txt monitoring! Our expanded robots.txt monitoring makes it easier to navigate, view, and analyze your robots.txt files with:

• Extensive change tracking of robots.txt
• HTTP headers for any robots.txt versions
• Ability to download full response
• Log file analysis for robots.txt

And not to mention, we made it look pretty good too! Our new redesign makes comparing different robots.txt versions a breeze.

Travel back in time and see all versions of robots.txt

Quickly and easily see all changes that have happened in your robots.txt in one place.

See HTTP headers for any robots.txt version

Ever wondered what the exact response was for your robots.txt? Or are you seeing two alternating versions of your robots.txt and need to investigate?

Check the HTTP headers for any robots.txt version to see:

• the headers of the HTTP request that we sent
• the headers of the HTTP response that we received

Download your robots.txt data with one click

Need to show the response and HTTP headers to your colleagues or the technical team? Easily download your robots.txt data and forward it directly to them.

See when your robots.txt is visited by search engines in real time

With our new Log File Analysis feature, you can know whether or not Google and Bing have seen each version directly in the robots.txt viewer.

Log File Analysis is available on our Pro and Enterprise plan. Haven’t checked out our new Log File Analysis feature yet? Learn more here.

Try it out now

Don’t wait any longer! Check out these new improvements to robots.txt monitoring today.

Check your robots.txt now (opens in a new tab)

Want to learn more? Reach out to us if you have any questions.

Not using ContentKing yet? Do your website a favor and start your free trial right now!