Hello, we noticed while running our SEO audit a lot of 403 errors from this page
https://embed.tawk.to/69ef498bd0fd481c759f8f34/1jn7beia0
While the same returns a 200 from regular browser.
Question do you block traffic for this types of crawlers? Per my investigation that seems to be the case.
I think this is their list of IPS
Hi Felipe,
Thanks for reaching out.
You are correct. The embed.tawk.to endpoint that serves the chat widget script blocks bots and crawlers on purpose, including the Ahrefs Site Audit bot. This keeps automated traffic out of your visitor monitoring and reports, so the visitors you see in your dashboard are real people. Regular visitors are not affected, which is why the same URL returns a 200 in your browser.
The widget script only loads the chat widget. It does not add indexable content to your pages. So the 403 in the audit should not affect your rankings. In Ahrefs you can mute this issue in Site Audit issue settings so it doesn’t count against your health score.
Let us know if you have any questions.
Hi thanks for replying.
By any chance can you allow those IPS on your side to prevent this noise during the audits?
That would be very nice if possible
Hi Felipe,
Thanks for your reply.
No, we can’t allow those IPs because the embed.tawk.to endpoint is shared by every website using tawk.to. Allowing the Ahrefs crawler would mean that this bot traffic gets tracked like real visitors by every single property.
Instead, I suggest you add a rule to remove it from your reports. In Ahrefs, go to Project Settings > Site Audit > Crawl Settings and add embed.tawk.to under “Don’t crawl URLs matching the pattern”. This should exclude the widget script from the reports when crawling your website.