SECURITY
Crawler and bot traffic causing Magento performance incidents
Aggressive crawlers hitting uncached filter and search URLs can overload Magento. Combine Fastly/WAF rules, caching and app fixes to stop it.
By VISIBI Security team·Reviewed by Saeed Ak, Co-founder & CTO·Updated 29 September 2026·7 min read
QUICK ANSWER
Aggressive crawlers hitting uncached filter and search URLs can overload Magento. Combine Fastly/WAF rules, caching and app fixes to stop it.
FROM OUR ENGINEERING WORKREAL CASE
Automated crawler traffic on expensive, uncached URLs created substantial application and database load until edge rules and caching were tightened.
KEY TAKEAWAYS
✓Bots often hit expensive, uncached URLs like layered navigation and search.
✓Rate limiting and bot rules at the edge protect origin capacity.
✓Don’t block good bots like Googlebot.
Why bots hurt
A single aggressive crawler requesting thousands of filter combinations can generate more database load than real shoppers.
RELATED GUIDEHow to stop spam and bot attacks on Adobe Commerce Cloud → Stop bots and card testingEdge and platform protection without annoying customers.Get a free bot audit →Defences
- Rate-limit expensive URL patterns at Fastly or the WAF
- Verify good bots, challenge unknown ones
- Cache or noindex filter combinations
- Block known bad ASNs
- Monitor top user agents and IPs
Common mistakes to avoid
✕Waiting until after a breach to act
✕Restoring a backup without fixing the entry point
✕Leaving old plugins, extensions and admin users in place
✕Relying on a single tool instead of layered defences
HOW VISIBI CAN HELP
How we help with security
01ScanFree scan for malware, vulnerabilities and blocklisting.
02Clean & patchEngineers remove threats and close the entry point.
03HardenWAF, 2FA, least privilege and platform hardening.
04MonitorContinuous scanning with unlimited cleanups on our plans.
Get a free bot audit →Free · No obligation · Reply within 24 hours
Frequently asked questions
Will this hurt SEO?
Not if you verify Googlebot and Bingbot and only limit unknown bots.
What about AI crawlers?
Decide deliberately — blocking them can reduce AI visibility.
Can you set up Fastly rules?
Yes.
SA
Reviewed by Saeed Ak · Co-founder & CTO25 years engineering high-traffic ecommerce, cloud and security platforms. Written by the VISIBI Security team.Meet the team →