SEO Lesson from SiapPergi: Our Site Suddenly Couldn't Be Indexed by Google Because of Cloudflare Security Rules
By the siappergi.com team · Published · 6 min read

We are writing this plainly. On 10–11 November 2026, while tidying up SEO on siappergi.com, Search Console showed that Google could not fetch our site at all. The mistake was ours. We hope this helps other business owners who use Cloudflare.
The symptom
From phones and laptops, siappergi.com loaded normally (status 200, fast). But Google Search Console's URL Inspection → Test live URL for the home page said: *URL is not available to Google. Page cannot be indexed: Unavailable due to a site-wide issue.* That means Googlebot failed to fetch the site as a whole — not a keyword, title or content problem.
What was not the cause
- Keyword cannibalisation (several pages on one topic) can make rankings fluctuate — it does not remove a site from the index.
- Targeting the same keywords as competitors is normal and not penalised.
- llms.txt is not used by Google for indexing or ranking.
- A sitemap with weekly `changefreq` blocks nothing — Google simply ignores the tag.
The cause we found: Cloudflare security was too strict
We had a "block fake bots" rule (blocking requests that claim to be Googlebot but are not recognised as verified bots), Security Level medium, Browser Integrity Check, and on one domain Bot Fight Mode. They were meant to protect the site, but protections like these can also stop Google tools (such as the inspection tool or the robots.txt fetcher) if they are not recognised correctly. When robots.txt cannot be fetched, Google treats the whole site as unavailable.
What we changed
- Block real attacks only — the remaining rules reject scanners probing for .env, .git, wp-login, phpmyadmin, .php and backup files. Our sites do not have such files, so real visitors and search engines are never affected.
- The fake-bot rule is off, Bot Fight Mode is off, Security Level is *essentially off*, Browser Integrity Check is off. Cloudflare's built-in DDoS protection stays on.
- Google (ASN 15169), Microsoft/Bing (ASN 8075) and /robots.txt are exempt from every blocking rule.
- Sitemap cleaned up — `changefreq` & `priority` removed, `lastmod` only where the update date is known.
- All URLs resubmitted via IndexNow (Bing, Yandex, Seznam, Naver), and Google asked to recrawl in Search Console (resubmit sitemaps & Request indexing).
Checklist for website owners on Cloudflare
- Run Search Console → URL Inspection → Test live URL on your home page. Green = fine; red "site-wide issue" = check security rules & robots.txt.
- Open Settings → Crawl stats → Host status to see whether robots.txt fetch, DNS or server connectivity failed.
- In Cloudflare, don't block by user-agent name alone; exempt official search-engine networks and robots.txt if you must.
- Turn off Bot Fight Mode if your business depends on Google Search — it can stop good bots too.
- After every security change, re-run the live test before assuming all is well.
Testing from your own phone is not enough — you are not Googlebot. Only Search Console's live test tells you whether Google can get in.
Status now
The changes are live on all our domains. We are monitoring recovery in Search Console and will update this article. Our services run as usual: scooter rental in Bali, Bali rentals and car with driver in Yogyakarta.
Questions or corrections: our only official email is halo@siappergi.com.
Ready to rent a scooter or car?
Fixed catalogue prices, delivered to your hotel or villa. Book on WhatsApp and pay when the vehicle arrives.
All areas & scooter models
Frequently asked questions
What does "Unavailable due to a site-wide issue" mean?
Googlebot failed to fetch your site as a whole — usually robots.txt could not be fetched, the server/DNS failed, or Googlebot was blocked by a security rule.
Can keyword cannibalisation stop a site from being indexed?
No. At most Google switches between your pages so rankings fluctuate; the site stays indexed.
Can Cloudflare block Googlebot?
Yes, if security is too strict (user-agent based rules, Bot Fight Mode, high security level). Check with URL Inspection → Test live URL.
How do I ask Google to come back?
Resubmit your sitemap in Search Console and press Request indexing in URL Inspection. Google's old "sitemap ping" was retired in 2023.
This guide follows the SiapPergi editorial policy: facts are checked, market prices are given as ranges and mistakes are corrected. Spotted something wrong? Report a correction.







