If your Melbourne business’s website traffic has quietly dropped over the past few months, the cause might not be your content or your competitors. It could be something far less visible: crawl errors. A crawl error happens when Googlebot, the automated program Google uses to read and catalogue web pages, tries to visit a page on your site and fails. Left alone, crawl errors waste the limited time Google spends on your site each week and slowly push your pages down the results. This guide covers what causes crawl errors, how to find them, and what fixing them costs.
What a crawl error actually is, and why it matters
A crawl error is any time Googlebot tries to reach a page on your website and something stops it: a broken link, a server timeout, a page blocked by your robots.txt file (the small text file telling search engines which parts of your site they may visit), or a redirect that loops back on itself. Google does not crawl your whole site every day. It allocates a rough budget of attention, known as crawl budget, based on your site’s size and how reliably it responds. If Googlebot keeps hitting dead ends on your Camberwell cafe’s booking page, it spends less time finding new content. For a forty-page local business site this rarely causes total collapse, but it slows down indexing and can quietly drag down rankings for pages Google struggles to reach.
The crawl errors we see most often on Melbourne business sites
After auditing hundreds of Melbourne sites, from Fitzroy cafes to Dandenong logistics firms, the same handful of issues show up again and again. Some are urgent, others can wait until your next scheduled update.
| Crawl error type | What it means | Typical fix | Priority |
|---|---|---|---|
| 404 Not Found | The page has been deleted or moved and nothing points to its new location. | Restore the page, or set up a 301 redirect to the closest relevant page. | Medium |
| Soft 404 | The page loads and returns a “200 OK” code, but shows “out of stock” or “not found” content, confusing Google about whether it is real. | Either restore genuine content or return a proper 404 or 410 status code. | High |
| 5xx server error | Your server failed to respond, often during a traffic spike or hosting issue. | Check hosting capacity and server logs with your developer. | Urgent |
| Redirect chain or loop | A page redirects to another redirect, sometimes back to itself. | Point every redirect straight to its final destination. | Medium |
| Blocked by robots.txt | Your own site is telling Google not to crawl a page you actually want indexed. | Edit the robots.txt file to allow the page. | High |
Not sure which apply to your site? Our team offers a free 30-minute technical audit checking this exact list against your live pages, no obligation attached. You can book your free SEO audit and get a plain-language answer within a day.
How to find crawl errors in Google Search Console
You do not need expensive software to start looking. Google Search Console is free, and if you have not set it up yet, that is worth doing before anything else. Google’s own Crawl Stats report guide explains what each metric means if you want the technical detail. Once it is connected to your website, the process is straightforward:
- Open the Pages report under the Indexing menu. This shows every page Google has tried to index, split into “Indexed” and “Not indexed”.
- Click into the reasons listed under “Not indexed”, such as “Not found (404)”, “Server error (5xx)”, or “Blocked by robots.txt”.
- Check the Crawl Stats report, tucked under Settings, to see how many requests Googlebot is making and how often it is hitting errors.
- Export the affected URLs and work through them by priority, fixing urgent server errors first and leaving cosmetic issues for last.
Search Console Pages report
the error type and affected URLs
the underlying cause, not just the symptom
validation or re-indexing in Search Console
What is usually causing it on Melbourne business sites
In our experience, three things account for most of the crawl errors on small and mid-sized Melbourne sites. The first is a website migration or redesign that changed URLs without setting up redirects, common when a business moves from a template builder to a custom WordPress site. The second is stock or menu changes on e-commerce and hospitality sites, where a product or dish is removed and the page is left live with no content, creating a soft 404. The third is shared hosting that struggles under load, particularly on budget hosting plans that see a spike in traffic from a Google Ads campaign or a mention in local news. None are especially difficult to fix, they are just easy to miss if nobody checks Search Console regularly. Our practical guide to technical SEO for Melbourne businesses covers the fuller picture, including Core Web Vitals and schema markup. It is also worth understanding how your sitemap and robots.txt file interact, since a mistake in either can quietly create new crawl errors; our beginner’s guide to XML sitemaps and robots.txt walks through both in plain language.
If a recent site migration is behind your errors, it is worth having a second set of eyes check your redirect map before it causes further ranking loss. Talk to our technical SEO team about a migration health check. Most Melbourne businesses get a response within the same week.
What this costs, how long it takes, and what goes wrong
Fixing crawl errors is rarely expensive on its own. A straightforward clean-up, covering redirect mapping, robots.txt corrections, and resolving soft 404s, typically takes a competent technical SEO consultant between three and eight hours, which in Melbourne generally costs between $450 and $1,600 depending on the size of the site and how tangled the redirect history is. Server-related errors can cost more if they require a hosting upgrade, which for a small business site usually adds $20 to $60 a month.
Timeframes vary too. Once a fix is live, Google typically re-crawls a small site within a few days to two weeks. Larger sites, or sites that update less often, can take four to six weeks before the change is fully reflected. This lines up with how much more Google is crawling generally: Googlebot’s crawl requests grew by 96% between May 2024 and May 2025, and Googlebot alone now generates around 4.5% of all HTML request traffic on the web, according to Cloudflare’s 2025 Radar Year in Review. More crawling activity means errors get discovered faster, but a badly configured site also accumulates them faster.
The most common mistake we see is redirecting every broken page straight to the homepage instead of the closest matching page. This passes almost no relevance signal to Google and looks uncomfortably close to a soft 404 in its own right. Google’s Search Advocate Gary Illyes confirmed at Search Central Live in 2025 that soft 404s keep consuming crawl budget even though they return a normal 200 OK status, unlike a genuine 404 which Google quickly stops requesting. The second common mistake is fixing the symptom, such as manually resubmitting a URL, without fixing the underlying cause, so the error reappears within weeks.
Frequently asked questions
For broader questions about working with an SEO agency, our FAQ section covers timelines, contracts, and pricing. A few crawl-specific questions come up often.
Do crawl errors get better on their own?
Occasionally, if the error was temporary, such as a brief server outage. Persistent errors, like broken links, missing pages, or blocked resources, will not resolve without action.
Will fixing crawl errors improve my rankings immediately?
Not immediately. Clearing crawl errors removes a barrier to ranking, but it usually takes several weeks after Google re-crawls the site for the benefit to show up in traffic.
How often should I check for crawl errors?
Monthly is a reasonable minimum for a small business site. Sites that change frequently, such as online stores, benefit from checking fortnightly.
Getting it fixed properly
Crawl errors are one of the few technical SEO problems that get cheaper and faster to fix the earlier you catch them. Left for a year, a small tangle of broken redirects can turn into hundreds of lost pages and a real dip in enquiries. These are the same fixes behind results such as a Melbourne homewares retailer’s rise to $2.1 million in annual organic revenue, covered on our results page. Our process always starts with a full technical audit before any content or link work begins. If you would like a straightforward look at where your own site stands, browse more articles on our SEO blog or get in touch directly. Our Melbourne-based team offers a free, no-obligation audit with a response within 24 hours and no lock-in contracts, so there is genuinely nothing to lose by asking us to check your site today.
Leave a Reply