How to use crawl stats in Google Search Console to identify technical SEO issues
The crawl stats feature in Google Search Console provides information about how Google’s crawlers are interacting with your website.
It can help you understand:
- how frequently your site is being crawled,
- which pages on your site are being crawled the most, and
- any crawl errors that the crawlers have encountered.
And the best part? There’s no need to wait for log files (although crawl stats offers a fraction of the insights).
Before diving into a full technical SEO audit, one of the first things I like to look at the crawl stats feature which is accessible in the settings panel of GSC.
In this article, I’m going to share with you 5 things I look for to quickly find out if there are any issues that have contributed towards a loss in organic traffic and/or keyword rankings. These are some of the items you will find in my FREE technical SEO checklist.
Let’s get started!
Are there any host status problems reported in Search Console crawl stats?
How to check if there are crawl stat problems in GSC:
- Log into Google Search Console
- On the left hand side navigation, click on SETTINGS
- Then click on OPEN REPORT in the Crawl stats section
- In the Hosts dashboard, check the Status column:
- If you see a green tick icon, mark ‘no’ in the checklist
- If you see a red exclamation icon, mark ‘yes’ in the checklist.
Possible answers:
- yes
- no
- not applicable
What this means:
> If ‘yes’, Google encountered a problem with your server or web host in the last 7 days. If the crawler consistently encounters 4XX and 5XX response codes for a long period of time, your URLs can drop out of Google’s index.
>> If ‘no’, Google didn’t encounter any significant crawl availability issues on your site in the past 90 days.
Recommended reading:
- ‘crawl stats report’ by Search Console Help
Are there crawl requests being made as reported in Search Console crawl stats?
How to find out if Google has made crawl requests in the last 90 days in GSC:
- Log into Google Search Console
- On the left hand side navigation, click on SETTINGS
- Click on OPEN REPORT in the Crawl stats section
- Look at TOTAL CRAWL REQUESTS
- If the number > 0, check ‘yes’ in the spreadsheet
- If the number 0, check ‘no’ in the spreadsheet
Possible answers:
- yes
- no
- not applicable
What this means:
> If ‘yes’, Google crawled the website in the last 90 days. If you think there are crawling issues, this data point indicates that the issue (or issues) is on your end and not due to Google’s inability to crawl the website.
>> If ‘no’, Google did not crawl any URLs in the last 90 days. This is a symptom that something crawling-related requires further investigation. I recommend checking if crawlers or assets are being blocked in robots.txt and if a sitemap has been submitted to Search Console.
Recommended reading:
- ‘crawl stats report’ by Search Console Help
Are there crawl requests being made for “refresh” purposes in Search Console crawl stats?
How to find out if Google has made crawl requests for refresh reasons in GSC:
- Log into Google Search Console
- On the left hand side navigation, click on SETTINGS
- Click on OPEN REPORT in the Crawl stats section
- Scroll down until you see BY PURPOSE
- If you see REFRESH followed by a % greater than zero, check ‘yes’ in the spreadsheet
- If you see a 0% next to REFRESH, check ‘no’ in the spreadsheet
Possible answers:
- yes
- no
- not applicable
What this means:
> If ‘yes’, Google crawled the website in the last 90 days and re-crawled known URLs. If you think there are crawling issues, this data point indicates that the issue (or issues) is on your end and not due to Google’s inability to crawl the website.
>> If ‘no’, Google did not re-crawl any URLs in the last 90 days. This is a symptom that something crawling-related requires further investigation. I recommend checking if crawlers or assets are being blocked in robots.txt and if a sitemap has been submitted to Search Console.
Recommended reading:
- ‘crawl stats report’ by Search Console Help
Are there crawl requests being made for “discovery” purposes in Search Console crawl stats?
How to find out if Google has tried to discover new URLs via GSC:
- Log into Google Search Console
- On the left hand side navigation, click on SETTINGS
- Click on OPEN REPORT in the Crawl stats section
- Scroll down until you see BY PURPOSE
- If you see DISCOVERY followed by a % greater than zero, check ‘yes’ in the spreadsheet
- If you see a 0% next to DISCOVERY, check ‘no’ in the spreadsheet.
Possible answers:
- yes
- no
- not applicable
What this means:
> If ‘yes’, Google has crawled new URLs in the last 90 days.
>> If ‘no’, Google did not any new URLs in the last 90 days. This is a symptom that something crawling-related requires further investigation. I recommend checking if crawlers or assets are being blocked in robots.txt and if a sitemap has been submitted to Search Console.
Recommended reading:
- ‘crawl stats report’ by Search Console Help
According to Search Console crawl stats, have HTML files been crawled in the last 72-hours?
How to find out if Google has crawled HTML files in GSC:
- Log into Google Search Console
- On the left hand side navigation, click on SETTINGS
- Click on OPEN REPORT in the Crawl stats section
- Scroll down until you see BY FILE TYPE
- If you see HTML followed by a % greater than zero, check ‘yes’ in the spreadsheet
- If you see a 0% next to HTML, check ‘no’ in the spreadsheet.
Possible answers:
- yes
- no
What this means:
> If ‘yes’, Google has crawled HTML files on the website in the last 72 hours. This indicates that Google has been able to crawl some sections of the website.
>> If ‘no’, Google has not crawled any HTML files on the website in the last 72 hours. This is a significant red flag and warrants further investigation as to why it was unable to do so. For example, review robots.txt, site-wide meta tags, and server status.
In closing
A full technical SEO audit takes a lot of time – even when you’re following checkboxes.
In most cases, an audit is a waste of time and by looking for clues in Search Console crawl stats, you can get confirmation on what areas to investigate so that time and resources can be better spent.
