This article covers:
- How to fix Google Search Console Coverage Server Error (5xx) for Magento stores
- How to create a large number of 301 redirects with Google Sheet
Audiences of this article:
- Magento stores experiencing indexing/coverage issues, which is likely to be caused by changing product categories
- This could also apply to other types of eCommerce platforms in need of creating a large number of 301 redirects
Why there are so many server errors when Google crawls your Magento store?
Magento has its built-in URL rewrites mechanism which automatically creates redirects for product URL updates. However, in some cases, a manual redirect is needed when below admin activities take place:
- Some previously active store categories have been deleted
- Admin -> Configuration -> Catalog -> Search Engine Optimisation _> Use Categories Path for Product URLs changed from Yes to No.
The above admin activity will result in previously indexing product URLs get 404 not found, which could be hundreds of thousands. If not taken care of, your store’s search performance will likely to take a toll.
The Solution – an effective way to create a large number of 301 redirects
Thousands of broken URL may sound horrible and tedious to fix, yet there are some pleasing factors about this issue, such as:
- Google search console has allowed us to download a list of 404 URLs (with a maximum number of 1000 at a time)
- In most cases, to fix a product 404 page caused by changing category URLs, all we need to do is redirect the broken URL to the product page without all the categories, see below example:
Redirect 301 https://mystore.com/category1/category2/product.phtml https://mystore.com/product.phtml
Sounds promising right? Let’s tackle this issue step by step.
Step 1 Download a list of 404 URLs
Go to your Google Search Console: Coverage -> Server error (5xx) and click on the Export data button to download as Google Sheet, if you have more than 1000 affected pages, after completing your 301 redirects and submitting for reviews, repeat these steps.
Two Google Sheet Functions for 301 redirects
- REGEXEXTRACT to extract the part of URL after last forward slash
- CONCATENATE to join “Redirect 301 “, “reditrct_url” and “destination_url” together, which can be pasted into your .htaccess file to complete the URL redirection.
Step 2 Extract last part of URL after last forward slash
The actual formula we used is:
=REGEXEXTRACT(A1,"(?:.*/)(.*)$")
Here is an example:
Above will help us get a list of product name URL key which is usually configured as the Canonical product URL. In other words, they are most likely working but we need to test the results before install 301 redirects on the server. Remember: always test your destination URLs!
Step 3 Test your destination URLs
When dealing with thousands of 301 redirects, we have to make sure each destination URL is working. We achieve this by running a custom script against the destination URL column. Open your Google Sheet -> Data -> Script Editor and add the function below:
function getStatusCode(url_key){
// add your store domain name for testing
var product_url = "https://eharvest.com.au/" + url_key
var response = UrlFetchApp.fetch(product_url);
return response.getResponseCode();
}
The above function will return 200 is the destination URL is working, otherwise it will return [#ERROR!], which we’ll have to take care of them manually. It could be products which are no longer available, so redirect them to similar products. See examples below:
Step 4 Form the final 301 Redirect Code by using CONCATENATE function
The actual formula we used here is:
=CONCATENATE("Redirect 301 ", A1, " ", "/", B1)
See example below:
If you’re familiar with Javascript, you can also write your own Google Script the achieve the same results as CONCATENATE function.
Now copy and paste the column C to the top of your .htaccess file to install 301 redirections, and don’t forget to test it. Once done you can go back to your Google Search Console and start the validation process.
If you have more than a thousand broken URLs, you will need to monitor your Search Console regularly to check your validation process. Once new errors appear in your example listing, download them and start over until all the errors are fixed.
