How to Send Bulk 410 Headers and 301 Redirects in Nginx

Google's Webmaster Tools warns you when there are a lot of 404 Not Found pages found on your website. If you've just deleted a lot of spam from a site, Google might keep recrawling those URLs as long as they send 404 headers.

A 404 header means Not Found.
A 410 header means Gone.

If you want to stop Google from adding those removed pages back to Google Webmaster Tools, you can send 410 headers for the removed URLs.

It took me a while to figure out how to do this with nginx, but I finally came up with this solution:

Create a new file called /etc/nginx/header-maps.conf. In that file map each gone URL to a variable, in this case the number 1 (chosen arbitrarily).

  1. map $request_uri $gone_var {
  2. /path-one 1;
  3. /path-two 1;
  4. /path-three 1;
  5. /path-four 1;
  6. }

Then in your main conf file for your domain, like /etc/nginx/sites-available/example.com (replacing example.com with your domain):

  1. include header-include.conf;
  2. server {
  3. # other things here...
  4.  
  5. if ($gone_var) {
  6. return 410 $gone_var;
  7. }
  8.  
  9. # other things here...
  10. }

You can also map redirects in the same file, for example:

  1. # header-include.conf
  2. map $request_uri $gone_var {
  3. /path-one 1;
  4. /path-two 1;
  5. /path-three 1;
  6. /path-four 1;
  7. }
  8.  
  9. map $request_uri $redirect_uri {
  10. /path-five /path-five.html;
  11. /path-six /path-six.html;
  12. /path-seven /path-seven.html;
  13. }
  1. # Your domain's conf file
  2. include header-include.conf;
  3. server {
  4. # other things here...
  5.  
  6. # if there is a mapped $gone_var set for the current path
  7. if ($gone_var) {
  8. return 410 $gone_var;
  9. }
  10.  
  11. # if there is a $redirect_uri set for the current URI:
  12. if ($redirect_uri) {
  13. return 301 $redirect_uri;
  14. }
  15.  
  16. # other things here...
  17. }

If you get an error about map_hash_bucket_size being to small, open the /etc/nginx/nginx.conf file (or wherever the http block lives) and increase it something like this: map_hash_bucket_size 256

map_hash_bucket_size nginx

More information:

I don't know yet how this affects performance, but it solved my current problem of sending 410 headers on a couple of hundred URLs.

If it doesn't work for you or if you have any suggestions for improving this idea, please comment below.