What is the use of Nginx's filtering module?

Posted May 27, 20206 min read

In the last article I wrote 11 stages of Nginx , many people said it was too long. This is for the sake of the completeness of the article. Well, in 11 stages, it's done in one go. Today this article is relatively short, no problem after reading it.

Filter module location

Earlier we introduced 11 stages of Nginx. In the content stage, Nginx will generate the response content returned to the user. In fact, the user's response content needs to be reprocessed. The Nginx filtering module is to reprocess the response content. Processed. So in fact the filtering module is located after the content stage and before the log stage.

Let's first look at a section of configuration instructions:

limit_req zone = req_one
burst = 120;
limit_conn c_zone 1;

satisfy any;
allow 192.168.1.0/32;
auth_basic_user_file access.pass;

gzip on;
image_filter resize 80 80;

What kind of request process will be followed under this configuration command? Please take a look at the following picture:

The process of the above picture is roughly speaking, if you don't understand the 11 stages of Nginx, go through the previous article.

Let me briefly talk about it here. First, the HTTP request is received by the Nginx framework. After processing in the preaccess, access, and content stages, when the response is generated after the static module, the response needs to be processed many times before it is returned to the client.

Here we assume that if the response is an image, then when you need to make a thumbnail, you must first go through the image \ _filter module. There is also a gzip module, these two modules also need to follow a strict order. Because if you do gzip compression first, there is no way to do it after the thumbnail.

The second concern is that the header is filtered first, and then the body is filtered. When we send a response to the user, we must first send the header, and then send the body, so all filtering modules will provide filtering of the header or body, of course, the image \ _filter and gzip modules can filter both.

Return response

As we said before, the 11 stages of Nginx have a strict order, and this order exists in the form of an array in the Nginx code, and the order of this array is from back to front. When returning a response to the user, the filtering module also has a strict order, which is also from back to front. Looking at the definition in the code, the red ones are the filtering modules I will mention below.

We need to focus on four filtering modules, what are their respective functions?

  • copy \ _filter:copy the contents of the package

    When we use the sendfile command, which is zero-copy technology, it does not pass through user-mode memory. This is directly sent to users without passing through Nginx, and when the gzip module is also used. Because gzip must compress the data in memory, then copy \ _filter will invalidate the sendfile instruction. Some modules do not need to process the data in memory, they need to be processed before the copy \ _filter module.

  • postpone \ _filter:handle sub-requests

    Used to process sub-requests, some filtering modules need to care about the processing results of sub-requests and need to be placed after this module.

  • header \ _filter:construct the response header

    It is used to construct the response header that is finally sent to the user, and some content such as Server and Nginx version number may be added.

  • write \ _filter:send response

    Used to actually call the operating system's write or send system calls to actually send the response.

After introducing the functions and stages of the filtering module, let's look at the two modules in detail.

sub module

Introduce a module that can replace the string content in the response:the sub module.

  • Function:Replace the character string specified in the response with a new character string
  • Module:ngx_http_sub_filter_module module, which is not compiled into Nginx by default, enabled by --with-http \ _sub \ _module

Directive

Syntax:sub_filter string replacement;
Default: 
Context:http, server, location

Syntax:sub_filter_last_modified on | off;
Default:sub_filter_last_modified off;
Context:http, server, location

Syntax:sub_filter_once on | off;
Default:sub_filter_once on;
Context:http, server, location

Syntax:sub_filter_types mime-type ...;
Default:sub_filter_types text/html;
Context:http, server, location

Let me explain what these four instructions mean.

  • sub_filter string replacement

    The sub_filter instruction will replace the matched string string with the string represented by replacement.

  • sub_filter_last_modified on | off

    The sub_filter_last_modified instruction means whether to return the original last_modified HTTP header, because we have modified the file content, if it is on, it will continue to return to the original header.

  • sub_filter_once on | off

    The meaning of sub_filter_once is whether to replace it only once, and it is turned on by default. If it is set to off, it will scan and replace all the contents in the response.

  • sub_filter_types mime-type

    This instruction is to replace those file types, which can be set to \ *, but the efficiency will be lower, and it needs to be considered according to the actual situation.

Actual combat

The configuration file is as follows:

server {
    server_name sub.ziyang.com;
    error_log logs/myerror.log info;

    location/{
        sub_filter 'Nginx.oRg' '$host/nginx';
        sub_filter 'nginX.cOm' '$host/nginx';
        #sub_filter_once on;
        sub_filter_once off;
        #sub_filter_last_modified off;
        sub_filter_last_modified on;
    }
}

Here, Nginx needs to be recompiled. I also compiled the modules needed in the next section together:

./configure --prefix =/Users/mtdp/myproject/nginx/test_nginx --with-http_sub_module --with-http_addition_module --with-http_realip_module
make
cp nginx ../../test_nginx/sbin/# Copy the compiled nginx to the previous directory

Perform hot deployment:

The process of hot deployment is detailed in Nginx Getting Started and Command Line Operations

kill -USR2 87693 # Use the new Nginx binaries to provide services
kill -WINCH 87693 # Quit the old Nginx worker process
kill -quit 87693 # Gracefully quit the old master process

Open sub.ziyang.com in your browser:

Here you will find that nginx.org has been replaced by sub.ziyang.com/nginx, and nginx.com has also been replaced by sub.ziyang.com/nginx.

addition module

Let's look at another filtering module, addition module, which can add content before and after the response.

  • Function:add content before or after the response, the way to add content is by adding a sub-request, according to the response of the sub-request to complete.

  • Module:ngx_http_addition_filter_module

    Not compiled into Nginx by default, enabled by --with-http \ _addition \ _module

Directive

Syntax:add_before_body uri;
Default: 
Context:http, server, location

Syntax:add_after_body uri;
Default: 
Context:http, server, location

Syntax:addition_types mime-type ...;
Default:addition_types text/html;
Context:http, server, location

The three instructions here are relatively simple. Let s talk about the uri behind add_before_body and add_after_body. This means that we initiate a sub-request to the specified uri and add content based on the response of the sub-request.

The addition_types directive specifies the file types to be added.

Actual combat

The configuration file is as follows:

server {
    server_name addition.ziyang.com;
    error_log logs/myerror.log info;

    location/{
        #add_before_body/before_action;
        #add_after_body/after_action;
        #addition_types *;
    }
    location/before_action {
        return 200 'new content before \ n';
    }
    location/after_action {
        return 200 'new content after \ n';
    }
    location/testhost {
        uninitialized_variable_warn on;
        set $foo 'testhost';
        return 200 '$gzip_ratio \ n';
    }
}

Let's first see what effect it will have if the instruction of the addition module is commented out:

  ~ curl addition.ziyang.com/a.txt
a

Then open the comment:

  ~ curl addition.ziyang.com/a.txt
new content before
a
new content after

Added content before and after the original response.

It should be noted that in the actual situation, add_before_body and add_after_body are followed by other uris. For simplicity, I will directly forward them to the corresponding location.

I have put all the configuration files involved in this article in Nginx configuration files , you can pick them up.


This article first appeared on my personal blog: iziyang.github.io