Egg multi-process model considerations

Posted May 29, 20204 min read

background

I recently used egg for server-side development in the project. I encountered a strange problem during the development process. The specific performance is that when mq listens to information, its callback function will be executed multiple times, so this will cause a file to be simultaneously Operation and other issues.

Cause of the problem

When sorting out the egg documents, I focused on the egg multi-process design pattern and learned about the egg's master-agent-worker pattern. Then there are some problems that we need to pay attention to during development.

First introduce the multi-process implementation of egg

Egg implements the multi-process mode through the cluster provided by node. In order to make better use of the multi-core environment, egg will generally enable workers equivalent to the number of cpu cores to maximize the use of cpu capabilities.

When the egg starts, the relationship between master, agent and worker is as shown

+ --------- + + --------- + + --------- +
| Master | | Agent | | Worker |
+ --------- + + ---- + ---- + + ---- + ---- +
     | fork agent | |
     + --------------------> | |
     | agent ready | |
     | <-------------------- + |
     | | fork worker |
     + -----------------------------------------> |
     | worker ready | |
     | <----------------------------------------- +
     | Egg ready | |
     + --------------------> | |
     | Egg ready | |
     + -----------------------------------------> |

In this mode, master, agent, and worker perform their duties, and the main production assignments are as follows:
master: is responsible for maintaining the stability of the entire application. When a worker quits due to an exception, the master is responsible for pulling up a new worker to ensure that the application runs normally.
agent: Since egg's multi-process model will run a copy of our application instance in each process, then in some cases, this mechanism will cause problems. For example, if the logic to save the log is executed in each process, when the log save operation is triggered, there will be multiple processes operating the log file at the same time, then this will cause file read and write problems. Therefore, egg has designed an agent process. There will be only one agent process, and the above problems will not occur. In this way, the logic of the background operation similar to the above is unified into the agent to deal with.
worker: Responsible for executing business code, processing user requests and timed tasks, egg guarantees that timed tasks will only be executed in a single worker in the framework layer, so it can be used with confidence.

Analysis of problems caused by egg multi-process

Above we have analyzed the multi-process mechanism of egg, so we know the cause of the problem. The reason for the problem that we said at the beginning is that we put the mq monitoring and processing logic in the worker, so in the actual operation process This will cause the callback function to be executed multiple times when mq receives the message.

At this point we already know how to optimize, that is, put the processing logic of mq in the agent to ensure that the callback of the mq message is only executed once. But carefully you must have discovered that there is a problem here. The agent has only one instance. If things are done in the agent, is it not possible to take advantage of multi-core performance?

Agent and worker communication

Indeed, we can deal with logic that only needs a single execution in the agent, but this way we can't take advantage of multi-core performance. So is there any way? Yes, it is inter-process communication. The specific idea is that the agent is still responsible for mq connection and monitoring logic, but the callback function is not executed in the agent, but written in the worker. So when does the worker execute this logic? The answer is that the agent notifies the worker through interprocess communication. Egg implements an inter-process communication mechanism, we can call it directly, the main implementation method is as follows:

Broadcast message:agent => all workers
                  + -------- + + ------- +
                  | Master | <--------- | Agent |
                  + -------- + + ------- +
                /| \
               /| \
              /| \
             /| \
             v v v
  + ---------- + + ---------- + + ---------- +
  | Worker 1 | | Worker 2 | | Worker 3 |
  + ---------- + + ---------- + + ---------- +

Specify the receiver:one worker => another worker
                  + -------- + + ------- +
                  | Master | ---------- | Agent |
                  + -------- + + ------- +
                 ^ |
     send to/|
    worker 2/|
             /|
            /v
  + ---------- + + ---------- + + ---------- +
  | Worker 1 | | Worker 2 | | Worker 3 |
  + ---------- + + ---------- + + ---------- +

Here we can see that the inter-process communication is based on master forwarding, so we can use the mechanism provided by egg to solve our problems.

Solution

As analyzed above, we put mq's connection and monitoring logic into the agent. When receiving the message, the notification is sent to the worker through inter-process communication, and then the worker can execute the specific business logic. The specific code can actually refer to vue's event mechanism and listen to specified events in the worker:

app.messenger.on(action, data => {
  //execute business logic
});

Establish an mq connection in the agent and listen to the message. After receiving the message, trigger the event:

exports.task = async ctx => {
  ... //The logic of receiving the mq message is omitted here
  //ready to send notification
  ctx.app.messenger.sendRandom(action);
};

Note that the task that needs to be executed once must call the sendRandom method, which is a method sent to a worker. Of course, if you want to execute multiple times, you can call the app.messenger.sendToApp() method, this method will send the message to all workers, and execute multiple processing logic.

to sum up

The use of egg in the multi-process model still requires some skills, so we need to be familiar with the multi-process mechanism of egg before conducting business development to avoid encountering strange pits and wasting time.