Posted May 27, 2020 • 7 min read
1. Introduction to Persistence
Because Redis is an in-memory database, it stores its database state in memory, so if you do not want to save the database state stored in memory to disk, then once the server process exits, the database state in the server will disappear.
Therefore Redis provides two persistence solutions, RDB(Redis DataBase) and AOF(Append Only File).
Second, RDB persistence
RDB is Redis' default persistence solution. RDB persistence can be performed manually or periodically based on server configuration options. This function saves the state of the database at a certain point in time to an .rdb file. Redis restart will restore the data by loading the .rdb file.
2.1 RDB manual backup and recovery
There are two redis commands that can be used to generate RDB files:
- save:The save command will block the redis server process until the rdb file is created. During the server process blocking, no command request can be processed;
- bgsave:The bgsave command will spawn a child process. The child process is responsible for creating the rdb file, and the parent process continues to process the redis command.
The data recovery of the rdb file is automatically completed when redis starts. There is no special command to manually perform the recovery. It is worth noting that the AOF update frequency is usually higher than the RDB update frequency. Redis prefers to use AOF to restore data, and only uses ADB to restore data when AOF persistence is turned off.
# Block the persistence of the main process 127.0.0.1:6379> save OK # Use child process persistence 127.0.0.1:6379> BGSAVE Background saving started # Execute any command will generate /var/lib/redis/dump.rdb file # rdb file recovery # View the storage location of the rdb file 127.0.0.1:6379> CONFIG GET dir 1) "dir" 2) "/var/lib/redis" # Place the RDB file to be restored under/var/lib/redis # Restart redis(note:don't use service redis-server restart/stop/start to restart, otherwise there will be some inexplicable error, rdb cannot be restored) redis-cli shutdown sudo redis-server /etc/redis/redis.conf # Log in to redis-cli and you will see the recovered data
2.2 RDB configuration
save 900 1 # Time strategy, the server has modified the database at least once within 900 seconds. save 300 10 # Time strategy, the server has modified the database at least 10 times within 300 seconds. save 60 10000 # Time strategy, the server has modified the database at least 1000 times within 60 seconds. dbfilename dump.rdb #file name dir/var/lib/redis #File save path stop-writes-on-bgsave-error yes # If the persistence error occurs, whether the main process stops writing rdbcompression yes # Whether to compress rdbchecksum yes # Whether to check during import # Through save 300 10 times, we can insert 10 data in 5 minutes, and in 5 minutes, we found that a new dump.rdb file was generated, indicating that the configuration takes effect
So why do you need to configure so many rules? Because Redis' read and write requests in each period are definitely not balanced, in order to balance performance and data security, we can freely customize when to trigger backups. So here is a reasonable configuration based on its own Redis write situation.
2.3 Advantages and Disadvantages of RDB
- Suitable for large-scale data recovery.
- If the business does not require high data integrity and consistency, RDB is a good choice.
- The data integrity and consistency are not high because the RDB may be down during the last backup.
- Memory is used during backup, because Redis will create a child process independently during backup, write data to a temporary file(the total memory occupied at this time is twice the original), and finally replace the temporary file with the previous backup file. Therefore, it is reasonable to choose to perform Redis persistence and data recovery at night.
3. AOF persistence
Redis is not enabled by default. It appears to make up for the lack of RDB(inconsistent data), so it uses the form of a log to record each write operation, and append to the file. When Redis restarts, it will execute the write command once from front to back according to the content of the log file to complete the data recovery work.
3.1 AOF persistence steps
Command append:When AOF mode is turned on, the server will append the executed write command to the end of the aof \ _buf buffer in the protocol format every time the server executes the write command;
Writing and synchronization of AOF files:Every time Redis executes an event loop, it is necessary to consider whether to write the data of the buffer to the AOF file. The write synchronization strategy is determined by the append_sync of the configuration file:append_sync There are three values
always:write and synchronize all contents of the buffer to the AOF file;
everysec(default):Write the contents of the buffer to the AOF file. If the time between the last synchronization of the AOF file and the current time exceeds 1 second, then synchronize the AOF file here, and this synchronization operation is dedicated by a thread. implemented;
no:Write the contents of the buffer to the file without synchronizing the AOF. When the synchronization is determined by the operating system.
In order to improve the efficiency of writing files, in modern operating systems, when the user calls the write function to write some data to the file, the operating system usually temporarily saves the written data in a memory buffer and waits until the buffer After the space is filled or exceeds the specified time limit, the data in the buffer is actually written to the disk. Although this approach improves efficiency, it also poses a security issue for writing data, because if the computer goes down, the write data stored in the memory buffer will be lost. To this end, the system provides two synchronization functions fsync and fdatasync, they can force the operating system to immediately write the data in the buffer to the hard disk, thereby ensuring the safety of the written data.
3.2 AOF rewriting mechanism
Because the AOF backup strategy is to save all write commands, when the write commands continue to increase, the AOF file will become larger and larger, which may affect system performance and data recovery performance. To solve this problem, Redis provides AOF file rewrite(rewrite) function. Through this function, the Redis server can create a new AOF file to replace the existing AOF file. The new and old AOF files keep the same database state, but the new AOF file will not contain any redundant commands that waste space, so the new The volume of AOF files is usually much smaller than that of old AOF files.
So how does it work? Although the name is called AOF rewriting, the new file will not read and analyze the old AOF file, but by reading the status of the current database. The process is to first read the current value of the key from the database, and then use a command to record the key-value pair, instead of multiple commands that previously recorded this key-value pair. This is the implementation principle of the AOF rewrite function.
Handling of inconsistent data
Redis uses a child process to handle the rewriting of AOF, which will lead to data inconsistency problems. For example, when the rewriting starts, the database only has key1, and key2 has been added during the rewriting process. Then, after rewriting, there is only key1 in AOF , Leading to inconsistencies in the data. In order to solve this data inconsistency problem, the Redis server sets an AOF rewriting buffer. This buffer is used after the server creates a child process. When the Redis server executes a write command, it will send the write command to the same time. AOF buffer and AOF rewrite buffer.
When the child process completes the AOF rewriting work, it will send a signal to the parent process. After receiving the signal, the parent process will call a signal processing function and perform the following tasks:
- Write all the contents of the AOF rewrite buffer to the new AOF file, then the database state saved by the new AOF file will be consistent with the current database state of the server.
- Rename the new AOF file, atomically overwrite the existing AOF file, and complete the replacement of the old and new AOF files. After the signal processing function is executed, the parent process can continue to accept command requests as usual.
In the entire AOF background rewriting process, only the signal processing function will block the server process(parent process) when executed. At other times, the AOF background rewriting will not block the parent process, which will rewrite the AOF The impact on server performance is minimized.
3.3 AOF configuration and recovery
appendonly yes # yes means enable AOF persistence appendfilename "appendonly.aof" appendfsync everysec # Data write synchronization strategy(everysec, always, no) #Set to yes means no fsync for new write operations during rewrite, temporarily stored in memory, and write after rewrite is complete, the default is no no-appendfsync-on-rewrite no #When the current AOF file size is twice the size of the last AOF file obtained by the last log rewriting, a new log rewriting process is automatically started. auto-aof-rewrite-percentage 100 #The minimum value of the current AOF file to start a new log rewriting process to avoid frequent rewriting due to the small file size when Reids is just started. auto-aof-rewrite-min-size 64mb
# Restart redis after configuration, write a few pieces of data, and find that the AOF file is generated # First move the aof file to another directory, then clear all the data, and then move it back. After restarting redis, the data is restored.
appendonly.aof is actually a text file, so you can manually modify the AOF file to restore it to the specified data:
# file appendonly.aof appendonly.aof:ASCII text, with CRLF line terminators # cat appendonly.aof *2 $6 SELECT $1 0 * 3 $3 ... omitted N lines here set $5 name4 $1 4
3.4 AOF advantages and disadvantages
Higher data integrity and consistency
Because AOF records more content, the file will become larger and larger, and data recovery will also become slower.
- Redis enables the RDB persistence mode by default. Within a specified time interval, when a specified number of write operations are performed, the data in memory is written to the disk.
- RDB persistence is suitable for large-scale data recovery but its data consistency and integrity are poor.
- Redis needs to manually enable the AOF persistence method. The default is to append the write log to the AOF file every second.
- The data integrity of AOF is higher than that of RDB, but there are more records, which will affect the efficiency of data recovery.
- Redis provides a rewritten slimming mechanism for AOF files.
- If you only intend to use Redis as a cache, you can turn off persistence.
- If you intend to use Redis persistence. It is recommended that both RDB and AOF be enabled. In fact, RDB is more suitable for data backup, stay behind. There is a problem with AOF, and RDB.
- Where are the files stored after RDB and AOF persisted?
RDB files are stored in
/var/lib/redis/dump.rdb, the actual location depends on
dir/var/lib/redis in the configuration file
/etc/redis/redis.conf Use the command
config get dir in cli.
- After AOF persistence is enabled, is it just to record the write operation after the opening time point, then how to ensure the backup of the previous data?
When AOF is rewritten, all write statements are generated based on the current database.