Serverless practice in large-scale data processing

Posted Jun 15, 20208 min read

Yunqi Information:[ Click to view more industry information ]
Here you can find first-hand information on the cloud in different industries, what are you waiting for, come on!


When you first came into contact with Serverless, there was a less obvious new way of using it:Compared with the traditional server-based method, the Serverless service platform can make your application scale horizontally and the work of parallel processing is more effective. This is mainly because Serverless does not have to pay for idle resources, and does not have to worry about not enough resources being reserved. In the traditional usage paradigm, users must reserve hundreds or thousands of servers to do some highly parallel tasks with short execution time, and must pay for each server, even if some servers are no longer working. .

Taking Alibaba Cloud Serverless products-function calculation as an example, you can perfectly solve all your above concerns:

  • If your task itself is not very computationally intensive, but there are a large number of concurrent task requests that need to be processed in parallel, such as multimedia file processing, document conversion, etc.;
  • A task itself is computationally intensive, requiring a single task to be processed quickly, and it can also support multiple tasks in parallel.

In this scenario, the only concern of the user is:your task can be divided and disassembled and subtasks can be processed in parallel. A long task that takes an hour to complete can be broken down into 360 independent 10 seconds Long subtasks are processed in parallel so that tasks that previously took you an hour to process can now be completed in only 10 seconds. Because the model of pay-as-you-go is used, the amount of calculation and cost of completion are roughly equivalent, while the traditional model is definitely wasteful because of the reserved resources, and the wasted costs need to be borne by you.

Next, we will elaborate on the practice of Serverless in large-scale data processing.

Extreme elastic expansion and contraction to deal with calculation fluctuations

Before introducing related large-scale data processing examples, here is a brief introduction to function calculation.

1. Introduction to function calculation


  • Developers use programming languages to write applications and services. For development languages supported by function calculation, please refer to the list of development languages;
  • Developer upload application to function calculation;
  • Trigger function execution:trigger methods include OSS, API gateway, log service, table storage, function calculation API, SDK, etc.;
  • Dynamic expansion to respond to requests:function calculation can automatically expand the capacity according to the user's request, the process is transparent and unconscious to you and your users;
  • Billing is based on the actual execution time of the function:after the execution of the function, you can check the execution cost through the bill, the charging granularity is accurate to 100 milliseconds.

At this point, you can roughly understand how the function calculation works. Next, we will explain it with a large number of video parallel transcoding cases:Assuming a home education or entertainment related company, the teacher's video or new source is generally centralized. Generated, and you want these videos to be quickly transcoded to allow customers to quickly see the video playback. For example, in the current epidemic, the courses generated by online education have surged, and the peaks of classes are generally obvious peaks such as 10 o'clock, 12 o'clock, 16 o'clock, and 18 o'clock. All new courses are processed within a specific time(such as half an hour) The uploaded video is a universal and universal demand.

2. Flexible audio and video processing system with high availability

  • OSS trigger


As shown in the figure above, the user uploads a video to the OSS, the OSS trigger automatically triggers the function execution, the function calculation is automatically expanded, the function logic in the execution environment calls FFmpeg to transcode the video, and save the transcoded video back to the OSS.

  • Message trigger


As shown in the figure above, the application only needs to send a message to automatically trigger the function to perform the audio and video processing tasks. The function calculation automatically expands, the function logic in the execution environment calls FFmpeg for video transcoding, and saves the transcoded video Back to OSS.

  • Directly call SDK to perform audio and video processing tasks

Take Python as an example, roughly as follows:

python # -*- coding:utf-8 -*-
import fc2
import json
client = fc2.Client(endpoint="",accessKeyID="xxxxxxxx",accessKeySecret="yyyyyy")
# Optional synchronous/asynchronous call
resp = client.invoke_function("FcOssFFmpeg", "transcode", payload=json.dumps(

From the above, we can also see that there are many ways to trigger the execution of the function. At the same time, simply configuring the SLS log can quickly realize a flexible and high-availability, pay-per-view audio and video processing system, while providing free operation and maintenance, specific Dashboard with powerful functions such as business data visualization and powerful custom monitoring alarms.


At present, the audio and video cases that have been implemented include UC, Yuque, Lieping Design House, Hupu, and several online education head customers. Some of these customers have flexibly used more than 10,000 cores of CPU computing resources during peak hours and processed them in parallel. Video reaches 1700+, while providing a very high cost performance.

Division of tasks, parallel acceleration

This idea of dividing and dividing tasks into function calculations is an interesting thing. Here is an example. For example, if you have a large 20G 1080P HD video that needs to be transcoded, even if you use a high-end machine, you need The time may still be measured in hours. If there is a problem and the transcoding is interrupted, you can only restart the transcoding process again. If you use the idea of divide and conquer + function calculation, the process of transcoding becomes sharding -> Parallel transcoding sharding -> merge sharding, so that you can solve the two pain points you mentioned above:

  • Fragmentation and synthetic sharding are memory-level copies, which require very little calculation. The transcoding that really consumes the calculation amount is split into many subtasks for parallel processing. In this model, the maximum time for sharding to transcode Basically equivalent to the transcoding time of the entire large video;
  • Even if the transcoding of a shard is abnormal in the middle, you only need to retry the transcoding of this shard, and you don't need to reinvent the whole big task.

By reasonably decomposing large tasks, using function calculations, and writing a little code, you can quickly complete a large data processing system with high elasticity and high availability, parallel acceleration, and pay-as-you-go.

Before introducing this solution, let's briefly introduce the Serverless workflow. The Serverless workflow can well organize functions and other cloud services and self-built services in an organized way.

1. Introduction to Serverless Workflow

Serverless Workflow(Serverless Workflow) is a fully managed cloud service used to coordinate the execution of multiple distributed tasks. In Serverless workflow, you can arrange distributed tasks in order, branch, parallel, etc. Serverless workflow will reliably coordinate task execution according to the set steps, track the state transition of each task, and when necessary Perform user-defined retry logic to ensure that the workflow is completed successfully. Serverless workflow simplifies the cumbersome tasks such as task coordination, state management, and error handling required to develop and run business processes, allowing you to focus on business logic development.

Next, a large video rapid transcoding case is used to explain the serverless work orchestration function, which realizes the decomposition of large computing tasks and parallel processing of subtasks, and finally achieves the goal of quickly completing a single large task.

2. Fast multi-target format transcoding for large videos


As shown in the figure above, suppose the user uploads a video in mov format to the OSS. The OSS trigger automatically triggers the function execution, and the function call FnF is executed. FnF simultaneously performs transcoding in one or more formats(by DST\ in template.yml _FORMATS parameter control), assuming that the configuration is to perform transcoding in both mp4 and flv formats.

  • A video file can be transcoded into various formats and other custom processing at the same time, such as adding watermark processing or updating information to the database after-process, etc.;
  • When multiple files are uploaded to OSS at the same time, the function calculation will automatically expand and contract to process multiple files in parallel, and at the same time, each file is converted to multiple formats in parallel;
  • Combined with NAS + video slicing, it can solve the transcoding of very large videos. For each video, slicing first, then transcoding and slicing in parallel, and finally synthesizing. By setting a reasonable slicing time, you can greatly accelerate the transcoding of larger videos speed;
  • fnf can track the execution of each step, and can customize the retry of each step to improve the robustness of the task system, such as:retry-example

In the specific case of task division and concurrency, parallel acceleration, the above sharing is CPU-intensive task decomposition, but IO-intensive task decomposition can also be performed, such as this requirement:a 20G large file in the OSS bucket of the Shanghai region, seconds The level is transferred back to the OSS Bucket in Hangzhou. Here you can also adopt the idea of divide and conquer. After receiving the dump task, the Master function allocates the range of the sharding of the large file to each Worker subfunction. The Worker subfunction transfers the part of its own segment in parallel, Master After all sub-workers have finished running, the function submits a merge sharding request to complete the entire dump task.


to sum up

This article discusses that the Serverless service platform can make your application scale horizontally quickly, and the parallel processing work is more effective, and gives specific practical cases. No matter in CPU-intensive or IO-intensive scenarios, function computing + Serverless can be perfectly solved Your following concerns:

  • No need to pay for idle resources
  • Don't worry about insufficient computing resource reservation
  • Tasks with large calculations need to be processed quickly
  • Better task flow tracking
  • Perfect monitoring and alarm, free operation and maintenance, business data visualization, etc.
  • ....

In this article, Serverless audio and video processing is just an example, which shows the ability and unique advantages of function computing in conjunction with Serverless workflow in offline computing scenarios. We can use divergent methods to expand the boundaries of Serverless' large-scale data processing practices, such as AI, genetic computing, and scientific simulation. I hope this article will attract you and start your wonderful journey of Serverless.

[Yunqi Online Classroom]Every day, product technology experts share!
Course address:

Join the community now, face-to-face with experts, and keep abreast of the latest developments in the course!
[Yunqi Online Classroom Community]

Original release time:2020-06-08
Author:West Ali cloud streaming technology experts
This article comes from:" dockone ", for related information, you can follow "dockone"