The practice and exploration of Tuya smart dubbo-go billion-level traffic

Posted Jun 16, 202010 min read

The practice and exploration of Tuya smart dubbo-go billion-level traffic

Dubbo is a high-performance lightweight RPC framework developed based on Java. Dubbo provides rich service governance functions and excellent scalability. And dubbo-go provides unified service capabilities and standards between java and golang, which is the main problem that Tuya Smart currently needs to solve. This article is divided into two parts:practice and fast access, sharing the actual combat experience of dubbo-go in Tuya Smart, which is intended to help users quickly access dubbo-go RPC Framework, I hope you can take less detours.

In addition, the test code in this article is based on dubbo-go version v1.4.0 .

dubbo-go gateway practice

image.png

The use of dubbo-go in the graffiti intelligence is as shown above. Next, I will introduce the details of the landing for everyone. I hope that the experience summarized in the production environment can help everyone.

BACKGROUND

In Tuya Smart, dubbo-go has become the go-to RPC framework for the golang service to connect with the original dubbo cluster. Among them, the representative open-gateway gateway system(hereinafter referred to as the gateway, see the open source version [ https://github.com/dubbogo/dubbo-go-proxy] ( https://github.com/dubbogo/dubbo- go-proxy)). The gateway dynamically loads the internal dubbo interface information and exposes it in the form of HTTP API. The gateway is intended to solve the following pain points of the previous generation gateway.

  • Configure the open rules of the dubbo interface through the page, the steps are cumbersome, and the permissions are difficult to control.
  • Interface is not RESTful style, not friendly to external developers.
  • Dependency is heavy, and the upgrade risk is high.
  • Concurrency performance issues.

Architecture Design

In response to the above pain points, we immediately started to design a new gateway architecture. The first is language selection. Golang's coroutine call model makes golang very suitable for building IO-intensive applications, and application deployment is also simpler than java. After investigation, we decided to use golang as the coding language of the proxy, and use dubbo-go to connect the dubbo provider cluster. The business application on the provider side configures the API configuration information in the form of annotations by using a java plug-in. The plug-in will update the configuration information and the metadata of the dubbo interface to the metadata registration center(redis in the figure below). In this way, the configuration is transferred from the management background page to the program code. Developers can easily see the external API description of the dubbo interface when coding, without having to configure the API usage from another management background.

Practice

As can be seen from the above figure, the gateway can dynamically load the dubbo interface information, and the call to the dubbo interface is based on the generalized call of dubbo. The generalized call eliminates the need for the client to build the interface code of the provider. In dubbo-go, it does not need to call the config.SetConsumerService and hessian.RegisterPOJO methods, but completes the pure parameters of the request model, which makes the client dynamically add and modify the interface. may. There is a generalization call in apache/dubbo-sample/golang/generic/go-client Demo code.

func test() {
    var appName = "UserProviderGer"
    var referenceConfig = config.ReferenceConfig{
        InterfaceName:"com.ikurento.user.UserProvider",
        Cluster:"failover",
        Registry:"hangzhouzk",
        Protocol:dubbo.DUBBO,
        Generic:true,
    }
    referenceConfig.GenericLoad(appName) //appName is the unique identification of RPCService

    time.Sleep(3 * time.Second)

    resp, err := referenceConfig.GetRPCService().(*config.GenericService).
        Invoke([]interface{}{"GetUser", []string{"java.lang.String"}, []interface{}{"A003"}})
    if err != nil {
        panic(err)
    }
}

The implementation of the generalization call is actually quite simple. Its function is in the Filter layer of dubbo. Generic Filter has been added to the dubbo Filter chain as the default enabled Filter. The core logic is as follows:

func(ef *GenericFilter) Invoke(ctx context.Context, invoker protocol.Invoker, invocation protocol.Invocation) protocol.Result {
    if invocation.MethodName() == constant.GENERIC && len(invocation.Arguments()) == 3 {
        oldArguments := invocation.Arguments()

        if oldParams, ok := oldArguments[2].([]interface{}); ok {
            newParams := make([]hessian.Object, 0, len(oldParams))
            for i := range oldParams {
                newParams = append(newParams, hessian.Object(struct2MapAll(oldParams[i])))
            }
            newArguments := []interface{}{
                oldArguments[0],
                oldArguments[1],
                newParams,
            }
            newInvocation := invocation2.NewRPCInvocation(invocation.MethodName(), newArguments, invocation.Attachments())
            newInvocation.SetReply(invocation.Reply())
            return invoker.Invoke(ctx, newInvocation)
        }
    }
    return invoker.Invoke(ctx, invocation)
}

The Generic Filter converts the structure parameters requested by the user into a uniform format map(struct2MapAll in the code), and turns the forward and reverse serialization operations of the class(struct in golang) into the forward and reverse serialization operations of the map. This eliminates the need for POJO descriptions to be hardcoded into the hessain library.

As you can see from the above code, there are four things that need to be dynamically built for generalized calls. The InterfaceName required in ReferenceConfig, the method in Parameter, ParameterTypes, and the actual input parameter requestParams.

So how are these parameters obtained from HTTP API matching?

Here we will use the above-mentioned provider's plugin for collecting metadata. After the plugin is introduced, the application will scan the dubbo interface that needs to be exposed when it starts, and associate the dubbo metadata with the HTTP API. The method of using the plug-in is roughly as follows. Here are a few simple configurations as examples, and there will be more annotations during actual production.

image-20200609112413216.png

The resulting dubbo metadata is as follows:

{
    "key":"POST:/hello/{uid}/add",
    "interfaceName":"com.tuya.hello.service.template.IUserServer",
    "methodName":"addUser",
    "parameterTypes":["com.tuya.gateway.Context", "java.lang.String", "com.tuya.hello.User"],
    "parameterNames":["context", "uid", "userInfo"],
    "updateTimestamp":"1234567890",
    "permissionDO":{},
    "voMap":{
        "userInfo":{
            "name":"java.lang.String",
            "sex":"java.lang.String",
            "age":"java.lang.Integer"
        }
    },
    "parameterNameHumpToLine":true,
    "resultFiledHumpToLine":false,
    "protocolName":"dubbo",
  .......
}

Gateway subscribes to the above information from the metadata configuration center and can match an API request to a dubbo interface. Then grab the parameters from the API request as input parameters. This function completes the flow closed loop.

Above, everyone should have a clear understanding of the gateway project topology. I then share the problems and tuning experience that the project encountered while using dubbo-go. At the beginning of 19, the dubbo-go project at that time was only in the early stages of construction, and there was no user experience. I also participated in community development while coding the company's internal gateway project. After solving a bunch of problems with hessain serialization and zookeeper registration center, the project finally ran through the closed loop. However, as a core application, running a closed loop from the production environment still has a long way to go, especially when using a new framework whose stability was to be tested at the time. The entire test plus function completion took a quarter of the time until the project stabilized and the pressure test effect was also good. Single gateway machine(2C 8G) full link simulated real environment pressure test reaches 2000 QPS. Due to the introduction of heavier business logic(a single request calls 3 dubbo interfaces on average), this pressure test result is in line with or even exceeds expectations.

Summarized some experience of tuning dubbo-go parameter configuration, mainly some network related configuration. When you run the demo, you should see a bunch of configurations at the end of the configuration file, but if you are not familiar with the underlying network model of dubbo-go, it is difficult to understand the meaning of these configurations. Currently, the dubbo-go network layer uses getty as the underlying framework to achieve read-write separation and coroutine pool management. getty exposes the concept of session to the outside world. Session provides a series of network layer method injection implementations, because this article is not a source code analysis document, but I won't discuss it here. Readers can simply assume that dubbo-go maintains a getty session pool, and the session maintains a TCP connection pool. For each connection, getty will have a read coroutine and a write coroutine companion, to achieve read and write separation. Here I try to use common comments to help you sort out the meaning of several configurations that have a great impact on performance:

protocol_conf:
  # This is a protocol-independent configuration. Under the dubbo protocol, most of the configuration is the configuration related to getty session.
  dubbo:
      # A session will always guarantee the number of connection_number tcp connections, the default is 16,
    # But here it is recommended that you configure a relatively small value, the general system does not require so many connections.
    # Every reconnect_interval time, check the number of connections, if less than connection_number,
    # Just establish a connection. Fill in 0 or not fill the default value of 300ms
    reconnect_interval:0
    connection_number:2
    # Client sends heartbeat interval
    heartbeat_period:"30s"
    # OnCron session timeout time, close session without returning if session_timeout
    session_timeout:"30s"
    # Each client of dubbo interface will maintain a session pool with a maximum size of pool_size.
    # Each request selects one from the session pool. So the actual tcp number is the number of sessions * connection_number,
    # And pool_size is the maximum number of sessions. The test concludes that the general program 4 tcp connection is enough.
    pool_size:4
    # session keepalive timeout, that is, if the session is not used beyond the session_timeout time, the session will be closed
    pool_ttl:600
    # The size of the coroutine pool that handles the return value
    gr_pool_size:1200
    # Read data and the buffer queue length in the coroutine pool, which is currently obsolete. Do not use buffer queue
    queue_len:64
    queue_number:60
    getty_session_param:
      compress_encoding:false
      tcp_no_delay:true
      tcp_keep_alive:true
      keep_alive_period:"120s"
      tcp_r_buf_size:262144
      tcp_w_buf_size:65536
      pkg_wq_size:512
      tcp_read_timeout:"1s" # timeout time for each packet read
      tcp_write_timeout:"5s" # Timeout time for each packet write
      wait_timeout:"1s"
      max_msg_len:102400 # Maximum data transmission length
      session_name:"client"

dubbo-go fast access

The previous article has shown the practical results of dubbo-go in graffiti intelligence, and then introduces the way to quickly access dubbo-go.

The first step:hello world

Dubbo-go usage examples are currently consistent with dubbo and are placed in the apache/dubbo-samples project. Under the dubbo-sample/golang directory, users can select the feature directory of interest to quickly test the code effect.

tree dubbo-samples/golang -L 1
dubbo-samples/golang
    README.md
    async
    ci.sh
    configcenter
    direct
    filter
    general
    generic
    go.mod
    go.sum
    helloworld
    multi_registry
    registry

Let's take hello world as an example and follow the steps in dubbo-samples/golang/README.md to start the server and client respectively. You can try golang calling java, java calling golang, golang calling golang, java calling java. Dubbo-go supports interworking with dubbo on the agreement.

Let's take the startup go-server as an example. The registration center uses zookeeper by default. First confirm whether the local zookeeper is operating normally. Then execute the following command, and then you can see the log of your service started normally.

export ARCH=mac
export ENV=dev
cd dubbo-samples/golang/helloworld/dubbo/go-server
sh ./assembly/$ARCH/$ENV.sh
cd ./target/darwin/user_info_server-2.6.0-20200608-1056-dev/
sh ./bin/load.sh start

Step 2:Use dubbo-go in the project

Above, we ran the use case through the test code and startup script maintained by the community. Next, we need to embed the dubbo-go framework in our own code. Many friends often encounter problems at this step. Here are some common problems that I have sorted out. I hope to help you.

1. Environment variables

Currently dubbo-go has 3 environment variables that need to be configured.

  • CONF_CONSUMER_FILE_PATH:Path of the configuration file on the consumer side, required when using consumer.
  • CONF_PROVIDER_FILE_PATH:Path of the configuration file on the provider side, required when using the provider.
  • APP_LOG_CONF_FILE:Log log file path, required.
  • CONF_ROUTER_FILE_PATH:File Router rule configuration file path, required when using File Router.
2. Code notes
  • Inject Service:Check if the following code is executed

    Client

    func init() {

      config.SetConsumerService(userProvider)

    }

    Server

    func init() {

      config.SetProviderService(new(UserProvider))

    }

  • Inject serialization description:Check if the following code is executed

      hessian.RegisterJavaEnum(Gender(MAN))
      hessian.RegisterJavaEnum(Gender(WOMAN))
      hessian.RegisterPOJO(&User{})
3. Understanding the configuration file correctly
  • The key under `references/services, such as the "UserProvider" in the following example, needs to be consistent with the return value of the Service Reference(). This is the key that identifies the interface.

    references:

    "UserProvider":
      registry:"hangzhouzk"
      protocol:"dubbo"
      interface:"com.ikurento.user.UserProvider"
      cluster:"failover"
      methods:
      -name:"GetUser"
        retries:3
  • `Registration center If there is only one registration center cluster, you only need to configure one. Multiple IPs are separated by commas, as follows:

    registries:

    "hangzhouzk":
      protocol:"zookeeper"
      timeout:"3s"
      address:"172.16.120.181:2181,172.16.120.182:2181"
      username:""
      password:""
4. Problems with java and go
  • Case of interaction between go and java:In order to adapt to the hump format of java, golang will automatically change the first letter of method and attribute to lower case when calling the java service. Many students deliberately wrote java code to adapt to the parameter definition of golang, capitalized the first letter, and finally could not serialize the match.

Step 3:Expand functionality

Both dubbo-go and dubbo provide a very rich expansion mechanism. You can implement custom modules instead of dubbo-go default modules, or add some new features. For example, to achieve Cluster, Filter, Router, etc. to adapt to business needs. These injection methods are exposed in dubbo-go/common/extension, allowing users to call and configure.

_ author:_
Pan Tianying, Github ID @pantianying, open source enthusiast, working at Tuya Smart.

Welcome to the dubbo-go community

If you have any questions about dubbo-go, you can add our Dingbo Group 23331795 for inquiry and discussion, we will give feedback as soon as possible.

latest events

Dubbo-go ASoC related topics , participation details please click