Spark Thinking: A technical solution to implement lightweight algorithm services based on offline label knowledge base and real-time data warehouse

background

• The development cycle is short, and model development and engineering deployment can be completed within 4 working weeks;

  
  

• Iteration and agility: Model adjustment is decoupled from the online environment, and iteration and release can be completed within a few working days;

  
  

• It uses a mature technology stack with a low threshold to support analyst development, providing branch algorithm services for large enterprises' short- and medium-term operational strategies, or providing basic algorithm services for small and medium-sized enterprises;

  
  

• Able to implement a variety of algorithm tasks (such as classification, clustering) and have good model performance;

  
  

• The entire system can realize automatic “periodic retraining” to keep the model adaptable to dynamic changes in data;

  
  

• It has strong reusability and good scalability, and supports the implementation of multiple algorithm services based on the same data underlying layer and technical foundation.

  
  

Technical Framework

• It bypasses the environmental requirements for online model deployment and avoids the computing pressure of real-time reasoning. Doris's efficient column storage and compression technology stores a large amount of hash table data at a low cost, significantly reducing the consumption of online service resources and ensuring the low threshold, low load and low cost advantages of the solution.

  
  

• The full-link technology risk is controllable, and the Hive SQL + Python technology stack is mature and stable. Flink + Doris is also the mainstream implementation method of real-time data warehouses on the market recently. Although there is generally a delay of about 3-5 minutes (pseudo real-time) to control the cost of Doris servers, it is sufficient for most business scenarios of Spark.

  
  

• Offline and real-time decoupling ensures the flexibility and stability of the system. Unlike the traditional complex MLOps process, the offline and real-time parts of this solution have clear functions and are independent of each other. It can isolate the faults of the two and enhance the robustness of the system. In addition, since the model optimization and adjustment are completed in the offline part, it does not affect the continuity and stability of the real-time service, thus improving the agility of iteration.

  
  

• At the same time, relying on the periodic scheduling function of Athena Data Factory, it can realize automatic "retraining" of models in daily/weekly/monthly dimensions, and achieve adaptation to business changes and data drift at a lower operation and maintenance cost, ensuring the continuous stability of model performance;

  
  

• It has good reusability and scalability. Based on the same technical base, it can support the parallel construction of multiple algorithm tasks (such as classification, clustering, etc.), which is convenient for supporting business needs in different scenarios. As the business develops, features can be added or removed or models can be adjusted without adjusting the overall framework, and it has good scalability. The technical framework can be easily integrated with other systems and platforms to meet the requirements of enterprise-level applications for system compatibility.

  
  

Key technology nodes

• The entire automatic label screening is divided into two steps. First, Hive SQL is used to count the sample size of the labels under each feature of the initial training set in the last n months. Generally speaking, for ensemble tree models (such as random forest, XGBoost, and LightGBM), the minimum sample size required for a single label under a feature should be >= 30. Here, we can also combine specific business scenarios. For labels with a small absolute value of sample size (>=30) or a low proportion, we believe that their probability of appearing in the future is low, and there is no need to waste hash table space resources for them, so they are directly eliminated.

  
  

• Then, the initial training set and the label sample size screening table (the above screening results) are associated. All the labels filtered out in the screening results are merged into the "other" label under the feature to generate a transition training set. After one-hot encoding, the model is trained, and the SHAP value or tree model feature contribution is printed. The valuable labels are further screened by the absolute value of the contribution or its ratio to the total contribution. The final label screening table under each feature is generated based on the sample size screening.

  
  

• The final label filtering table is associated with the transition training set. Similarly, all filtered labels are merged into the "other" label under the corresponding feature to generate the final training set. At the same time, the label filtering results are synchronized to Doris, and the labels are merged in the same way to achieve the same label "pruning" processing logic for offline and real-time modules.

• Through the above two steps, each time the model is automatically " retrained " , the label screening work is completed at the same time, and the label merging logic is synchronized to the real-time environment.

• Take a two-layer "printed" hash table as an example: the first layer uses 5 features (x1-x5) for training (Python task) and builds the corresponding hash table. In the same Python task, the model training, training set prediction (y1'), hash table prediction (y1) and result writing process are completed at the same time (Note: here y1' and y1 are both label values ​​after discretization);

  
  

• In the second layer, a new training set is constructed with the prediction results of the training set of the first layer (y1') and another 5 features (x6-x10). At the same time, a new hash table is constructed based on the prediction results of the hash table of the previous layer (y1') and these 5 features. After training the second-layer model, the hash table is predicted and written into Hive SQL; finally, the offline label knowledge base 1 and 2 (hash table prediction results) are synchronized to the Doris table to complete the engineering deployment of the entire model.

  
  

• Theoretically, the scheme of "printing" hash tables layer by layer can be extended indefinitely. However, considering that the engineering chain is too long and the complexity is too high, which affects the usability of this scheme, based on Spark's experience, it is recommended to control it to about 5-6 layers, covering about 20-30 features, which can basically guarantee a good model effect.

  
  

• Although the reading and writing difficulties and cost challenges caused by the large hash table have been solved by splitting the table, it comes at the cost of a certain loss of model effect: for the ensemble tree model, one of its major advantages is that it can learn the nonlinear relationship between features, while the layered printing method makes the nonlinear relationship between layer features insufficiently learned or impossible to learn, and fails to give full play to the advantages of the tree model, resulting in a certain loss of model effect; however, the more thorough our label screening in the early stage, the more features each layer can accommodate. The SHAP value is used to analyze the interaction between features, and features with obvious interactions are placed in the same layer as much as possible to ensure that the nonlinear relationship can be fully learned as much as possible to ensure the model effect.

  
  

• The historical "dictionary" has the following obvious advantages over the hash "dictionary":

  
  

  The table is smaller, and there is often no need to use the layer-by-layer "printing" method. The engineering chain is shorter, the deployment plan is simpler, and the advantages of the "lightweight" technical solution can be better utilized;

  
  

  Complete the model training of all features and "offline pre-calculation" of the knowledge base at one time, give full play to the advantages of tree models in learning non-linear relationships, and avoid information loss caused by layer-by-layer learning.

  
  

• But at the same time, the limitations of the historical "dictionary" are also very obvious: there may be a considerable number of users who cannot be effectively predicted; for a label combination, if all or most of its labels have appeared in the training set (labels that have not appeared have been encapsulated as "others"), but this combination has not appeared in the historical "dictionary", then it cannot be found in the historical "dictionary", but its corresponding probability value can be found in the hash "dictionary", and a more accurate prediction result can be obtained; in addition, the feature carrying capacity of the historical "dictionary" also has a certain upper limit ceiling. If there are more features, the success rate of its historical "dictionary" query will gradually decrease. According to Spark's experience, for business scenarios where the historical "dictionary" can be used, the upper limit of its features is about 40. If it exceeds this range, its query success rate will be lower than the acceptable threshold of the business (<70%);

• Therefore, we need to choose an appropriate "dictionary" deployment solution based on the specific business scenario and feature distribution. For business scenarios where most of the feature combinations in the future are expected to have appeared in the past, we can choose the "historical" dictionary solution; otherwise, we need to use the "hash" dictionary. Note: When using the "historical" dictionary, be sure to backtest the historical query success rate first, and closely monitor the indicator after going online. When the query success rate is significantly lower than a certain business acceptable threshold, it is necessary to promptly iterate to the "hash" dictionary solution.

Implementation Case

Summary and Outlook