Ant Group adds AI ecosystem, open-sources code model Ling-Coder-Lite
Ant Group open-sources Ling-Coder-Lite, a new breakthrough in AI code big model.
Core content:
1. Code big model redefines programming efficiency and possibilities
2. MoE architecture improves code big model performance and efficiency
3. Ling-Coder-Lite is open source and supports multi-language and multi-task scenarios
From LLM Challenge to MoE Breakthrough
With the rapid development of artificial intelligence technology, the Code Large Language Model (Code LLM) is becoming an important part of the developer tool chain. From code completion to error repair, from multi-language support to the automation of complex tasks, the Code Large Language Model is redefining the efficiency and possibilities of programming.
The key summary of this release is as follows:
Model and dataset open source : Two lightweight code models Ling-Coder-Lite and Ling-Coder-Lite-Base have been open sourced on Hugging Face and ModelScope. At the same time, Ant Group open sourced SyntheticQA for annealing training, SFT (Supervised Fine-tuning) and DPO (Direct Preference Optimization) for post-training, totaling about 30 million data (Figure 1(a)), to support further research and development in the community.
Disclosure of technical details : A technical report was released simultaneously with this open source release, disclosing more details about the method of building a high-quality training code dataset and the phased mixing and matching strategy of data during training, to help the industry jointly advance the research on large code models.
Balanced upgrade of efficiency and effect : Based on the Ling-MoE architecture, the total number of parameters of Ling-Coder-Lite is 16.8B, and the activation parameters during inference are only 2.75B, while taking into account higher efficiency and better effect.
Multi-language and multi-task support : Ling-Coder-Lite supports dozens of commonly used programming languages such as Python, Java, C++, JavaScript, and performs well in multi-language benchmarks such as MultiPL-E and MBXP. In addition to simple and multi-language code generation, it also supports multiple task scenarios such as competition and application advanced code generation, code understanding and input-output reasoning, data science and SQL data analysis, and code repair.
Model Effect
In 12 code benchmarks, Ling-Coder's performance is comparable to the best model of similar size (Qwen2.5-Coder-7B) (7 out of 12), ahead of OpenCoder-8B and DeepSeek-Coder-V2-lite, see Figure 1(b) for details; the reasoning efficiency is 1.5X~2X faster than Qwen2.5-Coder-7B (Figure 1(c)), which is particularly suitable for scenarios that require low-latency response, such as code completion in AI-IDE. In actual internal use, Ling-Coder-Lite saves half the deployment resources compared to previous models of similar size based on dense architecture under the same latency setting .
Explore the wonderful world of Markdown
Figure 1: Ling-Coder-Lite open source data, model code capabilities, and theoretical reasoning efficiency
Some high-quality data (about 30M samples) used by Ling-Coder-Lite during annealing and post-training have been open sourced; Performance of LLM codes of similar parameter size on 12 benchmarks; Comparison of various models’ performance (mean evaluation score) versus theoretical computational effort (TFLOPs required for a single inference with a context length of 4096).
In the future, we plan to continue to optimize Ling-Coder-Lite in multiple dimensions, including:
Continue to push the boundaries of performance and efficiency for large code models. By introducing reinforcement learning and execution feedback, the model's reasoning ability in handling actual software engineering tasks is improved . Continue to improve code data quality, especially the quality of synthetic data .
