DownloadX-DeepLearning (XDL for short) is a complete set of deep optimization solutions for high-dimensional sparse data scenarios (such as advertising/recommendation/search, etc.). XDL version 1.2 has been released recently. Performance optimization for large batch/low concurrency scenarios, 50-100% performance improvement in such scenarios. Storage and communication optimization, parameters are automatically allocated globally without manual intervention, and requests are merged to completely eliminate computing/storage/communication hotspots of ps. Complete streaming training features including feature admission, feature elimination, model incremental export, feature counting statistics, etc. Background: XDL1.0 focuses on throughput optimization and adopts the one request per thread processing model, which can significantly improve the limit throughput under ultra-high concurrency.
Features
- Performance optimization of large batch/single-sample massive feature scenarios
- Parameter assignment and communication optimization
- Unified storage and average distribution of parameters: ensure the average distribution of computing, communication, and memory on all servers
- Automatically analyze and merge undependent communication nodes in the calculation graph to reduce the number of communications and improve communication efficiency
- Simplify user usage costs, no longer need to provide the estimated value of the embedding parameter size, and no longer need to do regular rebalance
- In scenarios with massive features, performance and scalability are further improved
Project Samples
