期刊文献+
任意字段
题名或关键词
题名
关键词
摘要
作者
第一作者
机构
刊名
分类号
参考文献
作者简介
基金资助
栏目信息
共找到1篇文章
< 1 >
每页显示 20 50 100
已选0条
引用分析
参考文献
引证文献
 统计分析
检索结果
已选文献
显示方式：
Automatic parallelism strategy generation with minimal memory redundancy 认领 引用
1
作者 Yanqi SHI Peng LIANG +2 位作者 Hao ZHENG Linbo QIAO Dongsheng LI 《Frontiers of Information Technology & Electronic Engineering》 SCIE EI CSCD 2025年第1期109-118,共10页
Large-scale deep learning models are trained distributedly due to memory and computing resource limitations.Few existing strategy generation approaches take optimal memory minimization as the objective.To fill in this... Large-scale deep learning models are trained distributedly due to memory and computing resource limitations.Few existing strategy generation approaches take optimal memory minimization as the objective.To fill in this gap,we propose a novel algorithm that generates optimal parallelism strategies with the constraint of minimal memory redundancy.We propose a novel redundant memory cost model to calculate the memory overhead of each operator in a given parallel strategy.To generate the optimal parallelism strategy,we formulate the parallelism strategy search problem into an integer linear programming problem and use an efficient solver to find minimal-memory intra-operator parallelism strategies.Furthermore,the proposed algorithm has been extended and implemented in a multi-dimensional parallel training framework and is characterized by high throughput and minimal memory redundancy.Experimental results demonstrate that our approach achieves memory savings of up to 67%compared to the latest Megatron-LM strategies;in contrast,the gap between the throughput of our approach and its counterparts is not large. 展开更多
关键词 Deep learning Automatic parallelism Minimal memory redundancy
暂未订购 下载PDF
已选0条
引用分析
参考文献
引证文献
 统计分析
检索结果
已选文献
上一页 1 下一页 到第 页
在线咨询 使用帮助 返回顶部 意见反馈