[Pytorch]docker共享内存问题
文章目录
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)
问题
ERROR: Unexpected bus error encountered in worker. This might be caused by insufficient shared memory (shm)
出现这个错误的情况是,在服务器上的docker中运行训练代码时,batch size设置得过大,shared memory不够(因为docker限制了shm).
根据PyTorch README:
Please note that PyTorch uses shared memory to share data between processes, so if torch multiprocessing is used (e.g. for multithreaded data loaders) the default shared memory segment size that container runs with is not enough, and you should increase shared memory size either with --ipc=host or --shm-size command line options to nvidia-docker run.
解决方案
1.这里说明PyTorch的IPC会利用共享内存,所以共享内存必须足够大,可以通过docker run --shm-size进行修改
2.通过设置 --ipc=host
3.将Dataloader的num_workers设置为0.但训练会变慢

评论
登录后参与技术讨论,分享你的经验与见解。
登录 / 注册暂无评论