How to run with limited resources ????

[old0110]

Hello, I am a graduate student currently studying. My teacher asked me to reproduce this paper, but now I only have 3 * A10. How can I do it? sos

[chhzh123]

Please refer to the Slapo-artifact repo for more details. Notice that since you only have limited GPU resources, you cannot exactly reproduce the results we presented in the paper, but you can still modify the number of GPUs following the guide in this README file to run the experiments.

[old0110]

wow，thank you,But I see that the link you provided is deployed through Docker. Can i set the environment through other methods

[chhzh123]

You don't need to use docker. You can always modify the .cfg file and pass it into the execution script to achieve the same purpose

[old0110]

thanks,I understand, but before that, I had some issues running this project and suspected that my environment was not properly configured. I have configured according to the following command. but I found a benchmark folder. Should I follow the instructions here to install the environment

[old0110]

Hello author, I have installed Docker on my server and pulled the image you provided. However, you mentioned that I am using an environment compatible with V100, while mine is A10. Will this cause some errors? In the Kick the Tires section, I encountered the following problem while using the Docker command

=========================================================================== test session starts ===========================================================================
platform linux -- Python 3.8.10, pytest-7.4.4, pluggy-1.3.0
rootdir: /home/deepspeed/slapo
plugins: cov-4.1.0
collected 57 items

tests/test_checkpoint.py . [ 1%]
tests/test_shard_sync_op.py . [ 3%]
tests/test_annotate.py Fatal Python error: Fatal Python error: Bus errorBus error

Thread 0xThread 0x00007ed78a55f7000000750712dd4700 (most recent call first):
(most recent call first):

Thread 0xThread 0x00007ed78ad6070000007507135d5700 (most recent call first):
(most recent call first):

Thread 0xThread 0x00007ed7d0e267000000750752057700 (most recent call first):
(most recent call first):

Thread 0xThread 0x00007ed86929d74000007507ee85d740 (most recent call first):
(most recent call first):
File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//ttoorrcchh//ddiissttrriibbuutteedd//ddiissttrriibbuutteedd__cc1100dd..ppyy"", line , line 15701570 in in bbrrooaaddccaasstt

File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//ttoorrcchh//ddiissttrriibbuutteedd//ddiissttrriibbuutteedd__cc1100dd..ppyy"", line , line 14511451 in in wwrraappppeerr

File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 191191 in in __ccoonnssoolliiddaattee__aanndd__bbrrooaaddccaasstt

File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 213213 in in __ccoonnssoolliiddaattee__aanndd__bbrrooaaddccaasstt

File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 220220 in in ccoonnssoolliiddaattee__mmooddeell

File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//ssllaappoo//bbuuiilldd..ppyy"", line , line 313313 in in bbuuiilldd

File File ""//hhoommee//ddeeeeppssppeeeedd//ssllaappoo//tteessttss//tteesstt__aannnnoottaattee..ppyy"", line , line 6868 in in tteesstt__aarrbbiittrraarryy__aannnnoottaattee

File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//__ppyytteesstt//ppyytthhoonn..ppyy"", line , line 194194 in in ppyytteesstt__ppyyffuunncc__ccaallll

File File ""//uussrr//llooccaall//lliibb//ppyytthhoonn33..88//ddiisstt--ppaacckkaaggeess//pplluuggggyy//__ccaalllleerrss..ppyy"", line , line 7777 in in m_umlutlitciaclall
l File
" File /"u/surs/rl/olcoacla/ll/ilbi/bp/yptyhtohno3n.38./8d/idsits-tp-apcakcakgaegse/sp/lpulguggyg/y/m_amnaangaegre.rp.yp", line y115" in , line _115h in o_ohkoeoxkeecx
e File c"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdti-spta-cpkaacgkeasg/epsl/upglguyg/g_yh/o_ohkoso.kpsy."p, line y493" in , line 493 in c_a_lcla_l_l
_ File _"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdti-spta-cpkaacgkeasg/e_sp/y_tpeystte/spty/tphyotnh.opny."p, line y1792" in , line r1792u in nrtuenstte
s File t"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdti-spta-cpkaacgkeasg/e_sp/y_tpeystte/srtu/nrnuenrn.epry."p, line y169" in , line p169y in tpeystte_srtu_nrtuenstte_scta_lcla
l File l"
/ File u"s/ru/slro/claolc/alli/bl/ipby/tphyotnh3o.n83/.d8i/sdt-packages/pluggy/calilsetr-sp.apcyk"a, line g77e in s/mpullutgigcya/l_lc
a File l"l/eurssr./plyo"c, line a77l in /_lmiubl/tpiyctahloln
3 File ."8//udsirs/tl-opcaaclk/algiebs//ppyltuhgogny3/._8m/adniasgter.py", line 115 in -_phaocokkaegxeesc/
p File l"u/gugsyr//_lmoacnaalg/elri.bp/yp"y, line t115h in o_nh3o.o8k/edxiesct

• File p"a/cuksarg/elso/cpallu/glgiyb//phyotohkosn.3p.y8"/, line d493i in s_t-cpaalclk_a_g
  e File s"//pulsurg/glyo/c_ahlo/olkisb./ppyy"t, line h493o in n_3_.c8a/ldli_s_t
• File p"a/cuksarg/elso/c_aply/tleisbt//pryutnhnoenr3..p8y/"d, line i262s in t<-lpaamcbkdaag>e
  s File /"_/puystre/slto/crauln/nleirb./ppyy"t, line h262o in n<3l.a8m/bddias>t
• File p"a/cuksarg/elso/c_aply/tleisbt//pryutnhnoenr3..p8y/"d, line i341s in tf-rpoamc_kcaaglels
  / File "p/yutsers/tl/orcuanln/elri.bp/yp"y, line t341h in ofnr3o.m_c8a/ldli
  s File t"-/puascrk/algoecsa/l/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t261- in pcaaclkla_greusn/t_epsytt_ehsoto/kr
  u File n"n/eurs.rp/yl"o, line c261a in lc/allilb_/rpuynttheosnt3_.h8o/odki
  s File t"-/puascrk/algoecsa/l_/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t222- in pcaaclkla_gaensd/rpeyptoerstt
  / File r"u/nunserr/.lpoyc"a, line l222/ in lciabl/lp_yatnhdo_nr3e.p8o/rdti
  s File t"-/puascrk/algoecsa/l/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t133- in praucnktaegsetsp/r_optyotceoslt
  / File r"u/nunserr/.lpoyc"a, line l133/ in lriubn/tpeystthporno3t.o8c/odli
  s File t"-/puascrk/algoecsa/l/plyitbe/spty/trhuonnn3e.r8./pdyi"s, line t114- in ppayctkeasgte_sr/u_nptyetsets_tp/rroutnonceorl.
  p File y""/, line u114s in rp/yltoecsatl_/rluinbt/epsytt_hporno3t.o8c/odli
  s File t"-/puascrk/algoecsa/lp/lluigbg/yp/y_tchaolnl3e.r8s/.dpiys"t, line -77p in a_cmkualgteisc/aplllu
  g File g"y//u_scra/llloecrasl./plyi"b, line /77p in y_tmhuolnt3i.c8a/ldli
  s File t"-/puascrk/algoecsa/lp/lluigbg/yp/y_tmhaonna3g.e8r/.dpiys"t, line -115p in a_chkoaogkeesx/epcl
  u File g"g/yu/s_rm/alnoacgaelr/.lpiyb"/, line p115y in t_hhoono3k.e8x/edci
  s File t"-/puascrk/algoecsa/lp/lluigbg/yp/y_thhooonk3s..8p/y", line 493 in _call__d
  i File s"t/-upsarc/klaogceasl//plliubg/gpyy/t_hhoono3k.s8./pdyi"s, line t493- in p_a_cckaalgle_s/
  _ File p"y/tuessrt//lmoacianl./plyi"b, line /350p in yptyhtoens3t._8r/udnitsets-tplaocokpa
  ges/ File "p/yutsers/tl/omcaailn/.lpiyb"/, line p350y in tphyotne3s.t8/rduinstte-sptalcokoapg
  e File s"//pulsurg/glyo/c_acla/llliebr/sp.yptyh"o, line n773 in ._8m/udlitsitc-aplalc
  k File a"g/euss/rp/lluogcgayl//_lciabl/lpeyrtsh.opny3"., line 877/ in d_imsutl-tpiaccaklalg
  e File s"//pulsurg/glyo/c_amla/nlaigbe/rp.yptyh"o, line n1153 in .8h/odoiksetx-epca
  c File k"a/guessr//plloucgagly//l_imba/npaygtehro.np3y."8, line /115d in i_shto-opkaecxkeacg
  e File s"//pulsurg/glyo/c_ahlo/olkisb./ppyy"t, line h493o in n_3.c8a/ldli_s_t
• File p"a/cuksarg/elso/cpallu/glgiyb//phyotohkosn.3p.y8"/, line d493i in s_t-cpaalclk_a_g
  e File s"//upsytre/slto/cmaali/nl.ipby/"p, line y325t in h_omna3i.n8
  / File d"i/suts-rp/alcokcaagle/sl/i_bp/yptyetshto/nm3a.i8n/.dpiys"t, line -325p in a_cmkaaigne
  s/pytest File /"m/auisnr./plyo"c, line a271l in /wlriabp/psyetshsoino3n.
  8 File /"d/iusstr-/plaocckaalg/elsi/b/ppyyttehsotn/3m.a8i/nd.ipsyt"-, line p271a in cwkraagpe_ss/e_spsyitoens
  t File /"m/auisnr./plyo"c, line a318l in /plyitbe/spty_tchmodnl3i.n8e/dmiasitn-
  pa File c"k/augsers//l_opcyatl/lib/pytehsotn/3m.a8i/nd.ipsyt"-, line p318a in cpkyatgeesst/pclmudglgiyn/e__cmaalilne
  rs.p File y""/, line u77s in r/mluolctailc/allilb
  / File p"y/tuhsorn/3l.o8c/adli/slti-bp/apcyktahgoens3/.p8l/udgigsyt/-pcaaclklaegress./ppyl"u, line g77g in y/m_umlatniacgaelrl.
  p File y""/, line u115s in r/hlooockaelx/elci
  b File /"p/yutshro/nl3o.c8a/ld/ilsitb-/ppayctkhaogne3s./8p/lduigsgty-/p_amcaknaaggeesr/.pplyu"g, line g115y in /hhooookkesx.epcy
  " File , line "493/ in u_s_rc/allolc_a_l/lib/p
  y File t"h/ouns3r./8l/odciaslt/-lpiabc/kpaygtehso/np3l.u8g/gdyi/s_th-opoakcsk.apgye"s, line /493 in p_y_tceasltl/c_o
  n File f"i/gu/s_r/ilnoicta_l/.lpiyb"/, line p169y in tmhaoinn3
  . File 8"//duissrt/-lpoaccakla/gleisb//ppyyttheosnt3/.c8o/ndfiisgt/-p_aicnkiatg_e_s./p_yp"y, line t169e in smta/icno
  n File f"i/gu/s_r/ilnoicta_l/.lpiyb"/, line p192y in tchoonns3o.l8e/dmiasitn-
  p File a"c/kuasgre/sl/o_cpaylt/elsitb//cpoyntfhiogn/3.8i/ndiits_t-.ppayc"ka, line g192e in sc/opnystoelset/m_a_imna
  i File n"/u.spry/"l, line o5c in a<lm/oldiubl/ep>y
  th File o"n/3u.s8r//dliisbt/-ppyatchkoang3e.s8//pryutnepsyt./p_y"m, line a87i in n__r_u.np_yc"o, line d5e in
  < File m"o/duuslre/>l
  ib/py File t"h/ouns3r./8l/irbu/nppyyt.hpoyn"3, line .1948 in /rruunnp_ym.opdyu"l, line e87 in a_sr_umna_icno
  de
  File "/usr/lib/python3.8/runpy.py", line 194 in _run_module_as_main
  ERROR:torch.distributed.elastic.multiprocessing.api:failed (exitcode: -7) local_rank: 0 (pid: 99) of binary: /usr/bin/python3
  Traceback (most recent call last):
  File "/usr/local/bin/torchrun", line 8, in
  sys.exit(main())
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/elastic/multiprocessing/errors/init.py", line 346, in wrapper
  return f(*args, **kwargs)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 794, in main
  run(args)
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/run.py", line 785, in run
  elastic_launch(
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 134, in call
  return launch_agent(self._config, self._entrypoint, list(args))
  File "/usr/local/lib/python3.8/dist-packages/torch/distributed/launcher/api.py", line 250, in launch_agent
  raise ChildFailedError(
  torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
  ===================================================
  pytest FAILED

---

Failures:
[1]:
time : 2025-04-16_00:06:43
host : 494f2fcc8a8d
rank : 1 (local_rank: 1)
exitcode : -7 (pid: 100)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 100

Root Cause (first observed failure):
[0]:
time : 2025-04-16_00:06:43
host : 494f2fcc8a8d
rank : 0 (local_rank: 0)
exitcode : -7 (pid: 99)
error_file: <N/A>
traceback : Signal 7 (SIGBUS) received by PID 99

[chhzh123]

It does not seem to be an error on the Slapo side. Could you first check if torch.distributed is correctly configured? You can run basic tests from PyTorch. Our prebuilt docker probably has some device-dependent libraries installed. You can try building the docker image from source and see if it resolves the issue.

[old0110]

I have successfully run this section, but I encountered some issues in the experimental section of Figure 7. The content of my single_node_asplos24.cfg is as follows:
megatron bert-xlarge 1 256 0 2 0
slapo-megatron bert-xlarge 1 256 0 12 0
deepspeed bert-xlarge 1 256 0 10 0
slapo-deepspeed bert-xlarge 1 256 0 12 0
megatron bert-xlarge 2 512 0 4 0
slapo-megatron bert-xlarge 2 512 0 20 0
deepspeed bert-xlarge 2 512 0 12 0
slapo-deepspeed bert-xlarge 2 512 0 20 0

megatron bert-xlarge 3 512 0 30 0

slapo-megatron bert-xlarge 3 512 0 84 0

deepspeed bert-xlarge 3 512 0 24 0
slapo-deepspeed bert-xlarge 3 512 0 24 0
megatron roberta-xlarge 1 512 0 13 0
slapo-megatron roberta-xlarge 1 512 0 13 0
deepspeed roberta-xlarge 1 512 0 12 0
slapo-deepspeed roberta-xlarge 1 512 0 18 0
megatron roberta-xlarge 2 512 0 10 0
slapo-megatron roberta-xlarge 2 512 0 20 0
deepspeed roberta-xlarge 2 512 0 12 0
slapo-deepspeed roberta-xlarge 2 512 0 22 0

megatron roberta-xlarge 3 512 0 39 0

slapo-megatron roberta-xlarge 3 512 0 39 0

deepspeed roberta-xlarge 3 512 0 22 0
slapo-deepspeed roberta-xlarge 3 512 0 24 0
megatron EleutherAI/gpt-neo-2.7B 1 512 0 3 1
slapo-megatron EleutherAI/gpt-neo-2.7B 1 512 0 3.5 1
deepspeed EleutherAI/gpt-neo-2.7B 1 512 0 3 1
slapo-deepspeed EleutherAI/gpt-neo-2.7B 1 512 0 3 0.6
megatron EleutherAI/gpt-neo-2.7B 2 512 0 6 1
slapo-megatron EleutherAI/gpt-neo-2.7B 2 512 0 7 1
deepspeed EleutherAI/gpt-neo-2.7B 2 512 0 6 1
slapo-deepspeed EleutherAI/gpt-neo-2.7B 2 512 0 6 0.6

megatron EleutherAI/gpt-neo-2.7B 3 512 0 9 1

slapo-megatron EleutherAI/gpt-neo-2.7B 3 512 0 10.5 1

deepspeed EleutherAI/gpt-neo-2.7B 3 512 0 6 1
slapo-deepspeed EleutherAI/gpt-neo-2.7B 3 512 0 6 0.6
megatron t5-3b 1 1024 512 2 full
slapo-megatron t5-3b 1 1024 512 2 1
deepspeed t5-3b 1 1024 512 1 1
slapo-deepspeed t5-3b 1 1024 512 1 1
megatron t5-3b 2 1024 512 2 full
slapo-megatron t5-3b 2 1024 512 2 1
deepspeed t5-3b 2 1024 512 2 1
slapo-deepspeed t5-3b 2 1024 512 2 1

megatron t5-3b 3 1024 512 6 full

slapo-megatron t5-3b 3 1024 512 6 1

deepspeed t5-3b 3 1024 512 3 1
slapo-deepspeed t5-3b 3 1024 512 3 1
megatron wideresnet-2.4B 1 1 0 8 0
slapo-megatron wideresnet-2.4B 1 1 0 8 0
deepspeed wideresnet-2.4B 1 1 0 10 1
slapo-deepspeed wideresnet-2.4B 1 1 0 12 0.34
megatron wideresnet-2.4B 2 1 0 10 0
slapo-megatron wideresnet-2.4B 2 1 0 10 0
deepspeed wideresnet-2.4B 2 1 0 16 1
slapo-deepspeed wideresnet-2.4B 2 1 0 22 0.34

megatron wideresnet-2.4B 3 1 0 24 0

slapo-megatron wideresnet-2.4B 3 1 0 24 0

deepspeed wideresnet-2.4B 3 1 0 16 1
slapo-deepspeed wideresnet-2.4B 3 1 0 24 0.34

However, every time I execute the megatron roberta-xlarge section, an error message will appear:
Traceback (most recent call last):
File "bench_single_node.py", line 510, in
main()
File "bench_single_node.py", line 483, in main
runner(
File "/home/deepspeed/slapo/benchmark/megatron_utils.py", line 243, in run_megatron
script_file, data_args = gen(exp)
File "/home/deepspeed/slapo/benchmark/megatron_utils.py", line 83, in megatron_roberta_cmd
raise NotImplementedError
and the subsequent code will not execute properly：
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
BERT-1B {'megatron': [11.25, 0.0, 0.0], 'slapo-megatron': [26.5, 0.0, 0.0], 'deepspeed': [17.27, 0.0, 0.0], 'slapo-deepspeed': [22.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
RoBERTa-1.3B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
GPT-2.9B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
OPT-2.7B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
T5-2.9B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}
slapo-megatron na
slapo-megatron na
slapo-megatron na
slapo-deepspeed na
slapo-deepspeed na
slapo-deepspeed na
WideResNet-2.4B {'megatron': [0.0, 0.0, 0.0], 'slapo-megatron': [0.0, 0.0, 0.0], 'deepspeed': [0.0, 0.0, 0.0], 'slapo-deepspeed': [0.0, 0.0, 0.0]}

[old0110]

When running the script that generates Figure 7, the following issue was also encountered

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/dist-packages/filelock/_api.py", line 255, in acquire
self._acquire()
File "/usr/local/lib/python3.8/dist-packages/filelock/_unix.py", line 39, in _acquire
fd = os.open(self.lock_file, open_flags, self._context.mode)
PermissionError: [Errno 13] Permission denied: '/home/deepspeed/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock'