Parallel input / output

[bendudson]

To scale to more processors and make handling large numbers of output files less painful, we need a parallel I./O system. The ADIOS system from ORNL seems like a good option

http://www.olcf.ornl.gov/center-projects/adios/

[xxu]

Dear Ben,

Yes, it is. I have already written it into our SciDac proposal.
If we will get it funded, we will do it.

Xu

Ben Dudson wrote:

To scale to more processors and make handling large numbers of output files less painful, we need a parallel I./O system. The ADIOS system from ORNL seems like a good option

http://www.olcf.ornl.gov/center-projects/adios/

---

Reply to this email directly or view it on GitHub:
#33

[seanfarley]

I must recommend against ADIOS. (You know why they call it ADIOS, right? Because once you use it, you can say "adios" to your files).

ADIOS is just a gloried wrapper of MPI-IO with a bunch of xml files and obfuscation in-between. First of all, look at the requirements:

• MPI
• MPI-IO
• Python (!!)
• Mini-XML

If you're going to require python, why not ditch xml and go with yaml? Also, have you seen the manual?

http://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.3.1.pdf

It doesn't seem easier to me using this than writing MPI-IO. At least with your own MPI you could output VTK or something else easier that allows viewing (XDMF could work).

Why not just skip this layer of locking ourselves into a whacky framework with ADIOS, and use something much more lightweight (and also scalable): binary-appended data (with perhaps VTK) over MPI-IO? This would require no extra libraries since the format is dead simple.

Plus, it'd be easy to convert the data later if we decide to change the format.

[xxu]

Sean,

Thank you for your comments. If our SciDac project get funded, I will have computer scientists to work those issues out.

Xu

-----Original Message-----
From: Sean Farley [mailto:reply@reply.github.com]
Sent: Friday, December 16, 2011 3:13 PM
To: Xu, Xueqiao
Subject: Re: [BOUT] Parallel input / output (#33)

I must recommend against ADIOS. (You know why they call it ADIOS, right? Because once you use it, you can say "adios" to your files).

ADIOS is just a gloried wrapper of MPI-IO with a bunch of xml files and obfuscation in-between. First of all, look at the requirements:

• MPI
• MPI-IO
• Python (!!)
• Mini-XML

If you're going to require python, why not ditch xml and go with yaml? Also, have you seen the manual?

http://users.nccs.gov/~pnorbert/ADIOS-UsersManual-1.3.1.pdf

It doesn't seem easier to me using this than writing MPI-IO. At least with your own MPI you could output VTK or something else easier that allows viewing (XDMF could work).

Why not just skip this layer of locking ourselves into a whacky framework with ADIOS, and use something much more lightweight (and also scalable): binary-appended data (with perhaps VTK) over MPI-IO? This would require no extra libraries since the format is dead simple.

Plus, it'd be easy to convert the data later if we decide to change the format.

---

Reply to this email directly or view it on GitHub:
#33 (comment)