Musubi Performance degradation with increasing problem sizes

[haraldkl]

The performance of two coupled Musubi solvers decreases with increasing problem sizes for a cavity setup.

Timing results from the first Musubi:

#       Revision              SimName        DomSize    nProcs         MLUPs  MLUPs_kernel  imbalance(%)    timeMusubi   maxIter          totalDens    timeMainLoop    timeLoadMesh    timeInitLvlD    timewRestart     timeBalance      timeSource     timeAux   timeRelax     Comp(%)     Comm(%)    BCbuffer(%)       BC(%)     Intp(%)
   f80edce4580b+         cavity_fluid          29791       192    200.92E+00      5.07E+03        107.10    4.3648E+00      5000   29.791000000E+03          0.7686          0.1334          0.0223          0.0000          0.0000          0.0489        0.07        0.00        3.96       18.84           1.05        0.39        0.00
   f80edce4580b+         cavity_fluid          64000       192    391.42E+00      5.80E+03        105.22    3.7766E+00      5000   64.000000000E+03          0.8420          0.2910          0.0223          0.0000          0.0000          0.0986        0.11        0.01        6.74       15.83           1.77        0.37        0.00
   f80edce4580b+         cavity_fluid         125000       192    539.73E+00      6.87E+03        103.52    4.6628E+00      5000  125.000000000E+03          1.1858          0.7073          0.0310          0.0000          0.0000          0.1744        0.19        0.01        7.85       12.77           1.89        0.26        0.00
   f80edce4580b+         cavity_fluid         250047       192    427.97E+00      6.58E+03        111.40    6.6400E+00      5001  250.047000000E+03          2.9539          0.2247          0.0403          0.0000          0.0000          0.3337        0.33        0.01        6.51        7.54           1.10        0.12        0.00
   f80edce4580b+         cavity_fluid         493039       192    686.86E+00      6.06E+03        107.08    9.4986E+00      5001  493.039000000E+03          3.6192          0.2972          0.0488          0.0000          0.0000          0.6423        0.91        0.03       11.34        7.54           1.31        0.10        0.00
   f80edce4580b+         cavity_fluid        1000000       192    597.78E+00      3.60E+03        107.01   18.5579E+00      5000    1.000000000E+06          8.4201          0.2001          0.0806          0.0000          0.0000          1.4636        1.76        0.09       16.61       14.20           1.68        0.07        0.00
   f80edce4580b+         cavity_fluid        2000376       192    400.90E+00      1.36E+03        104.89   44.2614E+00      5001    2.000376000E+06         25.0388          0.7530          0.1505          0.0000          0.0000          4.3404        5.92        0.28       29.50        9.84           2.19        0.03        0.00
   f80edce4580b+         cavity_fluid        4019679       192    312.79E+00      1.03E+03        105.59   97.2472E+00      5000    4.019679000E+06         64.3636          0.8667          0.3023          0.0000          0.0000         13.6290       15.66        0.56       30.35        6.76           1.50        0.02        0.00
   f80edce4580b+         cavity_fluid        8000000       192    285.86E+00    946.83E+00        105.85  203.4730E+00      5000    8.000000000E+06        140.0620          0.9625          0.5397          0.0000          0.0000         33.1516       38.22        2.00       30.19       10.85           1.16        0.01        0.00
   f80edce4580b+         cavity_fluid       16003008       192    271.22E+00      1.01E+03        105.22  430.3999E+00      5001   16.003008000E+06        295.6390          2.3404          1.0559          0.0000          0.0000         77.7521       89.46        4.75       26.77       12.74           0.96        0.01        0.00

Timing results from the second Musubi:

   f80edce4580b+             cavity_T          29791       192    218.42E+00      2.31E+03        112.99    4.3593E+00      5000   14.843709680E+03          0.7090          1.4949          0.4072          0.0000          0.0000          0.0006        0.05        0.00        9.46       12.48           1.13        2.94        0.00
   f80edce4580b+             cavity_T          64000       192    491.81E+00      2.50E+03        109.97    3.7775E+00      5000   31.748950607E+03          0.6771          1.0617          0.0277          0.0000          0.0000          0.0006        0.06        0.00       19.66       13.58           2.07        4.04        0.00
   f80edce4580b+             cavity_T         125000       192    661.28E+00      2.56E+03        110.56    4.6638E+00      5000   60.755966848E+03          0.9729          1.1322          0.0317          0.0000          0.0000          0.0007        0.11        0.00       25.86       10.07           2.52        4.46        0.00
   f80edce4580b+             cavity_T         250047       192    420.54E+00      2.28E+03        128.81    6.6399E+00      5001  117.112515274E+03          3.0120          0.5505          0.0888          0.0000          0.0000          0.0009        0.17        0.00       18.42        3.87           1.11        2.11        0.00
   f80edce4580b+             cavity_T         493039       192    861.90E+00      2.33E+03        113.28    9.4982E+00      5001  219.241497252E+03          2.8871          1.2838          0.0824          0.0000          0.0000          0.0010        0.34        0.00       37.00       14.19           1.61        3.29        0.00
   f80edce4580b+             cavity_T        1000000       192    808.66E+00      2.00E+03        106.09   18.5589E+00      5000  412.217421027E+03          6.2318          1.2603          0.0899          0.0000          0.0000          0.0013        0.98        0.00       40.43       10.36           2.37        3.05        0.00
   f80edce4580b+             cavity_T        2000376       192    545.95E+00      1.13E+03        104.33   44.2620E+00      5001  749.071840980E+03         18.3551          4.8015          0.1785          0.0000          0.0000          0.0023        3.72        0.00       48.25        9.60           3.06        3.05        0.00
   f80edce4580b+             cavity_T        4019679       192    388.82E+00    832.38E+00        105.98   97.2482E+00      5000    1.340734242E+06         51.7386          1.8442          0.4857          0.0000          0.0000          0.0035        9.62        0.00       46.71        9.69           2.51        2.54        0.00
   f80edce4580b+             cavity_T        8000000       192    398.31E+00    674.56E+00        105.25  203.4739E+00      5000    2.342155756E+06        100.7803          2.0471          0.6323          0.0000          0.0000          0.0048       23.37        0.00       59.05        2.79           2.50        2.34        0.00
   f80edce4580b+             cavity_T       16003008       192    385.29E+00    619.10E+00        104.87  430.3996E+00      5001    4.076251599E+06        207.9912          4.1267          1.1955          0.0000          0.0000          0.0049       44.81        0.01       62.23        5.55           1.97        1.96        0.00

It seems that MLUPs_kernel becomes stable with the increase of elements per node.

Originally posted by @MikeW097 in #4 (comment)

[haraldkl]

There are some strange observations here, but it's hard to make any statements without knowing anything about the actual setup. Could you share that? Possibly put it into testsuite/musubi.
I also do not know if MLUPS_kernel includes times for the coupling computation. I can't really explain why that should be around twice as fast in the one domain than in the other. It would require a more detailed investigation into this to understand its causes.

[haraldkl]

2020-01-08_Masilamani_APESmate_MMCP2020.pdf offers some slides on APESmate, however the performance analysis is only done with Ateles there.

[MikeW097]

Hi @haraldkl , the testcase commits with 996677a.

Both MLUPs and MLUPs_kernel measure time within Musubi, with MLUPs for the entire solving process (mainloop), and MLUPs_kernel only for computation (collision). I'm not aware of any performance measurements in Apesmate.

[haraldkl]

Thanks @MikeW097, so this is a volume coupling with perfectly aligned meshes, one with the fluid simulation

identify = {
  label      = 'fluid',
  kind       = 'fluid_incompressible',
  relaxation = 'bgk',
  layout     = 'd3q19'
}

and one with the passive scalar transport:

identify = {
  label      = 'species',
  kind       = 'passive_scalar',
  relaxation = 'trt',
  layout     = 'd3q19',
}

For the volume coupling my previous elaborations on the coupling interface does not apply. And can not be a possible explanation. That one solves navier stokes and the other only passive scalar transport explains the large difference in the kernel performance between the two domains. However, that also means that the two domains are not well balanced, and one always will have to wait on the other if you give them each the same number of processors. But I'm doubtful that this can be an explanation for the performance drop in larger problems per node.

Apes itself should also write out a timing.res, I think.

[MikeW097]

Thanks @MikeW097, so this is a volume coupling with perfectly aligned meshes, one with the fluid simulation

identify = {
  label      = 'fluid',
  kind       = 'fluid_incompressible',
  relaxation = 'bgk',
  layout     = 'd3q19'
}

and one with the passive scalar transport:

identify = {
  label      = 'species',
  kind       = 'passive_scalar',
  relaxation = 'trt',
  layout     = 'd3q19',
}

For the volume coupling my previous elaborations on the coupling interface does not apply. And can not be a possible explanation. That one solves navier stokes and the other only passive scalar transport explains the large difference in the kernel performance between the two domains. However, that also means that the two domains are not well balanced, and one always will have to wait on the other if you give them each the same number of processors. But I'm doubtful that this can be an explanation for the performance drop in larger problems per node.

Apes itself should also write out a timing.res, I think.

Thanks a lot for your explanation @haraldkl !

If I understand you correctly, the imbalance between the Navier–Stokes domain and the passive-scalar domain should be more or less constant when I increase the number of elements per node, so by itself it shouldn’t cause the strong drop in MLUPS that I see for the largest problems. Could that be a combined result from both cache benefit and imbalance?

I attach here the timing.res from apesmate:

#     Revision             Casename   nProcs  threads      maxIter     timeApes  loadConfig|        init|  initSolver|     initCpl|   fillStFun|init_cplComm|  roundRobin|   checkVars|  identRanks|   exchRanks| exchCplData|createCplVar|    finalize| finalSolver| aps_simLoop|    syncDoms|    commWait|     evalVal|dom_fluidWai|dom_fluidSin|dom_fluidEva|   dom_TWait| dom_TSingle|   dom_TEval|       MemRSS       MemHWM
 f80edce4580b+      cavity_coupling     384        1         5000    4.463E+00    1.587E+00    1.737E+00  415.285E-03    1.703E+00  502.815E-06    1.703E+00    1.699E+00   81.792E-06   84.303E-06  261.209E-06   10.193E-03    1.672E-03   15.683E-03   15.676E-03    1.125E+00  236.294E-03  190.342E-03   63.976E-03  293.776E-03  781.774E-03   51.870E-03  289.069E-03  721.245E-03   59.880E-03      154800      154800
 f80edce4580b+      cavity_coupling     384        1         5000    3.856E+00    1.137E+00    1.311E+00   32.998E-03    1.278E+00  639.792E-06    1.278E+00    1.274E+00  457.054E-06  120.904E-06  461.786E-06   24.845E-03    1.401E-03    2.458E-03    1.604E-03    1.403E+00  504.662E-03  428.337E-03  120.563E-03  453.832E-03  854.987E-03  116.015E-03  440.168E-03  690.494E-03  111.976E-03      150984      150984
 f80edce4580b+      cavity_coupling     384        1         5000    5.123E+00    1.590E+00    1.341E+00   41.240E-03    1.304E+00  647.302E-06    1.304E+00    1.299E+00   64.773E-06  273.280E-06    3.441E-03   12.563E-03    1.582E-03    1.877E-03    1.869E-03    2.189E+00  961.251E-03  830.489E-03  256.842E-03  757.223E-03    1.199E+00  228.708E-03  732.224E-03  986.942E-03  251.885E-03      156008      156008
 f80edce4580b+      cavity_coupling     384        1         5001    6.721E+00  632.125E-03    1.362E+00   97.304E-03    1.309E+00  585.150E-06    1.309E+00    1.304E+00   60.421E-06  379.043E-06   81.663E-06   17.471E-03    2.098E-03    2.046E-03    2.037E-03    4.726E+00    1.288E+00    1.041E+00  483.248E-03    1.355E+00    2.969E+00  461.219E-03    1.302E+00    3.028E+00  477.696E-03      159512      159512
 f80edce4580b+      cavity_coupling     384        1         5001    9.601E+00    1.388E+00    1.431E+00   91.447E-03    1.349E+00  424.644E-06    1.349E+00    1.309E+00   53.892E-06  758.577E-06   11.820E-03   55.269E-03    2.504E-03    1.590E-03    1.579E-03    6.782E+00    3.447E+00    2.982E+00  953.587E-03    1.748E+00    3.636E+00  947.830E-03    1.784E+00    2.903E+00  858.414E-03      160564      160564
 f80edce4580b+      cavity_coupling     384        1         5000   18.641E+00    1.339E+00    1.460E+00  102.798E-03    1.336E+00  333.881E-06    1.336E+00    1.294E+00   64.992E-06    1.325E-03   13.836E-03   58.881E-03    5.325E-03    2.648E-03    1.564E-03   15.837E+00    8.166E+00    7.229E+00    1.951E+00    4.638E+00    8.446E+00    1.943E+00    4.564E+00    6.251E+00    1.583E+00      159176      159176
 f80edce4580b+      cavity_coupling     384        1         5001   44.437E+00    4.975E+00    1.608E+00  199.957E-03    1.392E+00  341.612E-06    1.392E+00    1.323E+00   48.292E-06    2.522E-03   35.765E-03   93.014E-03   10.240E-03    1.797E-03    1.775E-03   37.851E+00   18.330E+00   16.139E+00    4.296E+00    8.187E+00   25.076E+00    4.058E+00    8.299E+00   18.387E+00    4.287E+00      171820      171820
 f80edce4580b+      cavity_coupling     384        1         5000   97.340E+00    1.936E+00    2.031E+00  529.661E-03    1.628E+00  131.955E-06    1.628E+00    1.530E+00   55.052E-06    5.194E-03   49.031E-03   88.712E-03   21.076E-03    1.878E-03    1.840E-03   93.371E+00   38.350E+00   33.709E+00   10.721E+00   19.433E+00   64.415E+00   10.705E+00   19.694E+00   51.784E+00    8.575E+00      189108      191356
 f80edce4580b+      cavity_coupling     384        1         5000  203.600E+00    2.171E+00    2.191E+00  716.155E-03    1.489E+00   60.462E-06    1.489E+00    1.365E+00   60.802E-06   10.541E-03   48.073E-03   88.421E-03   44.242E-03    2.048E-03    2.011E-03  199.234E+00   92.742E+00   83.120E+00   21.529E+00   41.350E+00  140.126E+00   21.511E+00   41.452E+00  100.831E+00   15.622E+00      223608      235040
 f80edce4580b+      cavity_coupling     384        1         5001  430.561E+00    4.207E+00    2.914E+00    1.359E+00    1.544E+00   65.022E-06    1.544E+00    1.367E+00   64.963E-06   20.114E-03   31.357E-03   86.275E-03   88.707E-03   85.042E-03   84.895E-03  423.358E+00  203.976E+00  183.192E+00   37.143E+00   68.513E+00  295.718E+00   37.118E+00   69.341E+00  208.047E+00   33.711E+00      277940      285876

As far as I can see, MLUPS cannot be computed cleanly from it. But I can see that the dom_Wait takes up about 20% of the total time, and the commWait is roughly 30%.

[haraldkl]

so by itself it shouldn’t cause the strong drop in MLUPS that I see for the largest problems.

Yes, that's exactly what I think aswell.

Could that be a combined result from both cache benefit and imbalance?

The dropping out of the cache-region shouldn't be as significant, but if that affects also the coupling effort, we may see a compound effect emphasizing it and resulting in this strong drop.

The apes timer shows that the imbalance grows strongly somehow.
The syncDoms more than doubles with each doubling of the domain size. We'd expect a proportional growth there if the imbalance remains the same.

The commWait makes a notable jump from 34 s on 4 mio. elements to 83 s on 8 mio. elements. So something seems to significantly change at that point. Maybe the cache region ends are different for the respective solvers, and one falls out of while the other doesn't and thus, you end up with a larger imbalance and longer waiting times.
This should stabilize again for sufficiently large problems. Though it may be that the difference in the memory region is different from the one in the cache region.

I think you can use the dom_*Single metrics for an estimate of how much ressources you need in them respectively.

size	fluid	T
29791	781.774E-03	721.245E-03
64000	854.987E-03	690.494E-03
125000	1.199	986.942E-03
250047	2.969E+00	3.028E+00
493039	3.636E+00	2.903E+00
1000000	8.446E+00	6.251E+00
2000376	25.076E+00	18.387E+00

Notably it claims less time for the T domain on half a million elements than for a 250k elements. Maybe there is some run time variation on the machine at play there aswell. In any case as you can see, the relation between the two domains also depends on the problem size, so you'd need a different processor distribution in the different problem sizes for improving the balancing between domains.

[MikeW097]

@haraldkl Thanks again for your insights!

I did some additional tests to study the scaling behavior in more detail.
To separate the cache effect from the imbalance effect, I ran the two Musubi solvers individually.

Timing results for single musubi_fluid:

    65b56224dada         cavity_fluid          29791       192    198.72E+00      5.35E+03        109.73    2.3110E+00      5000   29.791000000E+03          0.7496          0.6634          0.2531          0.0000          0.0000          0.0484        0.07        0.00        3.71       21.17           0.88        0.39        0.00
    65b56224dada         cavity_fluid          64000       192    440.88E+00      6.77E+03        107.86  834.6533E-03      5000   64.000000000E+03          0.7258          0.0799          0.0682          0.0000          0.0000          0.0909        0.10        0.00        6.51       18.14           1.54        0.42        0.00
    65b56224dada         cavity_fluid         125000       192    632.13E+00      6.82E+03        106.95    1.7810E+00      5001  125.000000000E+03          0.9889          0.7582          0.7435          0.0000          0.0000          0.1681        0.18        0.01        9.26       13.82           1.89        0.30        0.00
    65b56224dada         cavity_fluid         250047       192    478.99E+00      5.82E+03        114.92    2.8896E+00      5000  250.047000000E+03          2.6101          0.2294          0.2296          0.0000          0.0000          0.3306        0.31        0.01        8.23        7.46           1.06        0.13        0.00
    65b56224dada         cavity_fluid         493039       192    621.42E+00      6.22E+03        105.60    5.7120E+00      5000  493.039000000E+03          3.9670          1.3690          1.4470          0.0000          0.0000          0.6428        1.14        0.02        9.98       14.71           0.98        0.08        0.00
    65b56224dada         cavity_fluid        1000000       192    757.92E+00      4.04E+03        106.13    6.9725E+00      5001    1.000000000E+06          6.5984          0.1384          0.1197          0.0000          0.0000          1.3606        1.48        0.08       18.76       10.92           2.09        0.09        0.00
    65b56224dada         cavity_fluid        2000376       192    431.71E+00      1.40E+03        104.69   23.9173E+00      5000    2.000376000E+06         23.1679          0.1652          0.1943          0.0000          0.0000          4.1911        5.74        0.25       30.81       10.67           2.77        0.03        0.00
    65b56224dada         cavity_fluid        4019679       192    327.82E+00      1.04E+03        106.26   61.8398E+00      5000    4.019679000E+06         61.3099          0.1546          0.3430          0.0000          0.0000         13.1709       15.23        0.70       31.62        6.87           1.91        0.02        0.00
    65b56224dada         cavity_fluid        8000000       192    295.34E+00    909.76E+00        105.94  137.3276E+00      5001    8.000000000E+06        135.4650          0.9368          1.4095          0.0000          0.0000         31.6667       38.61        2.20       32.46        5.97           1.27        0.01        0.00
    65b56224dada         cavity_fluid       16003008       192    273.88E+00    991.31E+00        105.23  295.5726E+00      5000   16.003008000E+06        292.1551          1.5984          1.6622          0.0000          0.0000         77.5002       89.54        4.70       27.63        8.91           1.02        0.01        0.00

Timing results for single musubi_T:

    65b56224dada             cavity_T          29791       192    224.11E+00      2.39E+03        118.04    1.0664E+00      5000   26.285529252E+03          0.6647          0.3707          0.3674          0.0000          0.0000          0.0006        0.05        0.00        9.39       13.89           1.00        3.09        0.00
    65b56224dada             cavity_T          64000       192    495.14E+00      2.49E+03        117.21  785.1669E-03      5000   57.152886419E+03          0.6463          0.1074          0.0989          0.0000          0.0000          0.0007        0.06        0.00       19.88       12.89           1.96        4.09        0.00
    65b56224dada             cavity_T         125000       192    724.99E+00      2.65E+03        115.00  998.4376E-03      5001  110.347883542E+03          0.8623          0.1038          0.0732          0.0000          0.0000          0.0010        0.10        0.00       27.35       10.89           2.15        4.81        0.00
    65b56224dada             cavity_T         250047       192    445.09E+00      2.58E+03        131.59    3.3391E+00      5000  209.303674251E+03          2.8090          0.4397          0.4762          0.0000          0.0000          0.0009        0.20        0.00       17.22        4.57           1.07        3.00        0.00
    65b56224dada             cavity_T         493039       192    903.62E+00      2.38E+03        114.98    3.3552E+00      5000  390.040334021E+03          2.7281          0.1038          0.5634          0.0000          0.0000          0.0011        0.31        0.00       38.03       14.52           1.57        3.45        0.00
    65b56224dada             cavity_T        1000000       192      1.09E+03      2.27E+03        107.09    5.2992E+00      5001  734.696258767E+03          4.5823          0.5906          0.6051          0.0000          0.0000          0.0011        0.69        0.00       48.02        7.02           1.93        3.21        0.00
    65b56224dada             cavity_T        2000376       192    585.60E+00      1.27E+03        103.75   17.4399E+00      5000    1.287588388E+06         17.0797          0.1639          0.2214          0.0000          0.0000          0.0652        3.37        0.00       46.12       10.43           3.28        3.35        0.00
    65b56224dada             cavity_T        4019679       192    405.95E+00    913.06E+00        105.47   51.2202E+00      5000    2.167181006E+06         49.5102          1.1614          1.5505          0.0000          0.0000          0.0038        9.74        0.00       44.46       10.82           2.63        2.70        0.00
    65b56224dada             cavity_T        8000000       192    416.56E+00    671.05E+00        105.69   97.6111E+00      5001    3.684414267E+06         96.0437          0.1554          1.3671          0.0000          0.0000          0.0058       22.83        0.01       62.08        2.30           2.18        1.93        0.00
    65b56224dada             cavity_T       16003008       192    423.89E+00    647.98E+00        104.31  190.8230E+00      5000    6.162637168E+06        188.7636          0.7154          1.5132          0.0000          0.0000          0.0047       43.82        0.01       65.42        2.99           1.94        2.24        0.00

A steep performance drop is also observed when running the solvers individually.
This indicates that the main cause of the drop seen in the coupled solver is likely related to the cache effects, rather than to the coupling imbalance itself.
A graphical comparison is shown on MLUPs and MLUPs_kernel:

It's only the MLUPs on 1 mio lattices differs largely with musubi_T.

[jainka]

@MikeW097 Thanks for this analysis. Just for reference I attach the full performance map of Musubi on Snellius. Here, as can be seen, the performance increases up to a certain number of cores and then drops due to increased comm/comp ratio. At large number of elements (purple and brown line), the performance does not drop which is expected as the nElems are quite large.
The so called caching effect is seen only in the green line. When the problem fits perfectly in the cache, there is an increase and there is a sudden drop after this region, which again stabilises upon increasing nCores.
Can you please run the uncoupled (fluid) case with 8^10 elements? If it is the same as seen with only Musubi as in the plot below, that will ensure that ApesMate behaves the same as Musubi.

[jainka]

@MikeW097 Also for my own reference, can you paste the compilers you are using on Snellius? Basically the outcome of module list command.

[MikeW097]

For the imbalance test, I changed the processor distribution between musubi_fluid and musubi_T.
Based on the previous results, for the large meshes with 8 and 16 million elements, the time-consumption ratio between the fluid and T domains was about 1.35.
Given a total of 384 processors, I assigned 221 to the fluid domain and 163 to the T domain.

With this setup, the commWait decreased significantly — 44 s vs 83 s for 8 million elements, and 94 s vs 183 s for 16 million elements.
The dom_*single ratio is now around 1.1, so the imbalance is largely reduced.

Timing result for Apesmate:

 0ccaacb5db22      cavity_coupling     384        1         5000  186.195E+00    7.026E+00    2.351E+00  840.503E-03    1.678E+00  105.104E-06    1.678E+00    1.532E+00   54.521E-06   20.750E-03   58.548E-03  109.653E-03   47.503E-03   36.909E-03   36.890E-03  176.778E+00   56.093E+00   44.249E+00   18.771E+00   36.539E+00  127.100E+00   15.901E+00   35.281E+00  116.412E+00   18.752E+00      214196      221580
  0ccaacb5db22      cavity_coupling     384        1         5001  390.759E+00   31.775E+00    3.160E+00    1.606E+00    1.918E+00   79.663E-06    1.918E+00    1.728E+00   76.053E-06   42.914E-03   66.828E-03  122.621E-03   94.385E-03    2.520E-03    2.491E-03  355.819E+00  119.095E+00   94.663E+00   36.391E+00   73.502E+00  257.461E+00   36.355E+00   71.726E+00  230.689E+00   33.614E+00      260856      268572