fed: Difference between fdsnws-station text and xml

[damb]

Continued discussion from eida_maint@gfz-potsdam.de.

---

Dear All,

At RESIF we are testing eida-federator, we are doing manual and automatic tests.

For our automatic tests we are using one script in Python and we will share it next week. For now it seems working well however we have strange things between text format and xml format

For example with net=Z3 eida-federator provides two different results:

federator-testing.ethz.ch/fdsnws/station/1/query?level=network&network=Z3&format=text

#Network|Description|StartTime|EndTime|TotalStations
Z3|AlpArray Seismic Network (AASN) temporary
component|2015-01-01T00:00:00|2020-07-01T00:00:00|29
Z3|AlpArray Seismic Network (AASN) temporary
component|2015-01-01T00:00:00|2020-07-01T00:00:00|66
Z3|AlpArray DSEBRA|1980-01-01T00:00:00||92
Z3|AlpArray Seismic Network (AASN) temporary
component|2015-01-01T00:00:00|2020-07-01T00:00:00|51
Z3|Egelados project, RUB Bochum,
Germany|2005-06-05T00:00:00|2007-04-30T00:00:00|56
Z3|AlpArray backbone temporary
stations|2015-07-01T00:00:00|2020-07-31T00:00:00|68

federator-testing.ethz.ch/fdsnws/station/1/query?level=network&network=Z3&format=xml

<FDSNStationXML schemaVersion="1.0">
<Source>EIDA</Source>
<Created>2018-07-27T10:11:57.586061</Created>
<Network code="Z3"
alternateCode="ALPARRAY"
startDate="2015-07-01T00:00:00.000000"
endDate="2020-07-31T00:00:00.000000"
restrictedStatus="closed">
<Description>AlpArray backbone temporary stations</Description>
<Comment>
<Value>DOI:http://dx.doi.org/10.12686/alparray/z3_2015</Value>
<BeginEffectiveTime>2015-07-01T00:00:00.000000</BeginEffectiveTime>
<EndEffectiveTime>2020-07-31T00:00:00.000000</EndEffectiveTime>
<Author><Name>Resif Information System</Name>
<Agency>Réseau Sismologique et géodésique Français (RESIF)</Agency>
<Email>sismob@resif.fr</Email></Author>
</Comment>
<TotalNumberStations>68</TotalNumberStations>
<SelectedNumberStations>306</SelectedNumberStations>
</Network>
</FDSNStationXML>

Best,
Gregory

[Jollyfant]

Is that because the network in the StationXML is being combined to a single entity? It is pretty tricky to combine "identical" networks if their descriptions are even slightly different.

[damb]

Hi Gregory,

in general fdsnws-station text and xml is handled completely separately.

However, there is a bug in https://github.com/EIDA/mediatorws/blob/master/eidangservices/federator/server/task.py#L314-L317. It exclusively occurs for fdsnws-station-xml and level=network in case the client requests a network which is distributed over multiple EIDA nodes (e.g. net=Z3).

I'm going to provide a fix.

cheers

[damb]

fdsnws-station-xml and level=station|channel|response also seems to be affected for net=Z3 (i.e. a distributed network). Correct, @Jollyfant, unfortunately I did not take <Description></Description> into account while merging.

[Jollyfant]

Yeah but we should probably agree within EIDA to describe Z3 in am identical way, so for users it becomes clear that it is one network.

[damb]

Yeah but we should probably agree within EIDA to describe Z3 in am identical way, so for users it becomes clear that it is one network.

Merging the `NetworkType` element unfortunately is not that trivial. [StationXML](http://www.fdsn.org/xml/station/fdsn-station-1.0.xsd) even does not prevent *EIDA* nodes from providing multiple network epochs with the same `code` attribute but different other `BaseNodeType` attributes (including additional `NetworkType` and `BaseNodeType` child elements (again see: http://www.fdsn.org/xml/station/fdsn-station-1.0.xsd)). IMO merging is no option. Rather, we should implement a solution which is able to handle multiple `NetworkType` elements (AKA network epochs) including a proper merging of child elements. Also, this approach prevents us from hiding data relevant information.

[damb]

Pushed new image to DockerHub containing the latest changes from next.

Note:
For fdsnws-station/format=text currently no merging is performed. Hence, the results are different when comparing with fdsnws-station/format=xml:

fdsnws/station/1/query?level=network&net=Z3&format=text

Z3|AlpArray Seismic Network (AASN) temporary component|2015-01-01T00:00:00|2020-07-01T00:00:00|29
Z3|AlpArray DSEBRA|1980-01-01T00:00:00||93
Z3|AlpArray Seismic Network (AASN) temporary component|2015-01-01T00:00:00|2020-07-01T00:00:00|66
Z3|AlpArray backbone temporary stations|2015-07-01T00:00:00|2020-07-31T00:00:00|68
Z3|Egelados project, RUB Bochum, Germany|2005-06-05T00:00:00|2007-04-30T00:00:00|56

fdsnws/station/1/query?level=network&net=Z3&format=xml

<FDSNStationXML schemaVersion="1.0"><Source>EIDA</Source><Created>2018-07-31T15:07:00.076980</Created>
<Network code="Z3" startDate="2015-01-01T00:00:00" endDate="2020-07-01T00:00:00" restrictedStatus="closed"><Description>AlpArray Seismic Network (AASN) temporary component</Description></Network>
<Network code="Z3" startDate="2005-06-05T00:00:00" endDate="2007-04-30T00:00:00" restrictedStatus="open"><Description>Egelados project, RUB Bochum, Germany</Description></Network>
<Network code="Z3" startDate="1980-01-01T00:00:00" restrictedStatus="closed"><Description>AlpArray DSEBRA</Description></Network>
<Network code="Z3" alternateCode="ALPARRAY" startDate="2015-07-01T00:00:00.000000" endDate="2020-07-31T00:00:00.000000" restrictedStatus="closed"><Comment><Value>DOI:http://dx.doi.org/10.12686/alparray/z3_2015</Value><BeginEffectiveTime>2015-07-01T00:00:00.000000</BeginEffectiveTime><EndEffectiveTime>2020-07-31T00:00:00.000000</EndEffectiveTime><Author><Name>Resif Information System</Name><Agency>Réseau Sismologique et géodésique Français (RESIF)</Agency><Email>sismob@resif.fr</Email></Author></Comment><Description>AlpArray backbone temporary stations</Description><SelectedNumberStations>306</SelectedNumberStations><TotalNumberStations>68</TotalNumberStations></Network>
</FDSNStationXML>

However, it only seems to make a difference when level=network.

[damb]

fdsnws/station/1/query?level=network&net=Z3&format=text

Z3|AlpArray Seismic Network (AASN) temporary component|2015-01-01T00:00:00|2020-07-01T00:00:00|29
Z3|AlpArray Seismic Network (AASN) temporary component|2015-01-01T00:00:00|2020-07-01T00:00:00|66
[...]

@kaestli:
I considered merging the response for fdsnws/station/1/query?level=network&format=text. Usually when having multiple network epochs IMO this seems to make sense. On the other side the number of TotalStations might differ which simply could be added. In case of net=Z3 this probably would work out properly.
For a more general implementation, however, this is not satisfying since always there might occur differences between federated fdsnws-station?format=text&level=network and fdsnws-station?format=xml&level=network results (fdsnws-station?format=xml provides merging under consideration of all <Network></Network> epoch properties (attributes and child elements)). Hence, merging fdsnws-station?format=text&level=network exclusively does make sense when having access to all <Network></Network> epoch properties.
In case of serving fdsnws-station?format=text&level=network directly using eida-stationlite both merging and keeping consistency with fdsnws-station?format=xml&level=network might be feasible (see: #28). Although, this would imply we'd have to harvest <Network></Network> epoch related information additionally.