Switch from XML::Simple to XML::LibXML or XML::Fast

[pplu]

XML::Simple is discouraged, although it works for us. Maybe a faster and better XML parser would be nice

[Grinnz]

While XML::Fast may be better than XML::Simple (I don't know if this is the case), it doesn't solve the fundamental problems that make XML::Simple a bad idea -- having to deal with endless edge cases of items that could be arrays or single values, and tags that have both attributes and contents... this will be a problem in any XML-to-hash converter.

Using XML::LibXML or something like Mojo::DOM58 or another DOM traversal parser would solve this issue, but would be a much trickier conversion, you have to change how you approach the data.

[pplu]

+1: XML as a datasctructure is a pain :(. Be sure some ugly code is laying around because of that fact.

If you have any suggestions of how you would use XML::LibXML or Mojo::DOM58 in Paws, I'd love to see them.

With the branch I've just pushed I was playing around to see if a quick win is possible (I haven't benchmarked it yet, but it feels faster). Responses from S3 ListObjects with lots of elements are slow with XML::Simple, and seem to be faster with XML::Fast. The bad part is XML::Fast has not been maintained since 2010, so maybe it's not very dependable as a Paws core dep.

[Grinnz]

I will take a look when I get a chance at how a DOM object might be used, but I don't understand enough of the code using XML responses yet.

[pplu]

I've just found out about https://metacpan.org/pod/XML::Compile. It could be an option.

[zostay]

I recommend leaving XML::Simple for security reasons in addition to performance.

There is a significant XXE vulnerability that was reported 4 years ago in RT for XML::Simple. There continues to be no fix or work around that I am aware of. Without doing a code review of every use of XML::Simple to parse XML data in Paws, I'm not exactly sure how the existing vulnerability might be exploited. Basically it works like this, though, given an XML file that has a particular kind of doctype entity tag, XML::Simple will read any arbitrary file from the local disk and embed it into the parsed XML document tree. If the program using that document tree at all passes that data on, it can get logged or otherwise passed through to something else to leak information like password files.

XML::LibXML::Simple can usually be used as a drop-in replacement for parsing XML data while XML::Simple can still be used for writing XML safely (I don't believe XML::LibXML::Simple can generate XML).

[pplu]

@zostay Thanks for the heads-up about XML::LibXML::Simple (didn't know it existed, and was drop-in for XML::Simple)

[pplu]

Hi,
I just tried out XML::LibXML::Simple on this branch. It fails tests because of differences in how it converts XML to a datastructure. https://github.com/pplu/aws-sdk-perl/tree/feature/xml-libxml-simple

perl -I lib/ -I auto-lib/ t/10_responses.t t/10_responses/s3-list-objects.1.response
[...]
$VAR1 = {
          'MaxKeys' => '1000',
          'Marker' => {},
          'Prefix' => {},

The problem seems to be that Marker, Prefix, etc, etc are empty HashRefs. The XML::LibXML::Simple documentation states that one of the differences with XML::Simple is that empty elements are not removed. https://metacpan.org/pod/XML::LibXML::Simple#empty-elements-are-not-removed.

AWS returns:

  <?xml version="1.0" encoding="UTF-8"?>
  <ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <Name>test-1357854246</Name>
    <Prefix></Prefix>
    <Marker></Marker>

[Grinnz]

I've started taking a look at how Paws could be converted to use a proper XML parser. The fundamental issue is that when doing it correctly, you do not deserialize the XML to a data structure at all but an object which you then query. Thus the method response_to_object in Paws::API::Caller would have to understand this. It currently appears to assume that unserialize_response will return a data structure. Either Paws::Net::XMLResponse and Paws::Net::RestXMLResponse would have to define a new method which returns the XML object, which Paws::API::Caller would then call and use based on some condition, or it would have to check whether unserialize_response returns an object rather than a basic data structure and handle it appropriately. What are your thoughts on the best way to do this in keeping with the existing structure? Mainly, response_to_object also has to handle JSON responses which deserialize cleanly to plain data structures.

If this fundamental issue can be worked out then it makes little difference whether XML::LibXML or Mojo::DOM58 or whatever other parser is used afterward. (And I don't consider XML::LibXML::Simple or XML::Fast or other xml-to-hash converters to be a real solution to this problem.)

[Grinnz]

Another kind of crazy idea: put JSON responses in an object too, using JSON pointers to query them. Unfortunately the implementation I'd want to use for that is Mojo::JSON::Pointer and it's a bit silly to depend on Mojolicious just for that. (JSON::Pointer seems super overcomplicated for this task)

[Grinnz]

A further problem: response_to_object primarily builds the response object by passing it the plain data structure it receives and relying on the constructor to build the object from there. If deserializing to an object instead of a plain data structure, it wouldn't be able to automatically build it this way; it needs to somehow query the object for each element it's looking for.

[Grinnz]

Perhaps this could be solved by a new method for XMLResponse and RestXMLResponse as well: new_from_result_object or so.

[Grinnz]

So it basically boils down to this: what would be the most appropriate way for response_to_object to determine that it needs to work with an XML object (from XMLResponse or RestXMLResponse) rather than a plain data structure (from JsonResponse or RestJsonResponse)?

[pplu]

Take a look at the code if the refactored code in this branc helps https://github.com/pplu/aws-sdk-perl/tree/release/0.37

[Grinnz]

Oh, on that branch each response type defines its own response_to_object, that should make things a lot easier.

[pplu]

🎉 🎈