Skip to content

Add the raw_parser option to pass XML directly to the exporter #4

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add the raw_parser option to pass XML directly to the exporter #4

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions lib/Catmandu/Store/Datahub.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ has client_id => (is => 'ro', required => 1);
has client_secret => (is => 'ro', required => 1);
has username => (is => 'ro', required => 1);
has password => (is => 'ro', required => 1);
has raw_parser => (is => 'ro', default => sub { return 0; });

has client => (is => 'lazy');
has access_token => (
Expand Down Expand Up @@ -97,6 +98,10 @@ Datahub username.

Datahub password.

=item C<raw_parser>

If the input was parsed by the raw XML parser (Catmandu::Importer::OAI::Parser::raw), set this to 1 to pass the XML directly to the exporter.

=back

=head1 USAGE
Expand Down
4 changes: 3 additions & 1 deletion lib/Catmandu/Store/Datahub/API.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ use warnings;
use Catmandu;
use Moo;
use JSON;
use Encode qw(encode_utf8);

use LWP::UserAgent;

Expand Down Expand Up @@ -90,7 +91,8 @@ sub add {
my $token = $self->access_token;
my $response;

$response = $self->client->post($url, Content_Type => 'application/lido+xml', Authorization => sprintf('Bearer %s', $token), Content => $data);
my $enc_data = encode_utf8($data);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do you this here? What's the added benefit of doing an encode_utf8()?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If $data contains non-ascii-characters, the client will fail.

$response = $self->client->post($url, Content_Type => 'application/lido+xml', Authorization => sprintf('Bearer %s', $token), Content => $enc_data);

if ($response->is_success) {
return $response->decoded_content;
Expand Down
9 changes: 8 additions & 1 deletion lib/Catmandu/Store/Datahub/Bag.pm
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,9 @@ sub _build_api {
around add => sub {
my $orig = shift;
my ($self, $data) = @_;
if ($self->store->raw_parser == 1) {
return $self->$orig($data);
}
delete $data->{'_id'};
return $self->$orig($data);
};
Expand Down Expand Up @@ -70,7 +73,11 @@ sub get {
# Create a new record
sub add {
my ($self, $data) = @_;
return $self->api->update($data->{'id'}, $data->{'_'});

if ($self->store->raw_parser == 1) {
return $self->api->add($data->{'_metadata'});
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think the Datahub API should make any assumptions about the datamodel of the incoming data. the _metadata construct is a typical ouptput of OAI. I think this would violate the open/closed principle.

Maybe a separate optional parser object could be injected here. Maybe not in the API implementation itself, but right before the data is passed of to this method.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, should be part of the transformation (perhaps in a fix?).

}
return $self->api->add($data->{'_'});
}

##
Expand Down