Skip to content

Conversation

@jdlesage
Copy link

This PR adds DelegationToken for the MapReduce History Server when submitting a new application

My use case
I call skein from a periodic job written in python to archive some files in HDFS using hadoop archive. The goal is to reduce the number of files in HDFS.
I call from a Skein Service the hadoop command hadoop archive that launches a Map Reduce job that creates the archive.
MapReduce jobs do some connections to the MapReduce History Server. It needs a delegation token to authenticate to this service. For this reason, I would like skein also to create a delegation token for this service.

Implementation details
The code is adapted from the hadoop mapreduce project. Unfortunately, I need some functions that are private. That's why I adapted the code in skein.
Original sourcefile: https://github.com/apache/hadoop/blob/branch-2.7.3/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/main/java/org/apache/hadoop/mapred/YARNRunner.java#L183
If the MapReduce History Server address is not present in the hadoop configuration files. The code does nothing.

How did I test it ?
I tested it by running a skein service that do a subprocess.check_output that triggers the hadoop archive command.

@jdlesage
Copy link
Author

The tests are failing because in the hadoop cluster used for test, the MR History Service is configured in mapred.xml but not started with the cluster
I did also a PR in hadoop-test-cluster project to start it also.

Introduce optional boolean in applicationspec

Change-Id: I5afa12e538629182dd7dbc6fe587925fb4d59557
@jdlesage
Copy link
Author

I added an option to request the new token. I consider it is safer because most of people doesn't care about map reduce history server, and I don't want to change the behavior in this case or break their apps.

Change-Id: Ife97067d5a0d57299092d97bf1863aef8dbfdfe0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant