loading patched avro jars

[piccolbo]

we have two patched jars to solve the NaN problem
both supplied with the libjars syntax
hadoop picks up only one of the two
very confusing because a different version is also available with Hadoop
test with as-planted to repro the error on the NaN
In standalone, workaround is to change HADOOP_CLASSPATH

[jamiefolson]

Have you been able to capture the command-line call to Hadoop?

[piccolbo]

debug(rmr.stream)
step until the system call
variable is final.command

On Fri, Mar 7, 2014 at 7:00 AM, Jamie F Olson notifications@github.comwrote:

Have you been able to capture the command-line call to Hadoop?

Reply to this email directly or view it on GitHubhttps://github.com//issues/3#issuecomment-37031388
.

[piccolbo]

workaround doesn't work in distributed mode. Reported to Cloudera here https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/LlH-ftwoNVY
Will try other conf adjustments. The problem is that it's never clear what changes apply to every node and what changes require a restart of at least the tasktracker.

[piccolbo]

Relevant doc here http://www.kiji.org/2013/10/08/fixing-classpath-ordering-issues-in-hadoop/
Amended here for cdh5 https://groups.google.com/a/cloudera.org/forum/#!topic/cdh-user/kiWv2PFjT1s
Tested on cdh4, makes job fail in setup phase, no workable messages in log that I could find.
Found this error
org.apache.hadoop.ipc.RemoteException: Server IPC version 7 cannot communicate with client version 4
which is commented upon in Stackoverflow (approved answer)
IPC version 4 is for Hadoop 1.0, whereas version 7 is for Hadoop 2.0, so it sounds like you're running pig with its embedded hadoop version but trying to talk to a Hadoop 2 cluster.
Since we compiled the jars on hadoop 2, I guess this may be a Yarn vs not yarn thing more than hadoop2
Would the client need both and if so, what is the mechanism to select them?

[piccolbo]

This problem has been reported before and fixed, but looking that the build script, only if we do build dist, which doesn't work, and not build test which was used to create the current jars. build dist doesn't work on tarball, one needs to check out from svn. I tried to use the github mirror since I can't remember a thing about svn other than it's broken, but build dist doesn't work and the patch fails. Reverted to svn, patch applies cleanly, build fails but after creating what we need. Checked in ef937e3af833af6f39d408a64a7cc2f0fda698d9. Apparently this only leads to a different error, still in job setup. Try again on avro 1.7.4 to align with cdh4.6, no improvement. Error here for future reference

54-86.compute-1.amazonaws.com:8021 failed on connection exception: java.net.ConnectException: Connection refused; For more details see:  http://wiki.apache.org/hadoop/ConnectionRefu
sed
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
        at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:782)
        at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:729)
        at org.apache.hadoop.ipc.Client.call(Client.java:1242)
        at org.apache.hadoop.ipc.WritableRpcEngine$Invoker.invoke(WritableRpcEngine.java:225)
        at org.apache.hadoop.mapred.$Proxy9.getBuildVersion(Unknown Source)
        at org.apache.hadoop.mapred.TaskTracker.offerService(TaskTracker.java:1958)
        at org.apache.hadoop.mapred.TaskTracker.run(TaskTracker.java:2875)
        at org.apache.hadoop.mapred.TaskTracker.main(TaskTracker.java:4125)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
        at sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:567)
        at org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:207)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:528)
        at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:492)
        at org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:510)
        at org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:604)
        at org.apache.hadoop.ipc.Client$Connection.access$2100(Client.java:252)
        at org.apache.hadoop.ipc.Client.getConnection(Client.java:1291)
        at org.apache.hadoop.ipc.Client.call(Client.java:1209)
        ... 5 more

[piccolbo]

Decided to comment in the docs and catch the error, see #4 and move on. Avro 1.7.7 will resolve the issue and it looks like bundling pre-compiled jars is a hornet's nest