Spark Tests unsuccesful

[saifellafi]

Hi,

Pulled from master. If I -DskipTests It compiles succesfully but I cannot run examples:
$ ./bin/run-example io.ddf.spark.examples.RowCount
Failed to find DDF examples assembly in /var/DDF/lib or /var/DDF/examples/target
You need to build DDF with mvn install before running this program

Running io.ddf.spark.SparkDDFManagerTests
org.apache.spark.serializer.KryoSerializer
null
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.647 sec - in io.ddf.spark.SparkDDFManagerTests

Results :

Tests in error:
RepresentationHandlerSuite.Can do sql queries after Transform Rserve » Spark J...
TransformationHandlerTest.testTransformNativeRserve » Spark Job aborted due to...

Tests run: 79, Failures: 0, Errors: 2, Skipped: 0

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 05:46 min
[INFO] Finished at: 2015-08-19T14:59:07-05:00
[INFO] Final Memory: 59M/3944M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.15:test (default-test) on project ddf_spark_2.10: There are test failures.
[ERROR]
[ERROR] Please refer to /var/DDF/spark/target/scala-2.10/surefire-reports for the individual test results.
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException

[saifellafi]

I believe the case is the firewall blocking my rServe start test.

mvn clean install

Being this the case, why isn't maven compiling DDF with -DskipTests?
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] ddf ................................................ SUCCESS [ 1.019 s]
[INFO] ddf_core ........................................... SUCCESS [ 27.010 s]
[INFO] ddf_jdbc ........................................... SUCCESS [ 2.645 s]
[INFO] ddf_spark .......................................... SUCCESS [ 26.850 s]
[INFO] ddf_examples ....................................... SUCCESS [ 4.675 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------

$ ./bin/run-example io.ddf.spark.examples.RowCount
Failed to find DDF examples assembly in /apps/cmor/DDF/lib or /apps/cmor/DDF/examples/target
You need to build DDF with mvn install before running this program

Very much appreciate the assistance

[Huandao0812]

@saifellafi Can you post the stack trace related to the RServe error?

[saifellafi]

@Huandao0812
Hi, can you assist me helping you, to do so? I am not much of a maven person.

I am not much worried about this test, but I am failing to understand why is it not compiling enough to run the examples with skip tests.
Let me know if the following helps:

Running io.ddf.spark.content.RepresentationHandlerSuite
rmr: DEPRECATED: Please use 'rm -r' instead.
Deleted file:///tmp/hive/warehouse/airline_delayed
StartRserve: first connect try failed with: Cannot connect: Connection refused
StartRserve: waiting for Rserve to start ... (java.lang.UNIXProcess@7b7f52b4)
StartRserve: Rserve>Error in library(Rserve) : there is no package called ‘Rserve’
StartRserve: Rserve>Execution halted
StartRserve: call terminated, let us try to connect ...
StartRserve: Try failed with: Cannot connect: Connection refused
StartRserve: Try failed with: Cannot connect: Connection refused
StartRserve: Try failed with: Cannot connect: Connection refused
StartRserve: Try failed with: Cannot connect: Connection refused
StartRserve: Try failed with: Cannot connect: Connection refused
StartRserve: waiting for Rserve to start ... (java.lang.UNIXProcess@45fe86d0)
StartRserve: Rserve>Error in library(Rserve) : there is no package called ‘Rserve’
StartRserve: Rserve>Execution halted
StartRserve: call terminated, let us try to connect ...
StartRserve: Try failed with: Cannot connect: Connection refused
StartRserve: Try failed with: Cannot connect: Connection refused
StartRserve: Try failed with: Cannot connect: Connection refused
StartRserve: Try failed with: Cannot connect: Connection refused
StartRserve: Try failed with: Cannot connect: Connection refused
Tests run: 9, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 19.585 sec <<< FAILURE! - in io.ddf.spark.content.RepresentationHandlerSuite
Can do sql queries after Transform Rserve(io.ddf.spark.content.RepresentationHandlerSuite) Time elapsed: 5.888 sec <<< ERROR!
org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 68.0 failed 1 times, most recent failure: Lost task 0.0 in stage 68.0 (TID 68, localhost): java.lang.RuntimeException: Unable to start Rserve
at io.ddf.spark.etl.TransformationHandler$$anonfun$4.apply(TransformationHandler.scala:139)
at io.ddf.spark.etl.TransformationHandler$$anonfun$4.apply(TransformationHandler.scala:136)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:328)
at scala.collection.Iterator$$anon$10.next(Iterator.scala:312)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103)
at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47)
at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:273)
at scala.collection.AbstractIterator.to(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:265)
at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1157)
at scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:252)
at scala.collection.AbstractIterator.toArray(Iterator.scala:1157)
at org.apache.spark.rdd.RDD$$anonfun$33.apply(RDD.scala:1177)
at org.apache.spark.rdd.RDD$$anonfun$33.apply(RDD.scala:1177)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1498)
at org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:1498)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:61)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Driver stacktrace:
at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1204)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1193)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1192)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1192)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:693)
at scala.Option.foreach(Option.scala:236)
at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:693)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1393)
at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:1354)
at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:48)