Issue with integration of faunus with titan

[varunkalluri]

Hi Okram,
I have integrated faunus with hdfs and titan with hbase, But when we are trying to access data from titan into faunus it giving following error,
Titan Version:titan-0.4.4
Faunus Version: faunus-0.4.4

Here is my property file:
faunus.graph.input.format=com.thinkaurelius.faunus.formats.titan.hbase.TitanHBaseInputFormat
faunus.graph.input.titan.storage.backend=hbase
faunus.graph.input.titan.storage.hostname=xyz( I am specifying zookeeper hosts here)
faunus.graph.input.titan.storage.port=2181
faunus.graph.input.titan.storage.tablename=titan

faunus.graph.output.format=com.thinkaurelius.faunus.formats.titan.hbase.TitanHBaseOutputFormat
faunus.graph.output.titan.storage.backend=hbase
faunus.graph.output.titan.storage.hostname=xyz( I am specifying zookeeper hosts here)
faunus.graph.output.titan.storage.port=2181
faunus.graph.output.titan.storage.tablename=titan
faunus.graph.output.titan.storage.batch-loading=true
faunus.output.location=output1
zookeeper.znode.parent=/hbase-unsecure

titan.graph.output.ids.block-size=100000

faunus.graph.output.titan.infer-schema=true

Execution:
g = FaunusFactory.open('/opt/faunus/faunus-0.4.4/bin/titan-hbase-inputoutput.properties')
==>faunusgraph[titanhbaseinputformat->titanhbaseoutputformat]
gremlin> g.v(2640092)

13:24:20 WARN mapreduce.FaunusCompiler: Using the distribution Faunus job jar: ../lib/faunus-0.4.4-job.jar
13:24:21 INFO mapreduce.FaunusCompiler: Compiled to 3 MapReduce job(s)
13:24:21 INFO mapreduce.FaunusCompiler: Executing job 1 out of 3: MapSequence[com.thinkaurelius.faunus.mapreduce.transform.VertexMap.Map, com.thinkaurelius.faunus.formats.titan.SchemaInferencerMapReduce.Map, com.thinkaurelius.faunus.formats.titan.SchemaInferencerMapReduce.Reduce]
13:24:21 INFO mapreduce.FaunusCompiler: Job data location: output1/job-0
13:24:21 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
13:24:22 INFO diskstorage.Backend: Initiated backend operations thread pool of size 4
13:24:22 INFO zookeeper.ZooKeeper: Client environment:zookeeper.version=3.4.5-1392090, built on 09/30/2012 17:52 GMT
13:24:22 INFO zookeeper.ZooKeeper: Client environment:host.name=euca-192-168-216-107.eucalyptus.internal.devlab.dev
13:24:22 INFO zookeeper.ZooKeeper: Client environment:java.version=1.7.0_45
13:24:22 INFO zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation
13:24:22 INFO zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_45/jre

13:24:24 INFO mapred.JobClient: Running job: job_201407181049_0009
13:24:25 INFO mapred.JobClient: map 0% reduce 0%
13:30:04 INFO mapred.JobClient: Task Id : attempt_201407181049_0009_m_000000_0, Status : FAILED
com.thinkaurelius.titan.core.TitanException: Exception in Titan
at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getAdminInterface(HBaseStoreManager.java:380)
at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.ensureColumnFamilyExists(HBaseStoreManager.java:275)
at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.openDatabase(HBaseStoreManager.java:228)
at com.thinkaurelius.titan.diskstorage.Backend.getStore(Backend.java:231)
at com.thinkaurelius.titan.diskstorage.Backend.initialize(Backend.java:243)
at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.getBackend(GraphDatabaseConfiguration.java:1174)
at com.thinkaurelius.titan.graphdb.database.StandardTitanGraph.(StandardTitanGraph.java:75)
at com.thinkaurelius.faunus.formats.titan.FaunusTitanGraph.(FaunusTitanGraph.java:30)
at com.thinkaurelius.faunus.formats.titan.FaunusTitanGraph.(FaunusTitanGraph.java:26)
at com.thinkaurelius.faunus.formats.titan.hbase.FaunusTitanHBaseGraph.(FaunusTitanHBaseGraph.java:29)
at com.thinkaurelius.faunus.formats.titan.hbase.TitanHBaseInputFormat.setConf(TitanHBaseInputFormat.java:53)
at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:62)
at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:721)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:363)
at org.apache.hadoop.mapred.Child$4.run(Child.java:255)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1232)
at org.apache.hadoop.mapred.Child.main(Child.java:249)
Caused by: org.apache.hadoop.hbase.MasterNotRunningException: Retried 14 times
at org.apache.hadoop.hbase.client.HBaseAdmin.(HBaseAdmin.java:139)
at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getAdminInterface(HBaseStoreManager.java:378)
... 19 more
13:46:45 INFO mapred.JobClient: Job complete: job_201407181049_0009
13:46:45 INFO mapred.JobClient: Counters: 8
13:46:45 INFO mapred.JobClient: Job Counters
13:46:45 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=1334254
13:46:45 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13:46:45 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13:46:45 INFO mapred.JobClient: Rack-local map tasks=3
13:46:45 INFO mapred.JobClient: Launched map tasks=4
13:46:45 INFO mapred.JobClient: Data-local map tasks=1
13:46:45 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0
13:46:45 INFO mapred.JobClient: Failed map tasks=1
13:46:45 ERROR mapreduce.FaunusCompiler: Faunus job error -- remaining MapReduce jobs have been canceled

please let me know if I am missing any configutions(or) settings.

Thanks,
varun

[spmallette]

@dalaro @okram any ideas what this is? i don't do much with hbase/faunus so the errors look sorta foreign to me.

[dalaro]

Based on this recent aureliusgraphs list discussion, I wonder if the problem was related to customized ZK root node (/hbase-unsecure): https://groups.google.com/d/msg/aureliusgraphs/t1FJQ_EByYg/EB2Z4a6UWdsJ. It seems like you're already setting the zookeeper.znode.parent key that resolved the issue in that thread, so maybe there's some other complication here. The connection is faint if it exists at all. There's not enough information in the trace to definitively say why the HBase client in Titan failed to find the HMaster (assuming the master was running). Can you turn up the logging level on the failing tasks to DEBUG? It would be interesting to see what they emit before dying with that exception.

I'm closing this ticket since I expect that you've moved on since July, but please do reopen if you want to pursue this further.

[dalaro]

Per your colleague Mirco Manucci's recent comment related to this issue on http://stackoverflow.com/questions/24886501/issue-while-ingesting-a-titan-graph-into-faunus directing me back to this issue, I'm reopening to reply and to try to gather this discussion into a single thread. I don't know if the SO question and this issue are crosspost of an identical problem, or if you guys are doing different work and encountering related problems, or something else. But Mirco directed me here when I asked for a full stacktrace in SO comments, so I assume moving the discussion here is OK.

Mirco or Varun, if you can turn up the logging level on the dying MR tasks to DEBUG to see why it's failing to connect to the HBase master, that would be really helpful. Right now, all I can see is that the task running Titan-HBase can't talk to the master, and I can't say why without more information.

[varunkalluri]

Hi LaRocque,
Thanks for your response, Right now we are integrated titan-0.5.0 but gremlin shell is hanging with our hbase 0.94 version.
I see that titan has 0.98 hbase jar files, I tried to copy libs from existing cluster still doesn't work .

Configuration file.
storage.backend=hbase
storage.hostname=xxxxxxx,xxxxxxxx,xxxxxxxx
storage.port=2181
storage.short-cf-names=true
storage.tablename=titan123
cache.db-cache = true
cache.db-cache-clean-wait = 20
cache.db-cache-time = 180000
cache.db-cache-size = 0.5
storage.index.search.backend=elasticsearch
storage.index.search.directory=/tmp/searchindex
storage.index.search.client-only=false
storage.index.search.local-mode=true

Tue Sep 23 11:16:15 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true
Tue Sep 23 11:17:35 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true
Tue Sep 23 11:18:55 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true
Tue Sep 23 11:20:15 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true
Tue Sep 23 11:21:35 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true
Tue Sep 23 11:22:55 EDT 2014, org.apache.hadoop.hbase.client.RpcRetryingCaller@2c7e7394, java.io.IOException: Failed to find location, tableName=hbase:meta, row=titan,,, reload=true

11:22:55 WARN com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager - Unexpected exception during getDeployment()
java.lang.RuntimeException: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:346)
at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getFeatures(HBaseStoreManager.java:384)
at com.thinkaurelius.titan.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1209)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:92)
at com.thinkaurelius.titan.core.TitanFactory.open(TitanFactory.java:60)
at com.thinkaurelius.titan.core.TitanFactory$open.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at groovysh_evaluate.run(groovysh_evaluate:82)
at groovysh_evaluate$run.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at groovysh_evaluate$run.call(Unknown Source)
at org.codehaus.groovy.tools.shell.Interpreter.evaluate(Interpreter.groovy:67)
at org.codehaus.groovy.tools.shell.Interpreter$evaluate.call(Unknown Source)
at org.codehaus.groovy.tools.shell.Groovysh.execute(Groovysh.groovy:152)
at org.codehaus.groovy.tools.shell.Shell.leftShift(Shell.groovy:114)
at org.codehaus.groovy.tools.shell.Shell$leftShift$0.call(Unknown Source)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:42)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:108)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:116)
at org.codehaus.groovy.tools.shell.ShellRunner.work(ShellRunner.groovy:88)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$work(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.work(InteractiveShellRunner.groovy:100)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite$PogoCachedMethodSiteNoUnwrapNoCoerce.invoke(PogoMetaMethodSite.java:272)
at org.codehaus.groovy.runtime.callsite.PogoMetaMethodSite.callCurrent(PogoMetaMethodSite.java:52)
at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCallCurrent(CallSiteArray.java:46)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:133)
at org.codehaus.groovy.runtime.callsite.AbstractCallSite.callCurrent(AbstractCallSite.java:137)
at org.codehaus.groovy.tools.shell.ShellRunner.run(ShellRunner.groovy:57)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.super$2$run(InteractiveShellRunner.groovy)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:90)
at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:233)
at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1079)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuperN(ScriptBytecodeAdapter.java:128)
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.invokeMethodOnSuper0(ScriptBytecodeAdapter.java:148)
at org.codehaus.groovy.tools.shell.InteractiveShellRunner.run(InteractiveShellRunner.groovy:66)
at com.thinkaurelius.titan.hadoop.tinkerpop.gremlin.Console.(Console.java:61)
at com.thinkaurelius.titan.hadoop.tinkerpop.gremlin.Console.(Console.java:68)
at com.thinkaurelius.titan.hadoop.tinkerpop.gremlin.Console.main(Console.java:73)
Caused by: com.thinkaurelius.titan.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.ensureTableExists(HBaseStoreManager.java:748)
at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getLocalKeyPartition(HBaseStoreManager.java:545)
at com.thinkaurelius.titan.diskstorage.hbase.HBaseStoreManager.getDeployment(HBaseStoreManager.java:342)

[varunkalluri]

We are using CDH4.5 hadoop version.

[dalaro]

@varunkalluri I'm a bit confused. The opening comment in this issue was about Faunus 0.4. Your later comment includes a call to TitanFactory, which isn't part of Faunus; it also mentions Titan 0.5, but the config file appears to be written for Titan 0.4 (the index keys have changed between 0.4 and 0.5).

Let's pursue Faunus 0.4 bugs here, but please file bugs related to 0.5 (whether Titan or Faunus) at https://github.com/thinkaurelius/titan/issues. There's also the https://groups.google.com/d/forum/aureliusgraphs list in case you prefer a discussion format.

Thanks!