curl: (52) Empty reply from server (MLR model training)

[vedrin]

Hi,

I've set up another experiment, involving runing 5 parallel MLR model training requests (all using the same dataset/feature URIs) and have noticed that in some cases curl returns code 52 ( Empty reply from server). Curl's man page guives some more details for the reported error code -- "The server didn't reply anything, which here is considered an error". Please note that this occurs after only aprroximately 30 seconds, while in other cases model training succeeds after a longer period of time, so it doesn't seem to be due to some kind of timeout. I'm not sure what the real issue is in this case, however it seems that we have some kind of a scalability problem either with the dataset, feature or model services. In addition, such setup also helps reproducing the Exception Details: java.lang.IllegalArgumentException: Can't delete class attribute, tracked as a separate issue. Here is the curl log:

ambit:~# time curl -iv -X POST -d 'dataset_uri=http://ambit.uni-plovdiv.bg:8080/ambit2/dataset/6?max=522&target=http://ambit.uni-plovdiv.bg:8080/ambit2/feature/11954' http://opentox.ntua.gr:3000/algorithm/mlr

• About to connect() to opentox.ntua.gr port 3000 (#0)
• Trying 147.102.82.32... connected
• Connected to opentox.ntua.gr (147.102.82.32) port 3000 (#0)

  POST /algorithm/mlr HTTP/1.1
  User-Agent: curl/7.18.2 (x86_64-pc-linux-gnu) libcurl/7.18.2 OpenSSL/0.9.8g zlib/1.2.3.3 libidn/1.8 libssh2/0.18
  Host: opentox.ntua.gr:3000
  Accept: /
  Content-Length: 130
  Content-Type: application/x-www-form-urlencoded

• Empty reply from server
• Connection #0 to host opentox.ntua.gr left intact
  curl: (52) Empty reply from server
• Closing connection #0

real 0m30.232s
user 0m0.004s
sys 0m0.004s

[alphaville]

Weird indeed! Of course scalability is an important issue but I think we first have to tackle http://github.com/sopasakis/yaqp/issues#issue/21 . java.lang.IllegalArgumentException is a bug of the RDF parsing method we use. Could this be a result of packet loss or other communication problems? Does this happen when you apply a GET request? Could you please try the same using the following curl command:

curl -v -X POST -d 'dataset_uri=http://opentox.ntua.gr/ds.rdf&target=http://sth.com/feature/1' http://opentox.ntua.gr:3000/algorithm/mlr

[vedrin]

Hi,

I'm copying here (for your reference) my findings, that I just posted to the OT-Dev list as well:

I've just finished a more thorough experiment, which involved running 10 concurrent instances of the following bash script:

#!/bin/bash

for i in seq 1 1000;

do
curl -i -X POST -d 'dataset_uri=http://opentox.ntua.gr/ds.rdf&target=http://sth.com/feature/1' http://opentox.ntua.gr:3000/algorithm/mlr >> curl.log
done

Each of the script instances should have created 1000 models, so the expected grand total is 10000. The total running time was approximately 2329 seconds (between Thu, 31 Dec 2009 17:51:22 GMT and Thu, 31 Dec 2009 18:30:11 GMT).

The results are as follows:

Response code Number of instances

HTTP/1.1 200 9915
HTTP/1.1 400 5
HTTP/1.1 500 1
Empty response 79

The associated error reports for HTTP/1.1 400 are like the one below:

HTTP/1.1 400 The request could not be understood by the server due to malformed syntax
Content-Type: text/plain
Date: Thu, 31 Dec 2009 17:58:49 GMT
Accept-Ranges: bytes
Server: Noelios-Restlet/2.0m3
Connection: close
Transfer-Encoding: chunked

Error Report.
TimeStamp: Thu Dec 31 12:58:49 EST 2009

Error #1
Exception Details: java.lang.NullPointerException
Explanation: Probably this exception is thrown because the dataset or target uri you provided is not valid or some other internal server error happened! Please verify that the target uri you specified is an attribute of the dataset and is not of type 'string'!
For debugging reasons we provide a brief list of the exceptions:

• org.opentox.algorithm.trainer.MlrTrainer.train(MlrTrainer.java:81)
• org.opentox.resource.Algorithm.post(Algorithm.java:143)
• org.restlet.resource.ServerResource.doHandle(ServerResource.java:340)
• org.restlet.resource.ServerResource.doNegotiatedHandle(ServerResource.java:592)
• org.restlet.resource.ServerResource.doConditionalHandle(ServerResource.java:260)

The associated error report for HTTP/1.1 500 is:

HTTP/1.1 500 The server encountered an unexpected condition which prevented it from fulfilling the request
Content-Type: text/plain
Date: Thu, 31 Dec 2009 18:15:34 GMT
Accept-Ranges: bytes
Server: Noelios-Restlet/2.0m3
Connection: close
Transfer-Encoding: chunked

Error Report.
TimeStamp: Thu Dec 31 13:15:34 EST 2009

Error #1
Exception Details: com.hp.hpl.jena.shared.PropertyNotFoundException: http://www.opentox.org/api/1.1#value
Explanation: Severe Error while trying to build an MLR model.
For debugging reasons we provide a brief list of the exceptions:

• com.hp.hpl.jena.rdf.model.impl.ModelCom.getRequiredProperty(ModelCom.java:992)
• com.hp.hpl.jena.rdf.model.impl.ResourceImpl.getRequiredProperty(ResourceImpl.java:145)
• com.hp.hpl.jena.rdf.model.impl.StatementImpl.getProperty(StatementImpl.java:89)
• org.opentox.rdf.Dataset.getWekaDatasetForTraining(Dataset.java:295)
• org.opentox.algorithm.trainer.MlrTrainer.train(MlrTrainer.java:80)

In summary, three different problems are likely to exist. They seem to be exhibited more often when the model service is under a bit heavier load. These problems seem to be independent from the dataset and feature services in use (in this particular case the dataset and features were local for the service). The issues might have something in common, however I can only guess what the real issue is.

Hope this helps!

[alphaville]

Hi Vedrin,
Looks like some concurrency issue related either to Jena or/and Restlet. Maybe I should declare some methods as synchronized and try again. What is your view on that issue?

[vedrin]

Hi Pantelis,
Well, as I already mentioned yesterday, the problem might have something to do with the following Restlet bug, which has been fixed in 2.0 M5:
http://restlet.tigris.org/issues/show_bug.cgi?id=889
(Extracting POST parameters fails sporadically)
As you know, upgrading to 2.0 M6 has fixed a somehow similar issue with feature POSTing. However it happens that this upgrade is not straightforward and there a number of things to consider and code changes to be made. The good news is that most (or even all) of them are already known (by Nina's experience), so it shouldn't be too difficult to implement (with her help if/when needed). I think that this might be a good approach to follow, because anyways you'll have certainly to implement this upgrade sometime and in this case it might indeed solve the 3 issues being reported with a single shot. After the upgrade we could run the tests again and check whether everything is ok. If it is ok indeed, that would be wonderful, otherwise we would at least know that the problem is elsewhere and have the latest restlet version running, which is a good thing :-)
There's still one more detail. According to:
http://restlet.tigris.org/ds/viewMessage.do?dsForumId=4447&dsMessageId=2391019
this particular problem is inherent to Restlet+Tomcat but is not exhibited with Restlet+Jetty. On the other hand, the solution has something to do with the Java garbage collector, so it might be applicable not only for the Tomcat case. Which application server do you use?

[alphaville]

I run the server as a standalone application using the Grizzly HTTP connector. Did you encounter any problem using the new version of Restlet? I will give it a try anyway and hope that these bugs will be fixed (At least the one with the local dataset which is not bug of the parser)!

[vedrin]

The Restlet API has been changed and there were a number of things to fix in our code to get it running with 2.0 M6. One of the less obvious fixes was the one Nina mentioned in a mail to the dev list a few days ago:

For those who consider upgrading to 2.0-M6, please note Restlet team has
changed the default mode for matching routes; in order to use the
default behaviour from previous releases, one need to set
"DefaultMatchingMode" and "RoutingMode" as below for all Routers,
otherwise everything might appear broken :
setDefaultMatchingMode(Template.MODE_STARTS_WITH);
setRoutingMode(Router.MODE_BEST_MATCH);

There were also other things to fix, however they might have been specific to our implementation. In any case, if you decide to go that way, don't hesitate to ask questions -- Nina would be happy to help.