Is SimpleInterSSMQ Inefficient?

[muppsy007]

Sorry for the obscure title. This isn't really a bug, but more a question around the design of SimpleInterSSMQ class. And My usage my be wrong.

At present, SimpleInterSSMQ->send() simply creates a curl request and attempts to send it to the given url. I see, and have been experiencing two issues with this as our SS cluster gets to 8 servers.

1. The requests are not queued to begin with. MessageQueue::send() calls the MQ type send() method. If the type is SimpleDBMQ, it makes use of the database table in that class's own send() method. If the type is SimpleInterSSMQ, that class's send() method simply creates a CURL request, sends, and throws an exception if the response is not the literal string "ok". As the list of servers gets bigger, this slows the response that triggers it (which may be due to me implementing this incorrectly).
2. The part about these CURL requests simply throwing an Exception when they fail is the next issue. Since failures are not stored or retried, any failure, for any reason, is obviously never sent.

This might be an question of implementation at my end. It might be a design oversight. It might even be that supporting 8-10 installations was never something this was designed to work with. I just want to get any ideas around this and there really isn't any other place to discuss this module.

[mrmorphic]

Hi. It's a fairly simple design, and not designed for high volume (e.g. it's not designed for 100's of requests per second) - it's really designed as an easy and cheap way to send data or events from one SS server to another. You should be able to use it to coordinate activity between a number of servers in a cluster (I have used it for sync'ing a CMS with 9 front ends, which worked out well.)

From memory, you can configure a SimpleInterSSMQ queue to use another SimpleDBMQ queue as a buffer, which is in the docs. By configuring the buffer to execute on shutdown, the curl request(s) can be sent asynchronously from the initiating HTTP request so the user will not experience slowdown.

I'm not sure with the error handling, I'd need to see the details. I didn't have that experience - it mostly just worked. Although on multiple servers you may have to pay close attention to timings, depending on what you're doing.

M

[muppsy007]

Cheers for the quick response.

Not sending many requests here. We're talking 3-4 per minute for each of the 8 front ends. So 32 CURL requests a min (in blocks of 8 quick successions).

I wasn't aware SimpleInterSSMQ was able to use the DBMQ as a buffer. I missed it in the docs but I will revisit. That may solve one issue.

As for the error handling, I'm not sure using DBMQ as a buffer will fix this unless it catches exceptions and re-queues. The last lines of SimpleInterSSMQ->send() are:

    $output = curl_exec($ch);
    curl_close($ch);
    if ($output != "ok") throw new Exception("Failed to deliver to remote server, " . $output);
}

I'm getting 6-10 of these messages per day in the logs. Which isn't really a problem of the module. They fail for reasons I am yet to work out. The thing is when they do fail, the request is never retried.

Or am I misunderstanding what I see in SimpleDBMQ's code if I were to use that as a buffer. I don't see that catching exceptions and retrying either.

[mrmorphic]

That volume will be no problem.

Have you done:

SimpleInterSSMQAccept::setEnabled(true);

on the receiving end? By default, the controller that the wget is hitting (SimpleInterSSMQ_Accept) is disabled for security, so it has to be explicitly enabled, say in mysite/_config.php. Generally I would configure infrastructure to block requests to that controller from external / untrusted sources as well.

M