Design protocol for server components to use to communicate

[huin]

This design work should naturally happen before anything else in this milestone (even if not in detail). Work on this should help make clear how the multiple servers architecture will look.

[huin]

I'm thinking of using protobufs for this, but am open to ideas on this.

Protobufs are good from my perspective as I don't know that we'll be using synchronous-style RPC calls, and protobufs do not enforce them. They are wire-efficient and allow for versioning.

[arsatiki]

Can you outline some characteristics of the communication? I'd avoid protobufs or any other non-textual format, if performance is not the #1 priority. (And if performance is #0 priority, then custom format maybe preferable!)

Here's some quick questions that can clarify the situation.

• What sort of data are you transmitting? Commands only, simple data, complex data?
• How large messages are we talking about?
• How much traffic (in messages / second) are you expecting?
• Do you need acknowledgements or any other fancy flow control stuff? Is the communication more like TCP or more like UDP?
• Do you need to talk to outside parties or is everything internal?

It may even be that you have two or more differing types of communication going on. Those may warrant a different kind of solution.

[huin]

Sorry for the slow reply - have been away from Internet access for 3 days.

Protobuf's primary format is binary, see: http://code.google.com/apis/protocolbuffers/docs/encoding.html - but agreed - textual isn't good for various performance reasons.

I wouldn't claim performance is the top priority, although it's certainly high. I'd put some level of reliability and simplicity quite high as well.

What sort of data are you transmitting? Commands only, simple data, complex data?

• Depends on definitions. Effectively "commands" so far, I guess I'd say fairly simple in content - but that would be part of the design goals to figure out a lot of this stuff.

How large messages are we talking about?

• For the most part I imagine fairly small - say on the order of 20 bytes as a wild guess. Some will contain variable payloads up to 4KiB and more for chunk data.

How much traffic (in messages / second) are you expecting?

• I don't have a good estimate on this yet, and it would have to be quantified in terms of how many users are present, how much activity is going on and being relayed between end points, etc.

Do you need acknowledgements or any other fancy flow control stuff? Is the communication more like TCP or more like UDP?

• I'm going to say "no UDP" for initial implementations. I've seen the result of using UDP prematurely - UDP is an optimization, and those can easily be premature. Additionally, I imagine this sort of communication to mostly take place over very local networks where TCP latencies are not particularly high. There's need of ordering as well. UDP can be a "someday maybe" for anything that turns out to need that extra speed.

Do you need to talk to outside parties or is everything internal?

• Internal, for the purposes I see at the moment.

[huin]

Tsk, sorry, misread your comment about the textual bit - thought you were recommending binary formats and generally got mixed up, many apologies (that's what happens when I get home late after travel and try to do stuff!)

In general I'm in favour of text formats, but I believe that they will add a lot of overhead here (yes, premature optimization, etc.). Whilst text formats are easier to read, human-wise, there are several drawbacks of them as well:

• You need to write complex-ish parsers and formatters. That's a lot of complication I would like to avoid.
• Text formats take more space over the wire, this consumes more time on the network interfaces.
• Text formats take more CPU time. Whereas binary formats tend to be simple conversions in worst cases, and can declare their "packet size" up front.

I'm vaguely thinking of logging data that goes over the wire and providing a utility to parse it out and display it in human-readable form if it turns out to be useful to be able to read these streams. This should recover many of the benefits of a text format. Additionally, if the network serialization layer is done well, it might be possible to easily replace the relevant code with text formats anyway.

[arsatiki]

Righty-o! It seems you have pretty much the same views on the topic as I do. The only point where I disagree is the amount of work needed for a textual protocol. It may require work in the beginning but saves a ton of work during debugging. Being able to netcat or telnet or wget to a server relieves more distributed system headaches than aspirin.

With a binary format, you'd probably have to roll out your own debugging tools at some point. If you don't release them, other developers will have to do the same. If you do, then that's more maintenance load for you. Damned if you do, damned if you don't.

But. YMMV and you might actually like the scenario outlined above :-)

As you said, if the network serialization layer is done well, the wire format shouldn't matter (that much).

Anyway. Prototypes rock. What would be the simplest prototype that would allow some sort of design exploration? I'm thinking something like a one-way communication from master server to a slave. The slave would be used to see that the state is transferred in a sane fashion. But there may be simpler alternatives yet...

[huin]

Yes, there's truth there in how much work there is. Textual protocols are less work to debug, but binary protocols are (at least, I believe) easier to write (with the other performance benefits listed). But anyway, the purpose of this issue is to foster chat about this stuff, so this is all useful discussion.

Actually I've just done a bit of looking, and protobufs have a text format as well that appears to have Go support. So that might well be a point in their favour that would allow switching between text for debugging, and binary for actual performance usage without writing (much) code to support both. (see http://code.google.com/p/goprotobuf/source/browse/proto/text_parser.go )

The current message model used inside chunkymonkey is somewhat peer-to-peer one-way. Messages are requests that do not require replies, although in several cases requests might have requests sent back in response - but the model is relatively stateless in itself. I wouldn't call things master/slave - as that would imply replicas of state. The model is more a case of different types of peers making requests on each other (e.g chunks talking to other chunks to transfer entities across as they move), or player frontends talking to chunks to perform interactions (digging, building, chest inventory interaction, etc.). This currently takes the form of passing values over channels, which I imagine will translate somewhat well to TCP connections (channels share the characteristics of being ordered, reliable and not having limits on the size of the items being passed).

[uriel]

I prefer textual formats (custom or JSON), but have you considered using gobs?

http://blog.golang.org/2011/03/gobs-of-data.html

You might also find useful the rpc package (which uses gobs): http://golang.org/pkg/rpc/

[huin]

Hadn't considered them that much, I must admit. I briefly played with netchan which uses them, although I think netchan might be a bit immature (at least - I ran into some problems with it).

RPC - maybe. I don't know if I want a request-reply model, I've currently been working with requests without replies. It's a possibility, though.