COUCHDB-769: Store attachments in the external storage.

[gilv]

Initial implementation that allows CouchDB to store attachments outside of the database file.
This implementation supports OpenStack Swift and SoftLayer Object store

[kxepal]

This option looks as boolean, but you use "external" value elsewhere in place of "true". If it pretend to be a flag, use config:get_boolean. Otherwise better pick less confusing name for default value.

[kxepal]

Sorry, was confused. So it's flag. Definitely use config:get_boolean and true/false atoms elsewhere.

[kxepal]

Please fix:

• Style: try to fit 80 chars line length, put space after comma, no tabs, correctly aligned clauses etc.
• Reduce amount of debug logs: debug logs should help you understand what going on in your code and in which state it is in places where problems may happens, not produce noise about every logical step.

[kxepal]

Why you store "external" as list, no at atom, or at least as binary?

[kxepal]

I also think that "external" value is not the best choice. Since field is already carries external word in the key, better specify which exact external we assume here. Today it's only swift, tomorrow it could be also S3/NFS/whatever which will have to use different integration logic.

[kxepal]

Would be nice to see some tests for this to prove that this all works.

[rnewson]

general note that we don't include attributions in the code.

[kxepal]

Oh, right. And we need ASF license header here.

[rnewson]

I'm not going to have time to review this in detail, but I agree it needs tests to go in, as well as considerable cleanup.

Noting here that this should not be merged before 2.0 is released whatever happens.

[gilv]

@kxepal @rnewson
I provided another patch, implementing most of the comments.
Here is what is not implemented in this patch:

1. Remark on the AttFun. Not yet implemented. Original remark: “AttFun could be actually an attachment data or some fold function? Something went wrong if you have to do this.”
2. usage of couch_util:to_hex
3. still no unitests

[kxepal]

I think this should be in swift backend module, at least. And certainly not couch_httpd (even if these two does the same thing) - just chttpd.

[gilv]

@kxepal . Thanks for review it.
fabric_at_handler is a layer between CouchDB and the backend store. It's not aware how backend store works or what it does internally. I completely agree with your other comments, for example "201" is something that belongs to the backend store and should be there.

Data = couch_httpd:recv(Req, ContentLen) is suppose to read data from the HTTP request, using the same mechanism as CouchDB works, so i use the same function to read data. Otherwise i will need to copy-paste the same code here.
Why is it different to function att_processor(DbName, Att)? That function also "aware" of CouchDB's internal functions, for example it uses couch_at module to update attachments.

[kxepal]

The reason why it's different is simple: couch_httpd/chttpd are frontend modules that provides HTTP API for internal stuff like couch_att.

The reason why couch_httpd better to avoid use here as that this module is for backdoor interface while chttpd that used for the front, cluster one, may eventually start work by different rules.

The reason why chttpd/couch_httpd shouldn't be here is that fabric is a layer to run cluster wide operations after http request get processed.

Also, have you checked fabric:att_receiver/2? Looks like exactly what you need here.