HInputStream should allow a user-supplied read function

[puellavulnerata]

(per private conversation with mlp, I understand there is an intent to add a capability to read from a file/network connection/etc.; this should be understood in that context as a note on flexibility)

Currently, HInputStream keeps a pointer to an in-memory buffer. It would be better if it could read data as needed into a sliding window / discard as no longer needed at the other end.

• Additional complication: HInputStream also has a length field; with this approach we may not know the length in advance

When this is implemented, I recommend doing it with a function pointer to a "read n more bytes from the source" function, and wrappers to create a new input stream around a {file descriptor|network connection|etc.}. This way, it will turn out to be flexible enough to accomodate unexpected sources (e.g. connect over SOCKS proxy, need to do protocol first, tunnel inside SSL, need to call SSL_read(), etc.)

See the implementation of the BIO mechanism in OpenSSL for a (somewhat overcomplex) example of this approach, and https://wiki.openssl.org/index.php/Manual:BIO_s_mem%283%29 for its flexibility.

• Additional consideration here: async I/O?

[pesco]

Have a look at the existing support for incremental parsing via h_parse_start, h_parse_chunk, h_parse_finish! We added and used this for the DNP3 proxy.

[nil-ableton]

A nice write up on io buffers for layering (say when you have stream sitting on top of a compressed stream, or some other structure)

https://fgiesen.wordpress.com/2011/11/21/buffer-centric-io/

[pesco]

nil-ableton on Thu, Mar 30 2017:

https://fgiesen.wordpress.com/2011/11/21/buffer-centric-io/

Nice. This is pretty close to Iteratees in that buffers are owned by the stream producer (enumerator). It stops short of the concept that the central data structure is a representation of the consumer (that would be our HParser), rather than the stream. Indeed, Iteratee I/O was the model for the "chunked" parsing support I mentioned above. :)