Tool for querying and viewing the event log

[krasserm]

From @magnusart on July 23, 2015 7:39

Based on the discussion on Gitter I'd like to discuss the design of a tool that would allow team members to query and view event logs for support issues, debugging and testing purposes.

Problem description

Event log payloads are not available directly by querying Cassandra or other storages since all event log payloads are serialized into binary blobs. Trying to make sense of these blobs without proper deserialization support becomes an exercise in frustration. With a hex editor you can discern some tidbits of information.

Open questions

Generic deserialization

Is there is a way to deserialize java serialization, proto buf or thrift into a human readable format without knowing anything about the data apart from the protocol before hand?

I'm not familiar enough with the formats to judge above, but I'm guessing it is not possible and that we need to have the data model available to deserialize with field labels visible.

Desktop or web app

I'm leaning towards a desktop application, since it is very convenient to not rely on a server to be up to be able to troubleshoot. But if we cannot solve generic deserialization then a server application would be the most obvious choice.

Copied from original issue: RBMHTechnology/eventuate#88

[krasserm]

From @magro on July 23, 2015 7:48

Sounds interesting!

Re Dekstop or web app: a third possibility would be a cli, that could be run on every server.

Just to mention it: perhaps spark could be helpful for the query / analysis part.

[krasserm]

From @magnusart on July 23, 2015 9:2

@magro I haven't played around with spark much yet. But I assume it is because you can send closures easily separate the serialization/deserialization from the tooling. As for using a cli, I agree. Perhaps that is a good place to start.

[krasserm]

From @matlockx on July 23, 2015 16:4

+1 for cli
Boosts acceptance imo

[krasserm]

Thank you guys for starting the discussion and your valuable input. Regardless whether we go for a desktop application, server application or CLI, I think the first step is having a read-only API that

• abstracts over data stores (Cassandra, LevelDB, ...)
• is system-schema-aware (DurableEvent)
• is application-schema-agnostic i.e. serves the DurableEvent payload as blob (byte[] or whatever).

Event log inspection tools should use this API and together with application-specific de-serializers and renderers to display payloads to the user. A de-serializer converts a blob into a domain event object and a renderer creates a human readable representation. The API should be accessible remotely and also locally within a server process.

The current abstraction over data stores are event log actors (LeveldbEventLog and CassandraEventLog) that both support the EventsourcingProtocol (for message exchange between event-sourced actors and the event log) and the ReplicationProtocol.

The main difference between the current abstraction and the proposed API is that the event log actors internally de-serialize payloads which is unnecessary overhead when being used remotely (which requires an additional serialization and de-serialization step). This is currently also the case for event replication. For this reason, I anyway planned to implement the proposed API which can then be additionally used by log inspection tools.

Assuming a multi-location Eventuate application, a log inspection tool (be it a standalone CLI or a desktop application) could then connect to all of these locations and use the API remotely for viewing event logs. If needed, we still could embed a CLI into a server process and use that API locally but I'm not sure if this is a benefit over remote usage.

If you think my proposal is reasonable, we could split up work into:

• Definition and implementation of the read-only API
• Refactoring of event log actors to use that API
• Re-implementation of event replication to use that API
• Implementation of a remote CLI client that is configurable with application-specific de-serializers and renderers or
• Implementation of a remote desktop application that is configurable with application-specific de-serializers and renderers as well.

In my opinion, the proposed API is the most basic way to get event data out of Eventuate data stores for further processing without having to make any assumptions on the underlying storage technology. I'm also thinking about writing a Spark Streaming custom receiver based on this API for further general-purpose event processing (and maybe make it part of akka-analytics).

WDYT?

[krasserm]

From @magnusart on July 25, 2015 9:11

Hi

Personally I like the suggestions made so far a lot.

However I do have a concern with having to distribute the application specific deserializers to all interested parties. For people not having direct access to git repositories and who typically do not have a scala compiler installed, continually distributing the new deserializers will become a hassle.

I miss a second option where someone helps a user (who do not have a compiler) to install a cli or desktop app and configure it once and then it just works.

I see two obvious options

Centralized server that takes care of deserialization

Two types of clients, one that accesses the serialized payload directly and one that client that communicates with a service that deserializes into a human readable format. This service would in turn be a client of the first type but deployed centrally to a server together with deserializers

Pros: Simple dumb client, less local config, easy to support and update deserializers in one central place
Cons: Have to manage yet another server, need maintain two clients, only one active configuration for everyone at a time

Central repository to download deserializers from

Just one client type that consumes serialized payloads, but deserializer packages are distributed via a central repository.

This model is well known and we could perhaps borrow code/concepts/syntax from SBT to have as few surprises as possible. It would also make it trivial for a developer to publish a local snapshot of the deserializers, update the config file and have the cli use that for their local machine.

Pros: No central infrastructure that needs to be running to read the log, well known model for developers
Cons: More complexity in the client, configuration that can be differentiate on many machines and cause increased support.

Is this a valid concern for you as well? I personally find the second option very compelling.

[krasserm]

@magnusart I agree with you and share your concerns. Good coverage of pros and cons (although the centralized server doesn't necessarily need to be a standalone server but could rather be embedded in the main server process).

It seems to me that there's less initial effort to go for the centralized server approach. If we require server applications to configure JSON renderers, for example, generic clients could be developed. But maybe its only my lack of experience of setting up and operating that separate repository that I think the second option is more effort (not only for initial development but also for maintainance). I personally find both options equally compelling at the moment.

[krasserm]

From @magro on July 26, 2015 6:40

If deserialization is not done by the server process itself deserialization might break due to a different configuration or the different environment. E.g. kryo can be configured to use asm or not, use unsafe or not etc. Therefore I find deserialization done by the server process itself a bit more robust.

[krasserm]

From @magro on July 26, 2015 6:54

On a second thought, in the case of downloaded deserializer configuration should be part of the deserializer (so this should be no more than a function). Environment however can be different, which also includes availability of required libraries.

[krasserm]

From @volkerstampa on July 26, 2015 7:26

I wonder if in case of the repository approach the first simple step would be to make the user drop the application-artifacts containing (de-)serialization code+configuration (that are most probably anyways published to an internal nexus repository) in a lib-folder of the viewer application.

With this a generic viewer application would still have to traverse the application specific event-classes, so something like a JSON-renderer probably makes sense in any case.

Just my 2c.

[krasserm]

From @magnusart on July 26, 2015 8:24

Hi

My thoughts were that the first option is less initial effort but means less flexibility. If it becomes part of the application server process then of course that is very convenient, but it would be the same scenario as with replication and the deserialize-serialize-deserialize flow. I'm fine with that but I thought that was something we were trying to avoid on the main server process by consuming serialized messages directly.

In the second option I envision that the configured deserialization packages are downloaded on client start via an embedded maven or ivy client and contains all code needed to convert payloads to human readable formats.

I assume most projects set up an artifact repository sooner or later, either maven or ivy style. If we can piggyback that infrastructure then the setup for the user is just publishing another artifact, either local or remote. The client would be aware of maven and ivy repositories, which in itself is not trivial even if there are some good libraries we can reuse.

The main difference I see is that this offloads the deserialization to the client. But if that is not important then I agree with you 100%, adding this to the main server process to be a part of eventuate (just another event view actor that exposes an api) feels much simpler and more robust.

In that actor a developer could then filter out or scramble sensitive information like password or social security numbers that should not be visible for support personnel for example.

[krasserm]

If the application doesn't need to satisfy high throughput requirements and log inspection doesn't occur frequently I think an embedded server is a good starting point. With the proposed API, a re-implementation of event replication would anyway avoid unnecessary (de)serialization cycles, so that only the log inspection server generates additional (de)serialization overhead.

Applications with higher throughput requirements still can go for a standalone server approach using the proposed API that should be accessible remotely as well. In any case, all options discussed so far can perfectly co-exist, so the main question is what are we starting with. If there are no objections, I'd say let's start with the embedded server approach. Thoughts?

[krasserm]

From @magnusart on July 26, 2015 11:41

I think you are right and I agree with the embedded approach. +1