Export heron-instances' streams-aggregated metrics

[maosongfu]

Currently heron-instances exports most of metrics scoped by different streams.

But a lot of customers expressed their interest on the value of all streams aggregated and for now they have to sum them up by themselves.
This pull request exports metrics aggregated on different streams too.
Tested with LocalScheduler.

[nlu90]

can we put this constant string into a separate final class like MetricsConstants?

Making a variable public in interface or abstract class smells dangerous.

[maosongfu]

One the one hand, it seems we don't have such Constants class; I don't see how necessary to add a class for a single filed;
on the other hand, this class is designed to abstract out the shared actions/implementations, which can be a good place to put a shared field.

BTW, I would prefer this class being an abstract class to an interface, since IMO this class is more used to reduce duplicated code, rather than to define some required behaviors.

[nlu90]

Making a field public while allowing the class to be extended is in general a bad design.

For the problem why it's an interface, @billonahill may have a better idea.

To land this metric fast, I'll approve it for now.

[maosongfu]

No rush for this pull request. Will also wait for Bill's comments.

[ashvina]

@maosongfu Could you please elaborate on how the customers use the aggregated metrics. Also graphite and other tools provide easy mechanism to aggregate metrics. Do we really need to worry about the intent inside the code? In my opinion metrics aggregation is better handled by tools.

[maosongfu]

@ashvina Sure. It is desired to handle by the tools. But:

1. Sometimes it is not easy or efficient for tools to do the aggregation. For example, at Twitter, we leverage a lot on wildcard to do pattern matching, and it does not work well on aggregation if there are metrics missing. Also, with wildcard pattern matching, the aggregation is inefficient.
2. For some metrics, people would care more on the value aggregated on all streams. For instance, "execute-ns", which indicates time spending on execute(). It is good to provide those metrics.

[ashvina]

Thanks @maosongfu
With the proposed change, every heron instance will be doing extra work for every tuple. Is this overhead less than the wildcard aggregate inefficiency? What if the aggregation takes place at container or topology level. For instance, the metrics cache could aggregate and push this information to sinks.

[maosongfu]

@ashvina From overhead point of view, doing aggregation on heron-instance side will be cheapest , most scalable and easy to implement: it will be an in-memory java hashmap get() and put(); also the aggregation will be done distributedly in different heron-instances.

What if the aggregation takes place at container or topology level.

Currently the common practice is that metrics-sinks in metricsmgr can handle the metrics in their own ways. For instance, scribe-sink/graphite-sink will send those metrics to Scribe/Graphite directly. It is unlikely to perform the aggregation at topology level.
If to do the aggregation at container level, i.e. at metrics-mgr, the implementation can be hard and tricky: it need to specify the metrics to aggregate, and the aggregation is on streams rather than time-series.