Metric format without cluster and host is not supported

[jeeveeyes]

Hi,

I am eagerly trying to get Pencil up and running. Thanks for the good work.

I tried both the original Pencil (0.3.1) as well as the "rewrite" (0.4.0a). The problem I am facing is to get Pencil display existing metrics which are not in the format that Pencil expects i.e they don't have .. format.

Lets say I wanted to display the memUsage stock metric that is exposed by Graphite's carbon instance out-of-the-box. Its format is carbon.agents..memUsage.

How do I configure metric_format to correctly fetch and render the above metric ?

Inside pencil.yml , I changed

:metric_format: "%m.%c.%h" #%m metric, %c cluster, %h host

to

:metric_format: "%m" #%m metric

But that throws up this error when I start Pencil:

pencil

I, [2012-11-23T18:37:14.786240 #20554] INFO -- : no :manifest key for webapp, using default
missing host (%h) in :metric_format

Please see #9 (comment)

As noted there, %c (and %h too) needs to be made optional.
It'd be great if Pencil does not enforce any metric format so that existing metrics can readily imported.
Not sure how various features of Pencil are coupled with the %c,%h assumption though.

But right now, the %h is holding me from using Pencil !

[whd]

Hi,

Pencil requires %h because that is what in principle all views are designed to be aggregating on (global, cluster, host). The idea is that you have many hosts reporting the same metrics with only their hostname and possibly clustername to differentiate them. Pencil stands out among graphite dashboards for its ability to generate all these different views, but at the price of requiring %h in each graphite metric. In the nearly finished rewrite branch %c has been made fully optional (fixing #9) but %h remains, because as stated above it's rather integral Pencil's design.

However, you raise a valid point that there may be metrics desirable to graph that don't necessarily correspond to a particular host. The only one I can think of is graphing meta-graphite metrics like you mentioned. I can add support for this by adding a metric format override or similar option per graph (an extension of the DSL) for such graphs. But again, by doing this for all your graphs you lose most of the value of Pencil over other dashboards.

[jeeveeyes]

Hello,

Thanks for the reply and clarification of the design rationale.
Maybe we can fully leverage Pencil only when we have hosts in the metric name as you pointed out , but perhaps that can be mentioned in the readme notes upfront.

I agree that in production systems, some form of tacking on the host to the metric might be prevalent. As for me, when I wanted to quickly check out what Pencil has to offer on a dummy non-production setup, the requirement of host in the metric took me by surprise.

I don't know the level of effort involved, but if we can support arbitrarily named metrics ( maybe at the cost of not showcasing Pencil's unique features ) that will allow folks to get started with Pencil and migrate at their pace if possible.

[whd]

Thanks for the feedback. I'll update the docs to make it more explicit, and I'll add support for arbitrary metrics (though having no %h in the metric format will cause a warning). I'll reopen this until that feature has been added.

Currently if you want to try pencil you can use the data generation script in examples/test_data/emit_test_data. I'm planning on spinning up an example instance somewhere so people can check out pencil without having to set up a graphite instance.

[jeeveeyes]

Thanks, looking forward to these changes...