added SeqApp code by DommarajuUmamaheswari · Pull Request #1 · kmdlogic/seq-app-prometheus-pushgateway

DommarajuUmamaheswari · 2019-08-07T09:13:02Z

Created new SeqApp to send data to Pushgateway
Added files PushgatewayCounterData.cs, SeqAppToPrometheusPushGateway.cs
Based on the signals we select for the events Seq will send those events to SeqApp.
SeqApp will send Prometheus counters for those events to the PushgatewayUrl

adamchester · 2019-08-16T23:30:07Z

+            customGauge.Labels(pushgatewayCounterData.ResourceName).Set(gaugeValue);
+        }
+
+        public static PushgatewayCounterData FormatTemplate(Event<LogEventData> evt, List<string> applicationNameKeyList)


Why is this method called ‘FormatTemplate’? It doesn’t appear to be doing that...?

Ya.. We have changed it to "ApplicationNameKeyValueMapping" now.

adamchester · 2019-08-16T23:31:09Z

+
+        public void On(Event<LogEventData> evt)
+        {
+            var additionalPropertiesList = SplitOnNewLine(this.ApplicationNameKeySet).ToList();


Why are these called ‘additionalPropertiesList’?

we can rename it to "applicationNameKeySet".

adamchester · 2019-08-16T23:33:40Z

+
+
+            var customGauge = Metrics.WithCustomRegistry(registry).CreateGauge(GaugeName, "To track the seq errors", new[] { "ApplicationName" });
+            var gaugeValue = DateTimeOffset.UtcNow.ToUnixTimeSeconds();


Why would the time of processing be appropriate here? Wouldn’t it better to use a date and time that is part of the actual event, so that we aren’t confused by delays between when the event was created originally and when it was eventually processed into Prometheus?

There is a version of Set method which takes timestamp but it never works. So how we do it is we enable the timestamp on the gauge and then set it's value later on with the set method. But when we use this extension method we never get the gauge in the pushgateway.

Also we observed that there are default events in the pushgateway they all follow the same format which is unix time milli seconds.. Not even a single event with date time stamp. So we assumed datetime might not work here.

The question here is about what date and time to use as the starting point, prior to formatting it for the gauge.

Why is DateTimeOffset.UtcNow the most appropriate starting point?

We did have a chat about this, whether we use the seq timestamp as the metric we send rather than the utcnow time. I would expect there to be very little difference between timestamp and utcnow, so either or is ok I guess. It probably makes more sense to go with the seq timestamp - although this could be added as a label on the metric thats pushed as well.

Yes. We are pushing the Seq Event Timestampinutc as another label in the gauge now.

I'm still struggling to understand why DateTimeOffset.UtcNow is a good value to use here (I also struggle to understand how it's a sensible value generally for any Guage). Sorry if I'm just being slow, it's totally possible I've completely misunderstood something, but I think this is really worth clarifying here.

I think we need to understand the end-to-end use case you're attempting to cover here a bit better. You could start by adding that to the README :) GIve a real-world example of the event you;'d expect to see in Seq, and the corresponding way it would be handled by this Seq app, and finally how it would used on the prometheus side. Would it be graphed and alerted on? What would they look like?

However, while we're here, let's see if I can convey my confusion a little bit better. Let's start with the official documentation from Prometheus about a Guage:

Gauge
A gauge is a metric that represents a single numerical value that can arbitrarily go up and down.

Gauges are typically used for measured values like temperatures or current memory usage, but also "counts" that can go up and down, like the number of concurrent requests.

I understand we could use a guage to "measure" healthy=0, unhealthy=1, and then we can look for any non-zero values and determine what is unhealthy. That makes sense to me.

However, why would we create a guage that always goes up (because generally time always goes forward, the number will only increase)? Why would the guage only go up in response to a particular event in seq? What are we measuring here?

What would make sense to me, is using a Counter that is incremented every time a certain event is found in Seq.

This is a limitation with pushgateway. We need to compare with a timeframe so we can create alerts etc. Prometheus isn't aware of when metrics are loaded into the pushgateway, so itll just scrape the same metric over and over again for each scrape iteration. You can image then that a single message (say like the current notification emails we have configured in seq prod) gets pushed to pushgateway, how do we know its old? If we include the time in the labels, then a counter is redundant anyhow. Every new message will create a time series and hence start a new counter.

We could perhaps investigate using the metric in combination with the last_pushed_time_seconds as per below, then move back to using a counter?

Push_time_seconds

adamchester · 2019-08-16T23:34:25Z

+            var pushgatewayCounterData = FormatTemplate(evt, additionalPropertiesList);
+
+
+            var customGauge = Metrics.WithCustomRegistry(registry).CreateGauge(GaugeName, "To track the seq errors", new[] { "ApplicationName" });


Why is “ApplicationName” here?

“To track the Seq errors” is a little confusing to me. What’s this description used for, and why is it only errors? Couldn’t this Seq app also send any events (not just errors) to Prometheus ?

Got it. Will be renaming this to "To track the Seq events based on the applied signal".

ApplicationName is to track track events by Application in the seq.. If not ApplicationName then what else this could be?

we have made it as an input from UI now.

I still don’t understand how ApplicationName is being used, could you explain further please?

Ok I got it now, it’s configuration now, as the Guatemalan name.

adamchester · 2019-08-16T23:41:52Z

+                {
+                    if (property.Key == name)
+                    {
+                        data.ResourceName = property.Value.ToString();


Why do we have potentially multiple properties, but then we always overwrite whatever value was previously collected? It looks like we only keep a single value here, so why have the list?

The reason for this logic here is in our seq logs we have lot of events which contains Application name to which the events belongs with the Key as Application as well as App. Now these two we have noticed till now. Not sure in how many more such keys are there cause if you see in this code we are defaulting the resource name as "ResourceNotFOund" and in pushgateway sometimes we get this as well.

adamchester · 2019-08-19T09:11:38Z

+
+        [SeqAppSetting(
+            DisplayName = "ApplicationNameKeyList",
+            HelpText = "The names of additional event properties.",


This description seems unclear, I could not guess what values might be required simply by reading this.

This we have changed it a while back

adamchester · 2019-08-19T09:12:34Z

+        public string ApplicationNameKeySet { get; set; }
+
+        public IMetricPushServer server;
+        public readonly string instanceName = "default";


Why bother making this an instance member variable? Wouldn’t a constant be clearer?

Have removed the instanceName

adamchester · 2019-08-19T09:17:06Z

+namespace Seq.App.Prometheus.Pushgateway
+{
+    [SeqApp("Seq.App.Prometheus.Pushgateway",
+       Description = "Filtered events are sent to the Prometheus Pushgateway.")]


It doesn’t seem like the events are being sent to Prometheus push gateway. It seems like a counter is being sent each time the event is processed. Have I misunderstood something?

We have changed the description as we are "GaugeLabelKey data from filtered events are sent through gauge to the Pushgateway."

Howage · 2019-08-08T03:52:45Z

+            data.RenderedMessage = evt.Data.RenderedMessage ?? evt.Data.MessageTemplate;
+            foreach (var property in properties)
+            {
+                if (property.Key == "Application" || property.Key == "App")


Should allow this to be customizable as part of configuration - This is too specific

This is not there now. It is outdated. We have already made it configurable.

Howage · 2019-08-20T03:12:03Z

+
+
+            var customGauge = Metrics.WithCustomRegistry(registry).CreateGauge(GaugeName, "To track the seq errors", new[] { "ApplicationName" });
+            var gaugeValue = DateTimeOffset.UtcNow.ToUnixTimeSeconds();


We did have a chat about this, whether we use the seq timestamp as the metric we send rather than the utcnow time. I would expect there to be very little difference between timestamp and utcnow, so either or is ok I guess. It probably makes more sense to go with the seq timestamp - although this could be added as a label on the metric thats pushed as well.

Howage · 2019-08-20T03:13:51Z

+        public void On(Event<LogEventData> evt)
+        {
+            var gaugeLabelValuesList = SplitOnNewLine(this.GaugeLabelValues).ToList();
+            var pushgatewayCounterData = ApplicationNameKeyValueMapping(evt, gaugeLabelValuesList);


Not really a counter anymore right? Perhaps this needs a rename

THis is also fixed now.

adamchester · 2019-08-26T01:05:37Z

+ * Fill the details `Title` 
+ * Check the box `Stream incoming events` to send events and select the signal
+ * Fill in the details `Pushgateway URL`, `Pushgateway Gauge Name`, `Pushgateway Gauge Label Key`, `Pushgateway Gague Label Values`
+ * click on `SAVE CHANGES`


Avoid listing the generic steps to configure a Seq app. Only list those things that are specific to this Seq app. Above there is a link to https://docs.getseq.net/docs/installing-seq-apps which is the official documentation and will be maintained well in the future, so we can just leverage that one.

DU00556591 added 3 commits August 7, 2019 14:28

added SeqApp code

5dfa5b0

updated readme

34643b5

changes done

a0d1e3a

Howage reviewed Aug 8, 2019

View reviewed changes

Comment thread src/Seq.App.Prometheus.Pushgateway/SeqAppToPrometheusPushGateway.cs Outdated

DU00556591 added 11 commits August 8, 2019 09:47

changes done

8de5099

added changes

e7db7df

changes done

f8cef9e

changes done

976ba88

changes done for readme

7d50807

changes done

86bdd21

changes done

0bd9562

changes done

682edff

changes done

5359aa1

changes done

0565b5e

changes done

1d1e6f8

adamchester reviewed Aug 16, 2019

View reviewed changes

DU00556591 added 6 commits August 19, 2019 09:26

changes done

fc13d8b

changes done

0434860

changes done

ca7447e

changes done

8b408c6

changes done

b3f1ab1

changes done

eb48f61

adamchester reviewed Aug 19, 2019

View reviewed changes

DU00556591 added 3 commits August 19, 2019 15:43

changes done

397d79e

changes done

7691efd

changes done

3f8047d

Howage reviewed Aug 21, 2019

View reviewed changes

DU00556591 added 2 commits August 21, 2019 10:51

changes done

d60571c

changes done

8fcfcbe

adamchester suggested changes Aug 29, 2019

View reviewed changes

DU00556591 added 27 commits August 30, 2019 11:34

changes done

12c0d41

added counter

68f19b4

changes done

74e0083

changes done

18467aa

changes done

117f4dd

changes done

14b94ab

changes done

43a67ff

added changes

99ebffc

changes done

6a172b7

changes done

36c9f6d

changes done

78bc153

changes done added counter

17c5c2e

changes done

e4d50ce

changes done

7c0270c

changes done

2a0bcad

changes done

52a80f0

changes done

fb04779

changes done

2d2b4cd

changes done

d62e6cc

added counter reset

f7a21a7

added code for reset

12165fe

added changes

b9caa96

added changes

355be9c

changes done

c41add8

added changes

e792051

Added changes

b64332e

added changes

cc80dc6



		var customGauge = Metrics.WithCustomRegistry(registry).CreateGauge(GaugeName, "To track the seq errors", new[] { "ApplicationName" });
		var gaugeValue = DateTimeOffset.UtcNow.ToUnixTimeSeconds();

		var pushgatewayCounterData = FormatTemplate(evt, additionalPropertiesList);


		var customGauge = Metrics.WithCustomRegistry(registry).CreateGauge(GaugeName, "To track the seq errors", new[] { "ApplicationName" });

Conversation

DommarajuUmamaheswari commented Aug 7, 2019

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

amitxagarwal Aug 19, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

adamchester Aug 16, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

amitxagarwal Aug 19, 2019 •

edited

Loading

adamchester Aug 16, 2019 •

edited

Loading