Event mgmt by spenugondaa · Pull Request #7813 · sonic-net/sonic-buildimage

spenugondaa · 2021-06-07T23:40:48Z

Why I did it

This PR contains code changes for providing Event and Alarm Framework as specified in the HLD

How I did it

Followed design specified in HLD.

A new container - eventd - is added to provide the framework. (buildimage/src/sonic-eventd/)
Applications need to link libeventnotify library to raise events or alarms. (buildimage/src/sonic-eventd/lib/)
API is defined at buildimage/src/sonic-eventd/lib/include/eventnotify.h
A new DB - EVENT_DB - to store event/alarm/event-stats/alarm-stats tables.
Default event profile is maintained at buildimage/src/sonic-eventd/var/evprofile and gets installed on the device at /etc/evprofile/default.json
Manifest file to control the size of event table is maintained at buildimage/src/sonic-eventd/etc/eventd.json and gets installed on the device at /etc/eventd.json

How to verify it

A python script is developed to raise events, raise/clear/ack/unack alarms. The script verifies various counters from EVENT_STATS and ALARM_STATS tables.

Ran tests/event_unit_test.py

root@sonic:/# python3 event_unittest.py

TEST: raise event
test_event success

TEST: raise alarm
test_raise_alarm success

TEST: raise and clear alarm
test_clear_alarm success

TEST: ack alarm
('42006',)
raise_id is 42006
test_ack_alarm success

TEST: unack alarm
('42009',)
raise_id is 42009
test_unack_alarm success
root@sonic:/#

Dependent PRs

sonic-mgmt-framework
sonic-mgmt-common
sonic-swss-common

Define libeventnotify for applications to use LOG_EVENT to raise events/alarms. Eventd to receive and store them in the DB. Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

…isord; Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

This reverts commit 7964d26.

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

…tency. Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

…gnore it. throttle-timeout support to specify time duration to keep throttling the event from an applicaiton. Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

rajendra-dendukuri

Please consider removing blank lines at at the end of newly added files
At what point are Event tables saved from redisdb to the corresponding dump.rdb file? If so is there a usecase where a thermal shutdown happens and all the event history is lost. I agree that syslogs will have information of all the event notifications.

rajendra-dendukuri · 2021-10-28T19:25:42Z

dockers/docker-database/database_config.json.j2

            "unix_socket_path": "/var/run/redis-chassis/redis_chassis.sock",
            "persistence_for_warm_boot" : "yes"
+        },
+        "redis1":{


The reason for having a dedicated redis instance for event mgmt is because they are supposed to persist across reboots. If that is the intention, can we name this instance as redisp instead of redis1. Other applications can also use it.

During the HLD review, we were asked to keep event/alarm tables in separate instance so that they wont grow too big that impacts other tables.

rajendra-dendukuri · 2021-10-28T19:32:16Z

files/build_templates/eventd.service.j2

+Description=Eventd container
+Requires=database.service
+After=database.service
+Before=ntp-config.service


Any particular reason for the before ntp-config service dependency?

At that time, I thought there might be a scenario ntpd may want to raise an event/alarm.

rajendra-dendukuri · 2021-10-28T19:39:19Z

src/sonic-eventd/debian/changelog

+
+  * Initial release.
+
+ -- Eventd <support@dell.com>  Thu, 9 Apr 2020 12:00:00 -0800


Can you confirm the date? 2020 or 2021?

Will change

rajendra-dendukuri · 2021-10-28T19:52:25Z

src/sonic-eventd/lib/debian/changelog

+
+  * Initial release.
+
+ -- Eventd <support@dell.com>  Thu, 9 Apr 2020 12:00:00 -0800


will change

rajendra-dendukuri · 2021-10-28T19:53:53Z

src/sonic-eventd/lib/debian/libeventnotify-dev.links

@@ -0,0 +1,2 @@
+#! /usr/bin/dh-exec
+/usr/lib/x86_64-linux-gnu/libeventnotify.so.0 /usr/lib/x86_64-linux-gnu/libeventnotify.so


What about arm64 platforms?

rajendra-dendukuri · 2021-10-28T21:04:52Z

src/sonic-eventd/src/eventconsume.cpp

+            EventInfo tmp = (EventInfo) (it->second);
+            // discard the event as event_static_map shows enable is false for this event
+            if (tmp.enable == EVENT_ENABLE_FALSE_STR) {
+                SWSS_LOG_NOTICE("Discarding event <%s> as it is set to disabled", ev_id.c_str());


Can this be SWSS_LOG_INFO? These are disabled events, may not be of great interest.

rajendra-dendukuri · 2021-10-28T21:23:08Z

src/sonic-eventd/src/eventconsume.cpp

+                        m_alarmTable.set(ev_src, ack_vec);
+                        m_eventTable.set(ev_src, ack_vec);
+                    } else {
+                        SWSS_LOG_ERROR(" %s/%s is already un-acknowledged", ev_id.c_str(), ev_src.c_str());


Can this be NOTICE instead of ERROR. There is no error in eventd that has happened. It is an invalid input.

rajendra-dendukuri · 2021-10-28T21:23:44Z

src/sonic-eventd/src/eventconsume.cpp

+                        m_alarmTable.set(ev_src, ack_vec);
+                        m_eventTable.set(ev_src, ack_vec);
+                    } else {
+                        SWSS_LOG_ERROR(" %s/%s is not in raised state", ev_id.c_str(), ev_src.c_str());


Can we make this NOTICE

rajendra-dendukuri · 2021-10-29T02:19:07Z

src/sonic-eventd/debian/eventd.install

@@ -0,0 +1,2 @@
+/etc/eventd.json
+/var/evprofile/default.json


Change this to /var/lib/evprofile/default.json

May I know the reason?
The default.json is placed under sonic-eventd/var/evprofile/default.json so, the .install is using that path.
Should we rather put default.json in sonic-eventd/var/lib/evprofile/ ?

But why?

rajendra-dendukuri · 2021-10-29T02:19:42Z

dockers/docker-eventd/docker-init.sh

+
+sonic-db-cli EVENT_DB config set  notify-keyspace-events AKE
+
+cp /var/evprofile/default.json /etc/evprofile/default.json


cp /var/lib/evprofile/default.json /etc/evprofile/default.json

Previous comment.

Signed-off-by: Srinadh Penugonda <srinadh.penugondaa@dell.com>

spenugondaa added 13 commits June 3, 2021 01:15

Event and Alarm Framework

7416337

Define libeventnotify for applications to use LOG_EVENT to raise events/alarms. Eventd to receive and store them in the DB. Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

Restore persisted redis data on cold boot

7964d26

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

use python3; fix event stats intialization; use right path for superv…

766a30a

…isord; Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

updated syslog message format

081b20d

updated purge events code

773fd20

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

Use time_ns() from python library

8528868

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

changed eventd.json limits to max-days and max-records

af8b22a

update ack counter on clear action

136aa90

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

update acknowledged counter on alarm clear

4f9000f

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

update acknowledged event stats counter on clear

d51e64f

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

event acknowledged counter update

cf2506f

protect against a bad application flooding

98682d9

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

remove unused start.sh; cleanup eventconsumer

82e1312

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

spenugondaa requested review from lguohan, qiluo-msft and xumia as code owners June 7, 2021 23:40

This was referenced Jun 7, 2021

Event mgmt sonic-net/sonic-mgmt-common#48

Open

Declare tables for Event and Alarm Framework. sonic-net/sonic-swss-common#490

Open

Event Management support sonic-net/sonic-mgmt-framework#85

Open

spenugondaa added 7 commits June 8, 2021 01:14

Merge branch 'master' into event-mgmt

42dc096

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

Revert "Restore persisted redis data on cold boot"

385bc0b

This reverts commit 7964d26.

docker eventd infra cleanup

1296e95

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

change instance name for event-db. Support cold boot to provide peris…

135275b

…tency. Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

Merge branch 'master' into event-mgmt

1187a22

on bootup, if the read event profile is same as already configured, i…

09f7b1a

…gnore it. throttle-timeout support to specify time duration to keep throttling the event from an applicaiton. Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

check message as well while throttling.

96f887d

Signed-off-by: spenugondaa <srinadh.penugondaa@dell.com>

bhaveshdell mentioned this pull request Jun 26, 2021

Event mgmt sonic-net/sonic-utilities#1695

Open

rajendra-dendukuri suggested changes Oct 29, 2021

View reviewed changes

spenugondaa mentioned this pull request Nov 3, 2021

Event and Alarm Framework HLD sonic-net/SONiC#761

Closed

Updated based on review comments; log levels, time stampswq

6db5f12

Signed-off-by: Srinadh Penugonda <srinadh.penugondaa@dell.com>

zhangyanzhao requested a review from renukamanavalan January 27, 2022 16:45

vboykox mentioned this pull request Feb 16, 2023

Event and Alarm Managment Framework HLD. sonic-net/SONiC#1064

Merged

zhangyanzhao mentioned this pull request Jan 12, 2024

Event/alarm framework HLD update. Integrate with event producer framework. sonic-net/SONiC#1409

Merged


		* Initial release.

		-- Eventd <support@dell.com> Thu, 9 Apr 2020 12:00:00 -0800

		@@ -0,0 +1,2 @@
		#! /usr/bin/dh-exec
		/usr/lib/x86_64-linux-gnu/libeventnotify.so.0 /usr/lib/x86_64-linux-gnu/libeventnotify.so


		sonic-db-cli EVENT_DB config set notify-keyspace-events AKE

		cp /var/evprofile/default.json /etc/evprofile/default.json

Conversation

spenugondaa commented Jun 7, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why I did it

How I did it

How to verify it

Dependent PRs

Uh oh!

rajendra-dendukuri left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

spenugondaa commented Jun 7, 2021 •

edited

Loading