feat: critical alerts by modules #263
Open
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Make several changes in the current critical alerts engine. 1. Now critical alerts are sent for each module individually. 2. Introduce new `CRITICAL_ALERTS_MIN_VAL_CSM_ABSOLUTE_COUNT` env variable. If the number of validators in the CSM module affected by the `CriticalMissedAttestations` or `CriticalNegativeDelta` alert is greater than the value specified in this variable, the appropriate alert will be triggered. For validators in curated modules the logic of sending alerts is kept the same as before (alerts are sent depending on the total number of active validators). 3. Ignore the number of active validators for node operators in the CSM module for `CriticalMissedProposes` alert. If there are validators in the CSM module affected by this alert, they all will be included in the alert summary regardless of the total number of validators for the node operator. 4. Add a new `nos_module_id` label to all critical alerts. So now it is possible to route alerts depending on the module to different channels via Alertmanager. 5. Rules for sending critical alerts were slightly loosened. Previously alerts were sent when the number of affected validators was greater than the particular threshold. Now alerts are sent when the number of affected validators is greater or equal to the threshold. 6. Add information about the module to the alert summary. 7. Add a new `CSM_MODULE_ID` env variable. Update information about all new envs in README. 8. Slightly change log info for critical alerts. Now logs display the particular critical alert type together with the modules for which it was sent.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Make several changes in the current critical alerts engine.
Now critical alerts are sent for each module individually.
Introduce new
CRITICAL_ALERTS_MIN_VAL_CSM_ABSOLUTE_COUNTenv variable. If the number of validators in the CSM module affected by theCriticalMissedAttestationsorCriticalNegativeDeltaalert is greater than the value specified in this variable, the appropriate alert will be triggered. For validators in curated modules the logic of sending alerts is kept the same as before (alerts are sent depending on the total number of active validators).Ignore the number of active validators for node operators in the CSM module for
CriticalMissedProposesalert. If there are validators in the CSM module affected by this alert, they all will be included in the alert summary regardless of the total number of validators for the node operator.Add a new
nos_module_idlabel to all critical alerts. So now it is possible to route alerts depending on the module to different channels via Alertmanager.Rules for sending critical alerts were slightly loosened. Previously alerts were sent when the number of affected validators was greater than the particular threshold. Now alerts are sent when the number of affected validators is greater or equal to the threshold.
Add information about the module to the alert summary.
Add a new
CSM_MODULE_IDenv variable. Update information about all new envs in README.Slightly change log info for critical alerts. Now logs display the particular critical alert type together with the modules for which it was sent.