How to send OpenShift alerts to Gmail

Lately I’ve been exploring new monitoring capabilities which are built-in the OpenShift product — without the need for a 3rd party monitoring tool. Even though most of the times 3rd party companies develop more advanced monitoring tools, the purpose of this blog is to show off one of the many out of the box monitoring features that come with OpenShift.

In this blog, I’ll be going step by step through the procedure of integrating the OpenShift monitoring stack to a Gmail account.

Why would I do that?

I think that the reasons for integrating a platform with a an external and independent source are quite obvious -

  • Gmail will aggregate alerts from the past and the present. Making it easier to trace a reoccurring error.
  • Alerts that were sent during off-work time will be saved in the mailbox, while they can disappear in the OpenShift console.
  • An external source does not depend on the stability of the OpenShift cluster. The cluster can be down, but the administrator could still debug the errors received by the Alert Manager.
  • A central Gmail account can manage multiple Alert Managers. Meaning that you could connect multiple OpenShift clusters to it, or even standalone Alert Managers that perform non-OpenShift related monitoring.

NOTE: while this blog is about connecting an OpenShift Alert Manager to Gmail, it is important to note that you can integrate Alert Manager with many other systems, like Telegram, Discord, AWS SNS and more. To read about Alert Manager integrations go to — https://prometheus.io/docs/operating/integrations/

Let’s get to it

Now that we know why would we need to integrate Alert Manager with a Gmail account. Let’s go through the configuration needed for it to happen!

Prerequisites —

  1. OpenShift 4.4+.
  2. Internet connection.
  3. oc CLI tool.

Procedure —

  • We will need to create a Gmail account that Alert Manager will use to send emails on its behalf. Since it will not be too secure to store the account’s password in OpenShift’s Alert Manager configuration, it is better to generate an app password for Alert Manager at — https://support.google.com/accounts/answer/185833?hl=en
  • Now that we have an account, lets generate Alert Manager’s configuration.
  1. smtp_from, smtp_auth_username and smtp_auth_password describe the source user’s information. Alert Manager uses the credentials to send an email through the mentioned account.
  2. The to field describes the destination email to which alert manager will forward to alerts.
  3. The severity field describes the severity of the alerts that will be send to Gmail. In my case, I’m using critical, but warning can be used as well.
  • Now let’s apply the configuration onto the Alert Manager. Use the file created in the previous step.
$ oc set data secret/alertmanager-main -n openshift-monitoring --from-file <path to file>secret/alertmanager-main data updated
  • Wait for Alert Manager to reload it’s configuration
$oc logs alertmanager-main-0 -c alertmanager -f -n openshift-monitoringlevel=info ts=2020-11-28T11:24:49.521Z caller=coordinator.go:131 component=configuration msg="Completed loading of configuration file" file=/etc/alertmanager/config/alertmanager.yaml

And you are done! Alert Manager should now redirect critical errors to your Gmail account.

Testing —

NOTE: DO NOT PERFORM THIS TEST IN PRODUCTION ENVIRONMENTS

To test the integration, let’s simulate a scenario in which a filesystem runs out of space. To perform the test, I will create a large file in the /tmp directory on one of the nodes in my OpenShift cluster. Creating a large file will trigger a NodeFilesystemAlmostOutOfSpace Alert in OpenShift, which will get sent to my Gmail mailbox.

  • Access the console of one of the worker nodes in the OpenShift cluster —
$ oc get nodesNAME                                                 STATUS   ROLES    AGE   VERSION
...
worker-2.tommercluster.lab.upshift.rdu2.redhat.com Ready worker 59d v1.18.3+2cf11e2
$ oc debug node/worker-2.tommercluster.lab.upshift.rdu2.redhat.comStarting pod/worker-2tommerclusterlabupshiftrdu2redhatcom-debug ...
To use host binaries, run `chroot /host`
Pod IP: 10.0.94.48
If you don't see a command prompt, try pressing enter.
  • Create a large file in the /tmp directory (leave less than 3% in the filesystem to trigger the alert) —
sh-4.2# chroot /hostsh-4.4# dd if=/dev/urandom of=/tmp/large_file bs=1M count=39003900+0 records in
3900+0 records out
4089446400 bytes (4.1 GB, 3.8 GiB) copied, 26.1484 s, 156 MB/s
  • Check that the filesystem is indeed full —
sh-4.4# df -h | grep -Ev "var|run"Filesystem                            Size  Used Avail Use% Mounted on
/dev/mapper/coreos-luks-root-nocrypt 40G 13G 27G 33% /
tmpfs 3.9G 0 3.9G 0% /sys/fs/cgroup
devtmpfs 3.9G 0 3.9G 0% /dev
tmpfs 3.9G 84K 3.9G 1% /dev/shm
tmpfs 3.9G 3.9G 86M 98% /tmp
/dev/vda1 364M 167M 174M 50% /boot
/dev/vda2 127M 6.8M 121M 6% /boot/efi
  • Make sure that an alert has been triggered in the OpenShift Web console (Using an administrative user, navigate to Monitoring -> Alerting) —
  • Log into your Gmail mailbox, and you will be able to see the alert!

Conclusion

In the article we went through some of the basics of OpenShift’s monitoring stack. You should now be able to integrate OpenShift’s Alert Manager with your Gmail account or even your organization’s mailing server.

Make sure to check out the rest of OpenShift’s monitoring features at —

https://docs.openshift.com/container-platform/4.6/monitoring

And along the way, make sure familiarize yourself with Prometheus, to fully understand how monitoring in OpenShift works out of the box!

And, most importantly, have fun monitoring!

--

--

--

Cloud Consultant, Red Hat

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

LRU and LFU

How to Hack Ancestry.com Links Down to a Manageable Size

Our Smart Contract Development Process in Four Minutes

Like on #Instagram #foto: http://ift.tt/2jLQZvG | @Regrann from…

How Long Does It Take To Learn C? The Answer: It Really Depends

How to get an ambassador from the Aleo project?

What is AWS CLI?

Discoveries from a Hobby Project

before, after, final, from engineering discoveries to personal discovery

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Michael Kotelnikov

Michael Kotelnikov

Cloud Consultant, Red Hat

More from Medium

Kubernetes Introduction: Istio

Architecting a development lifecycle for a Kubernetes-based deployments

OpenDistro with cert-manager