Ansible for Networking - Part 1: The start of the series

For those who have been reading my posts for a while, they’ll know that while currently I’m a DevOps Engineer, I spent the previous decade managing and configuring service provider networks. For the majority of that time, the network was configured by hand. The closest most people in the industry had to an automation toolset was either using a spreadsheet with variables, their own scripts they had created, or delegating the task to multiple junior engineers. [Read More]

Building Windows AWS AMIs using Packer and Ansible

Like many other companies that are deploying their applications to the cloud, the majority of our estate uses Linux. However we do need to use Windows for a couple of purposes. This could be for application testing, or for specific Windows features. We also recently adopted Packer to build our machine images, to allow them to be defined in code (and therefore within version control). In Amazon, these machine images are called AMIs. [Read More]
devops  aws  packer  windows  aws  ami 

Prometheus: snmp_exporter and OpenBSD

In a previous post, I showed how to run the Prometheus node_exporter on a number of different operating systems, including OpenBSD. Many OpenBSD installs are used as, or to replace, network appliances (e.g. peering routers, firewalls, VPN concentrators). Traditionally, you would monitor networking equipment using SNMP. OpenBSDs snmpd(8) can expose a number of metrics that cover carp(4), pf(4), relayd(8) and more. Prometheus and SNMP The snmp_exporter is used so that Prometheus can monitor devices via SNMP. [Read More]

OpenBSD: High-Availability Firewalling

While most posts on this site usually concern Linux, I have a bit of a soft spot for OpenBSD. OpenBSD is an operating system from the Unix lineage, started in Bell Labs many years ago, eventually giving rise to the Berkley Software Distribution (BSD). The most known versions of BSD are NetBSD (who focus on portability, running on pretty much any hardware), FreeBSD (who focus on covering as many purposes as possible) and OpenBSD (who focus on security, sometimes at the expense of performance). [Read More]

Prometheus: Consul Service Discovery for blackbox and snmp exporter

In a previous post I covered how to use Consul for service discovery of standard exporters, allowing Prometheus to automatically discover what services to monitor. However, this configuration didn’t cater to exporters like the snmp_exporter or blackbox_exporter. What is interesting about both of the above is that rather than generating metrics for a local application, they are a proxy for other services. For example, you can use BlackBox exporter to do ICMP checks or HTTPS checks, without running an exporter on the services themselves. [Read More]

Prometheus: Discover services with DNS

In a previous post I covered how to use Consul for service discovery, allowing Prometheus to automatically discover what services to monitor. There are some cases where either setting up Consul (or similar) is not viable, or adds complexity that is not required. If you are already running your own DNS nameservers, you could make use of DNS SRV records. Common DNS record types The most common DNS records are A, AAAA and PTR. [Read More]

DNS Anycast: Using BGP for DNS High-Availability

DNS has a number of mechanisms for redundancy and high availability. More often than not, clients will have a primary and secondary nameserver to talk to. However, if the primary nameserver fails for whatever reason, then the queries to the primary usually need to timeout before attempting queries to the secondary. Also the speed of general web browsing can often be dictated by how long it takes to receive a valid DNS response to the query. [Read More]

Configuration Seasoning: Getting started with Saltstack

Configuration management is the practice of deploying and managing your application and infrastructure configuration through automated tooling, rather than managing all of your infrastructure manually. This can cover everything from Linux servers, to network equipment, installing packages to updating existing services. The primary benefits are that you can manage more infrastructure without the operational burden increasing significantly, and that your configuration is consistent across your estate. There are already a number of tools which achieve this: - [Read More]

Prometheus: Monitoring node statistics on other operating systems

In my previous posts on Prometheus, most of the monitoring has been geared to either getting metrics from Linux hosts. There have been a couple of exceptions, like the blackbox_exporter (for ICMP/Ping, HTTP(S) and TCP socket monitoring) or the snmp_exporter (more commonly used for networking gear). Linux is not the only operating system you can monitor node metrics for though. Other operating systems have either their own port of the node_exporter, or via other exporters entirely. [Read More]

Prometheus: Discovering Services with Consul

In my previous post, I detailed moving my home monitoring over to Prometheus. I’ve gained huge insights into my home network (and a few external services I rely on), and have been very happy with it. Adding new endpoints has been pretty straightforward. I have been using Ansible to generate the prometheus.yml configuration file, using variables to generate each section of the scrape configuration. This has worked equally well for both services exposing native Prometheus endpoints (eg Cadvisor or Traefik) and for the numerous exporters I am running. [Read More]