labman

Recently I decided to try and throw together a one stop shop for managing frequently performed actions in my homelab. The goal was to enable me to faster create VMs, test software, rip them down, and rebuild. This is where I started with LabMan - HomeLab Manager.

It is very alpha. Very very alpha. It consists of 2 components: a Rails application that handles background jobs and the web UI, and an agent written in Go that runs on the servers and checks them in while providing some general info such as installed packages.

At a glance features:

  • Create new VMs
  • IPAM
  • Execute workflows against hosts
  • Host monitoring
  • Host firewall management
  • DNS management
  • Network device configuration
  • Network visualization
  • Alerting
  • Service monitoring

Services that enable this Lab management system to function:

  • VMware VCenter provides an interface for the ruby application to create VMs
  • Cobbler provides an interface to configure the DHCP server to PXE boot hosts with specific configurations and CentOS kickstart files
  • PowerDNS provides an interface to create/update/delete DNSrecords
  • InfluxDB is the interface for storing metrics from hosts, and allows labman to read those metrics for alarming or graph generation
  • Resque is used to queue background jobs (pinging hosts, evaluating alarm status, provisioning hosts) with multiple workers per queue, and redis as the datastore
  • MySQL is used as the database to store state of the system
  • An agent written in Go checks hosts into the Labman Check-In API, and reports current status of outstanding updates, kernel version, etc.

Picture Walkthrough

Hosts View

Shows hosts in the lab, a glimpse of their health, IP, and when the labman agent last checked in (WIP).

labman

Single Host View - Dashboard

This view allows you to see a dashboard with some metrics about a given host (CPU, RAM, Network, etc)

labman

Single Host View - Workflow

Shows information about workflows that have run against this host (more on that later).

labman

Single Host View - Firewall

Allows you to visually manipulate firewall rules which apply to this host. This works by applying changes to iptables chains.

You can also define services with ports to make adding firewall rules a bit easier. Upon saving the rules, hitting an iptables API endpoint allows the labman agent to update rules on the local host.

labman

labman

Single Host View - Alarms

Shows alarms running against this host, their current status, and exit codes from alarm runs (0=ok, 1=warning, 2=critical)

Alarm notifications are sent through slack. Alarms are created via the admin panel using influxdb metrics, configuring the comparison operator, and setting thresholds.

labman

labman

Single Host View - Console

Access a VMware Console view to interact with VMs having network issues, etc. HTML5. This is only available for VMs created in VMware.

labman

Single Host View - Configuration

Enable/disable alarms, set SSH access information, and even reprovision or deprovision the VM.

labman

Create a new VM

Wizard walks through VM creation and kicks off a provisioning workflow (more info later). You can also create multiple VMs at once, and a random string of characters is appended to the hostname prior to the first right leading subdomain defined in the name.

labman

labman

labman

labman

Network Devices

Show dashboard of network devices and some brief metrics.

labman

Network Devices - Interfaces

Show interfaces, connections, and even update the configuration of an interface (Cisco devices supported right now). This allows for editing interfaces on Cisco Catalyst devices directly via the UI. The config is fetched, parsed, displayed on the page in UI form, and then changes are turned into commands executed via SSH on the network devices.

labman

labman

Network Devices - Visualization

Show connections between physical devices allowing you to see which network ports are chaining network devices, or network devices and host interfaces together.

labman

IPAM

Show subnets in parent-child view, VLANs, and utilization. Gives an at-a-glance view of network topology.

labman

labman

IPAM - Subnet View

Show subnet details, utilization, hosts, DHCP scopes, etc. Subnets are automatically swept to detect new hosts. IP, Host, DNS, and Network Device objects are displayed to associated IPs.

labman

labman

labman

DNS - Server View

Show all zones on a DNS server. These zones are pulled from PowerDNS servers using the API configured in admin settings.

labman

DNS - Zone View

Show records for a DNS zone, create new records, delete stale records. These changes are sync’d to the PowerDNS servers bidirectionally. When provisioning a new host, these records are automatically created and cleaned up upon deprovisioning.

labman

labman

Services

List services and their availability. These services are pulled from Consul servers configured in the Admin settings. Availability is determined using Consul metrics published to InfluxDB.

labman

Services - Parent Service with Children

Show a service, and subservices, with their health status as determined from Consul and InfluxDB metric data.

labman

labman

Workflows

Shows workflows that have been created, as well as lists workflows that have been run. This includes host provisioning.

labman

labman

Workflows - Create Workflow

Create a workflow by defining steps to perform against a host. This is a simple way to script repetitive tasks to automate basic tasks like installing packages, or updating a host.

labman

Workflow - Run Example

Install Consul workflow run example

labman

Workflow - Provision Host

Example host provisioning workflow run. This provisioning process does a few things:

  • Creates a new VM in VMware on the proper network
  • Creates DHCP entry in Cobbler for the given MAC address of the newly created VM
  • Generates a DNS record for the host and syncs to the PowerDNS nameservers
  • Boots the newly created VMware VM, which PXE boots and begins configuring the OS selected during the creation process
  • Kickstart does a base configuration of the new host
  • LabMan validates connectivity to the newly created host by waiting for basic host metrics to be created (Memory usage, CPU, etc)

labman

labman

Workflow - Deprovision Host

This cleans up an existing VMware VM:

  • Destroys the VM within VCenter
  • Removes DHCP static record which involves removing the host from Cobbler and forcing a sync
  • Removes DNS record from PowerDNS

labman

Alarms

Shows all alarms and their current state. Alarm notifications are sent through slack. These alarms are defined in the admin area by evaluating InfluxDB metrics for a given value, or change in value velocity, then applied to hosts.

labman

Alarms - Single alarm view

Shows a single alarm, current state, and history

labman