Recently I decided to try and throw together a one stop shop for managing frequently performed actions in my homelab. The goal was to enable me to faster create VMs, test software, rip them down, and rebuild. This is where I started with LabMan - HomeLab Manager.
It is very alpha. Very very alpha. It consists of 2 components: a Rails application that handles background jobs and the web UI, and an agent written in Go that runs on the servers and checks them in while providing some general info such as installed packages.
At a glance features:
- Create new VMs
- Execute workflows against hosts
- Host monitoring
- Host firewall management
- DNS management
- Network device configuration
- Network visualization
- Service monitoring
Services that enable this Lab management system to function:
- VMware VCenter provides an interface for the ruby application to create VMs
- Cobbler provides an interface to configure the DHCP server to PXE boot hosts with specific configurations and CentOS kickstart files
- PowerDNS provides an interface to create/update/delete DNSrecords
- InfluxDB is the interface for storing metrics from hosts, and allows labman to read those metrics for alarming or graph generation
- Resque is used to queue background jobs (pinging hosts, evaluating alarm status, provisioning hosts) with multiple workers per queue, and redis as the datastore
- MySQL is used as the database to store state of the system
- An agent written in Go checks hosts into the Labman Check-In API, and reports current status of outstanding updates, kernel version, etc.
Shows hosts in the lab, a glimpse of their health, IP, and when the labman agent last checked in (WIP).
Single Host View - Dashboard⌗
This view allows you to see a dashboard with some metrics about a given host (CPU, RAM, Network, etc)
Single Host View - Workflow⌗
Shows information about workflows that have run against this host (more on that later).
Single Host View - Firewall⌗
Allows you to visually manipulate firewall rules which apply to this host. This works by applying changes to iptables chains.
You can also define services with ports to make adding firewall rules a bit easier. Upon saving the rules, hitting an iptables API endpoint allows the labman agent to update rules on the local host.
Single Host View - Alarms⌗
Shows alarms running against this host, their current status, and exit codes from alarm runs (0=ok, 1=warning, 2=critical)
Alarm notifications are sent through slack. Alarms are created via the admin panel using influxdb metrics, configuring the comparison operator, and setting thresholds.
Single Host View - Console⌗
Access a VMware Console view to interact with VMs having network issues, etc. HTML5. This is only available for VMs created in VMware.
Single Host View - Configuration⌗
Enable/disable alarms, set SSH access information, and even reprovision or deprovision the VM.
Create a new VM⌗
Wizard walks through VM creation and kicks off a provisioning workflow (more info later). You can also create multiple VMs at once, and a random string of characters is appended to the hostname prior to the first right leading subdomain defined in the name.
Show dashboard of network devices and some brief metrics.
Network Devices - Interfaces⌗
Show interfaces, connections, and even update the configuration of an interface (Cisco devices supported right now). This allows for editing interfaces on Cisco Catalyst devices directly via the UI. The config is fetched, parsed, displayed on the page in UI form, and then changes are turned into commands executed via SSH on the network devices.
Network Devices - Visualization⌗
Show connections between physical devices allowing you to see which network ports are chaining network devices, or network devices and host interfaces together.
Show subnets in parent-child view, VLANs, and utilization. Gives an at-a-glance view of network topology.
IPAM - Subnet View⌗
Show subnet details, utilization, hosts, DHCP scopes, etc. Subnets are automatically swept to detect new hosts. IP, Host, DNS, and Network Device objects are displayed to associated IPs.
DNS - Server View⌗
Show all zones on a DNS server. These zones are pulled from PowerDNS servers using the API configured in admin settings.
DNS - Zone View⌗
Show records for a DNS zone, create new records, delete stale records. These changes are sync’d to the PowerDNS servers bidirectionally. When provisioning a new host, these records are automatically created and cleaned up upon deprovisioning.
List services and their availability. These services are pulled from Consul servers configured in the Admin settings. Availability is determined using Consul metrics published to InfluxDB.
Services - Parent Service with Children⌗
Show a service, and subservices, with their health status as determined from Consul and InfluxDB metric data.
Shows workflows that have been created, as well as lists workflows that have been run. This includes host provisioning.
Workflows - Create Workflow⌗
Create a workflow by defining steps to perform against a host. This is a simple way to script repetitive tasks to automate basic tasks like installing packages, or updating a host.
Workflow - Run Example⌗
Install Consul workflow run example
Workflow - Provision Host⌗
Example host provisioning workflow run. This provisioning process does a few things:
- Creates a new VM in VMware on the proper network
- Creates DHCP entry in Cobbler for the given MAC address of the newly created VM
- Generates a DNS record for the host and syncs to the PowerDNS nameservers
- Boots the newly created VMware VM, which PXE boots and begins configuring the OS selected during the creation process
- Kickstart does a base configuration of the new host
- LabMan validates connectivity to the newly created host by waiting for basic host metrics to be created (Memory usage, CPU, etc)
Workflow - Deprovision Host⌗
This cleans up an existing VMware VM:
- Destroys the VM within VCenter
- Removes DHCP static record which involves removing the host from Cobbler and forcing a sync
- Removes DNS record from PowerDNS
Shows all alarms and their current state. Alarm notifications are sent through slack. These alarms are defined in the admin area by evaluating InfluxDB metrics for a given value, or change in value velocity, then applied to hosts.
Alarms - Single alarm view⌗
Shows a single alarm, current state, and history