Colourful Norwich skyline illustration

Michael Sage

IT, Digital & Culture

Category: Blog

Posted on

Proxmox Nagios Monitoring

Using the excellent check_pve.rb script I have setup comprehensive monitoring for proxmox servers, both locally and remotely.

In case the script disappears, here are some useful bits

pveum useradd monitoring@pve -comment "Monitoring User"
pveum passwd monitoring@pve
pveum roleadd PVE_monitoring -privs "Datastore.Audit,Sys.Audit,Sys.Modify,VM.Audit"
pveum aclmod / -user monitoring@pve -role PVE_monitoring

Usage:

check_pve v0.2.5 [https://gitlab.com/6uellerBpanda/check_pve]

This plugin checks various parameters of Proxmox Virtual Environment via API(v2)

Mode:
  Cluster:
    cluster         Checks quorum of cluster
  Node:
    smart           Checks SMART health of disks
    updates         Checks for available updates
    subscription    Checks for valid subscription
    services        Checks if services are running
    storage         Checks storage usage in percentage
    cpu             Checks CPU usage in percentage
    memory          Checks Memory usage in gigabytes
    io_wait         Checks IO wait in percentage
    net_in          Checks inbound network usage in kilobytes
    net_out         Checks outbound network usage in kilobytes
    ksm             Checks KSM sharing usage in megabytes
  VM:
    vm_cpu          Checks CPU usage in percentage
    vm_disk_read    Checks how many kb last 60s was read (timeframe: hour)
    vm_disk_write   Checks how many kb last 60s was written (timeframe: hour)
    vm_net_in       Checks incoming kb from last 60s (timeframe: hour)
    vm_net_out      Checks outgoing kb from last 60s (timeframe: hour)

Usage: check_pve.rb [options]

Options:
    -s, -H, --address ADDRESS        PVE host address
    -k, --insecure                   No SSL verification
    -m, --mode MODE                  Mode to check
    -n, --node NODE                  PVE Node name
    -u, --username USERNAME          Username with auth realm e.g. monitoring@pve
    -p, --password PASSWORD          Password
    -w, --warning WARNING            Warning threshold
    -c, --critical CRITICAL          Critical threshold
        --name NAME                  Name for storage
    -i, --vmid VMID                  Vmid of lxc,qemu
    -t, --type TYPE                  VM type lxc or qemu
    -x, --exclude EXCLUDE            Exclude (regex)
        --timeframe TIMEFRAME        Timeframe for vm checks: hour,day,week,month or year
        --cf CONSOLIDATION_FUNCTION  RRD cf: average or max
    -v, --version                    Print version information
    -h, --help                       Show this help message
Posted on

Proxmox P2V Migration

Windows Physical Server to Proxmox

I am fortunate enough that even with my main job, I am allowed to take side hustles. These are normally technical in nature, something that my primary role doesn’t involve so much any more, and I only take the ones that give me an opportunity to learn.

Just before Christmas a friend got in touch, one of their customers was being charged a huge amount for colocation of a single server (as it turns out protected by a woefully under powered firewall). I hadn’t done a physical to virtual migration (P2V) in about 10 years so I quoted for the job and we won the business. 

As part of the quote we suggested a trial migration, as the customer was wary of virtualisation, their current provider had actively discouraged it. It was time to find a P2V solution… In the past I have used the vmware convertor tool, there were two issues with this, firstly it hasn’t been updated in years (and has now been pulled) and secondly I wanted a proxmox compatible disk at the end.

Looking for new tool proved harder than I expected, until I stumbled upon Disk2vhd, a tool provided by Microsoft’s sysinternals “brand”. This tool will create a VHDX (or VHD) file for a physical drive and yes you can save it to the same drive! 

Screenshot of Disk2vhd
Disk2vhd GUI

As you can see the GUI for Disk2vhd is incredibly easy to use.  I learnt a couple of things, the first is to make sure you create the image at a disk level not a partition level else you end up with a confusing set of VHDs. Also don’t create one VHD and then disk2VHD that partition (because the image gets huge).

Once you have the VHDx images copy them to your proxmox server and convert them to qcow2

qemu-img convert -O qcow2 /var/lib/vz/harddrives/XXXX.vhdx /var/lib/vz/images/xxx/vm-xxx-disk-X.qcow2

That’s it, Disk2vhd is such a great tool and it’s available for free and appears to be updated!

Now you have the qcow images you can add them to your target VM. You will need to do a little prework to the VM before you can get the best performance. 

What I did:

– Create an empty 1Gb disk with the virtio controller (this forces the virtio driver install)
– Add the converted disk(s) to the VM as IDE
– Boot the server off the IDE disks
– Install the latest virtio drivers
– Change the IDE disks to virtio and remove the empty disk
– Boot and check

The live migration has been booked and I will update this post once this is complete with any further information.

 

Posted on

Backing up a Pi (with USB HDD)

Raspberry Pi Logo

Pi Backup

Time for another text based pi article.. sorry!

Attach the USB disk to the pi.

Next download pi shrink

wget https://raw.githubusercontent.com/Drewsif/PiShrink/master/pishrink.sh
sudo chmod +x pishrink.sh
sudo mv pishrink.sh /usr/local/bin

Next check for the correct paths

lsblk

Now run a dd copy to create the image

sudo dd if=/dev/mmcblk0 of=[mount point]/myimg.img bs=1M

Finally shrink the image

sudo pishrink.sh -z myimg.img

You now have a size optimized image that you can restore using the “official” pi image writer software. You can stick this in a cron job to create a regular image backup of any pi.

VNC

If you are cloning to test or deploy into another Pi, you will need to delete the VNC config or you will get a “cloned” error in the VNC client and it could stop working. Luckily it’s simple to fix, on the cloned machine do the following:

sudo systemctl stop vncserver-x11-serviced
sudo rm -rf /root/.vnc
sudo systemctl start vncserver-x11-serviced
Posted on

ZFS Expand with new disk

I realised that one of my 3Tb drives was very, very old. It wasn’t failing yet, but it would probably be best to proactively replaced it. The other drive in the mirror is a 4Tb drive, so I took the opportunity to buy another 4Tb drive and add it to the ZFS mirror… This was so simple it’s scary. Firstly I removed the old drive from the mirror. 
#zpool detach <Pool Name> <Device to Remove>

i.e #zpool detach SATAPool sdd
Then I shutdown the machine removed the old drive and put the new one in. When the box rebooted I added the new disk to the mirror. 
#zpool attach <Pool Name> <Existing Disk><New Disk>

i.e #zpool attach SATAPool sdc sdd
I let it resilver… but it didn’t auto expand. The main reason for this was I didn’t have autoexpand turned on. 
#zpool set autoexpand=on <Pool Name>
This didn’t fix the issue, but it did show the space was available.  The next thing I did was to use the online -e on every disk in the mirror. 
#zpool online -e <Pool Name> <Disk Name>

i.e. #zpool online -e SATAPool sdd
And then I had an extra Tb to play with!
Posted on

Hyper-v to Proxmox

This one is going to be word and code heavy.

Firstly shutdown the hyper-v gen2 VM.

Copy the VHDX to the proxmox server, remember you will need twice the space of the VHDX available temporarily to completed the move.

Create the VM on Proxmox, you will need to create it using a OVMF (UEFI) BIOS. You will also need to create a 1Gb virtio hard disk (again this is temporarily)

Make a note of the VM ID (i.e. 103), you’ll need this to import the hard disk.

Once the VHDX is copied across you need to import the disk

 qm importdisk VMID SORCEDISK.vhdx DATASTORE --format qcow2

i.e. qm importdisk 103 /tmp/data.vhdx SATA –format qcow2

This will then import the disk to the VM.

Next you need to head over to the VM in proxmox and attach the disk in the gui, you will need to install it as SATA or IDE (not virtio yet)

Boot the machine up into windows, install the latest tools, the temporary 1Gb disk we installed earlier will mean the virtio drivers are installed.

Shut the VM down, remove and reattach your windows drive as virtio, remove and delete the temporary 1Gb drive and turn the machine back on. If your machine had a static IP you will need to readd it, you may get a warning about it being assigned to another NIC. You can ignore this, however, I would recommend removing it at some point!

You can now delete the VHDX file you copied to the proxmox server

Job done!

Graphic with 3 monitors symbolising virtual machines
Posted on

Pi Rack & Pi KVM

My Pi Rack including 2 Pi's and PiKVM and USB HDD components
Pi Rack

I decided that I wanted to tidy up my lack rack and put my two Pi’s rack mounted and out of the way. 

The first pi is a backup “server” it runs rsync and has a 3Tb USB drive, it is a staging server between my backups and OneDrive sync. This was easy to do, I printed one module and one pi rack module, works great! I am thinking I might PoE my rack mount Pi’s but this would mean two HATs and a PoE switch for the study.

The second Pi is my Pi KVM, this Pi is slightly more complex as it has some external components, these are mounted in a case, this case was too big for the module, so I 3D printed cases for the individual cases for the rack module. This has tidied it up and it now fits snuggly in the module.

I have done a lot of 3D printing to support the Pi Rack and Pi KVM components, you can see them on the media carousel, links are below.

The printed components are; Modular Pi Rack, USB power splitter case and HDMI to CSI-2 case.

Previous
Next

Links:

Posted on

A Couple of PDFs…

Over the last couple of months I have relied on a couple of web articles to get me through particularly specific issues. I hope these sites will remain available forever… However, I have “printed” the sites to PDFs to make sure they are available should the sites disappear / migrate / melt… etc

pfSense - Mikrotik Open VPN

The key piece of information in this one is around the client specific override!

Unifi CK2+ Backup

How to backup a Unifi controller / CK2+ using the amazing rclone.

Proxmox vTPM Win 11

A guide for setting up vTPM for virtual machines running on proxmox 7 (NB: This is no longer needed as proxmox 7 now has vTPM built in)

Posted on

SES – SNS -Lamdba – Dynamodb – Alllll the AWS

I’ve been using SES for a while for sending email from servers.

A few months ago one of my servers was compromised and I discovered there is no such thing as good logging in SES…. whooops… Amazon suspended my account and this lead me to investigate how I could get some visibility of SES logging. This was considerably harder than I would have thought. There are no SMTP logs, not real recording of any information and no diagnostics. I nearly left SES for this, then I decided to use it as a learning opportunity to understand more about AWS & SES.

I used this guide to get it all working: https://blog.andreev.it/?p=5513

Hopefully this site never goes down, if so I will need to write my own guide! 

It’s easy to add extra fields to the database. These are all the objects available https://docs.aws.amazon.com/ses/latest/DeveloperGuide/notification-contents.html

Finally this docker image allows you to connect to dynamodb easily https://github.com/YoyaTeam/dynamodb-manager

The only thing I haven’t managed to do is find out which IAM user actually sent the email… I need to do more research on this to try and enhance my logging further.

AWS Logos for SES, SNS, Lamdba and Dynamodb
Posted on

Emergency Pi Zero

I have had a couple of requirements recently where I have needed to leave a device onsite for remote access. I initially thought that the best way to handle this would be to put a pi onsite that joined my VPN and then I could connect to all the machines on the remote network and do my troubleshooting like that…

I looked at my pi shelf and there looking all little and shiny was a Pi Zero… At this point I thought to myself… what do I actually need from this device.

 

Pi Zero 2 W Photo

 

I have a couple of Emergency Linux VMs on dedicated servers that run a lightweight GUI with pretty much just a web browser. These emergency VMs work with VNC, but VNC isn’t running all the time. You SSH into them and as part of the login it fires up a VNC session and when you log off it tears the connection down. So you SSH in with 2FA, this starts the VNC Server (with a password), when you finish you log out and it clears down the VNC session. I also installed a cron job that checks the devices external IP address and emails you when it changes (for when it goes into a residential setting).

Here’s how it’s setup

  • Install Raspberry OS (with desktop)
  • Enable SSH
  • Add to .bashrc

if [[ -n $SSH_CONNECTION ]] ;
vncserver
fi

  • Add to .bash_logout

 vncserver kill :1

When you arrive onsite, connect the pi zero to the network, add a port forward for port 22 and 5901 to the pi. Check the IP emailer works (see link above).

Posted on

Migrations & Pi KVM

Over the last month I have migrated my home server from a Gen 8 HP Microserver to a Lenovo P500 workstation. There are many reasons for my migration the two biggest were that I was being constrained by the amount of RAM the Microserver could take (16Gb vs 512Gb), the processor was also becoming a bit of a bottle neck.

The second was that in my professional life I have moved from VMWare ESXi to Proxmox and my home lab was the only ESXi server that I was left managing, it also meant I wasn’t reflecting my professional install base so making it hard to test things.

Migrations are horrible, no matter how much planning you do, they take time and suck! No matter how many trials and tests you do there will always be something.

I used an old desktop PC with a 500Gb SATA drive and a 240Gb SSD to migrate all servers other than the Windows server (not enough space or grunt).

Although exceptionally boring and probably of no interest to anyone this was my migration plan..

  • Shutdown new host
  • NIC in new host
  • Check Second Network Card
  • Restore firewall
  • Copy all Proxmox machines from test proxmox host 
  • —–
  • Migration
  • Copy latest backup
  • Run Full backup c:\backups\backup.bat
  • Check USB disk on another PC
  • Close OneDrive
  • Restart PC
  • Check OneDrive is stopped
  • Shutdown VM
  • Convert System Disk
  • Check Proxmox Boot
  • —–
  • Move 2Tb disks to think station
  • Create new ZFS 2Tb for File Server
  • Boot File Server
  • Add 1.8Tb disk
  • Setup OneDrive if needed

20/02/2020 20:12 <JUNCTION> data [d:\data]
17/09/2020 08:48 <JUNCTION> media [D:\media]
17/09/2020 09:07 <JUNCTION> server backups [D:\backups]

  • Start OneDrive
  • Undisable Start with Windows OneDrive
  • Check shares
  • Remove “to watch” from backups
  • USB pass through
  • Setup Proxmox Backups (Exclude File Server d drive)
  • —–
  • Remove SSDs from Microserver and check
  • Rebuild Test Proxmox as Hobby PC with 240Gb SSD
  • —–
  • Take old Hobby PC
  • Check 120Gb SSDs
  • —–
  • 2FA for SSH and Proxmox on New Host
  • Add New Host to Nagios

This was all in a text file which I constantly updated and changed during the actual migration. It went well and there were only a couple of hiccups. The testing had paid off.

 

Photo of ThinkStation Home Lab

Hopper – New Host

Running proxmox with a number of Windows, Linux and BSD VMs.

  • Intel(R) Xeon(R) CPU E5-2609 v3 @ 1.90GHz (1 Socket)
  • 48Gb RAM
  • 2x 480Gb NAS SSD (ZFS), 2x 3Tb NAS SATA (ZFS), 1x 2Tb SATA (Backups)

The two USB cables – One going to an external HDD for file level backups, the second goes to the Pi KVM (for keyboard and mouse control)

Photo of two Raspberry Pi's one with an external hard drive and one with power hanging over the edge (whoops)

The Pis

Tron – Pi 2 3+
4Tb USB Drive
Backup Pi (Rsync and rclone)

IP KVM – Pi 4 (2Gb)
Power/Data Splitter at the back
USB to HDMI Capture Card

Both are cabled into the network. The Pi 2 only has 100Mbps network, so it’s likely to need replacing soon to keep up with my internet, but for now it works! The Pi 3+ has “Gb network” however due to it using the same USB bus it can only realistically achieve 300Mbps.

Pi KVM

This part of the project nearly got it’s own page… However, I don’t have much to say! One of the biggest drawbacks of migrating to the workstation was that I lost iLO (intelligent lights out / IPMI). I use iLO rarely but it is an incredibly useful when you do need it!

I was looking at aftermarket cards and IP based KVMs and they are expensive! I couldn’t justify the cost for a single host or the amount of time I use it. 

Then I came across Pi KVM, it looked hugely daunting until I started reading about it. For simple KVM features (and a host of other features) it was incredibly easy to build a Pi 4 KVM (you can use other Pi generations but you will need to do more work). Just one cable and an HDMI capture card and it just works! 

They are also developing their own Pi HAT with all the features (including power management (i.e. remote reboot)), I’ll probably buy one when they are released as I can think of a number of locations where a sub £100 KVM would be a life saver, especially with the remote reboot abilities.

Pi KVM can be found here: https://www.pikvm.org/

Another Pi KVM project can be found here: https://tinypilotkvm.com/

Bits I bought to make my Pi KVM

That was it! I had a case, power supply and SD card knocking about any way… When the hat is released I will need to think about a different case.

Screenshot of Pi KVM in a web browser
Pi KVM (Currently only using KVM, power control to come later)