theory-masters.md

@@ -22,6 +22,16 @@ This configuration is compared once a minute via a crontab, and if the dhcpd con
 
 An additional aspect of this is that it allows for secondary masters to be defined, and booted via PXE/DHCP into a preparatory state. Further details are below.
 
+## Compute node execution - TFTP/NFS
+
+To reduce the amount of external complexity, and to ensure a clean working environment for the compute nodes, these devices are configured to boot over the network via PXE. In order to give them their kernel, once PXE has passed into GRUB, the kernel and initramfs is transferred via Trivial File Transfer Protocol (TFTP), which is the standard way to netboot. This daemon is also controlled on the masters via Pacemaker.
+
+The next stage of booting is to gain a root filesystem. For this system, the root filesystem is hosted from the masters via NFS in a readonly fashion, ensuring that on each boot, the compute nodes always get the same configuration. Once this is done, however, the network configuration of the compute nodes can no longer be changed without disrupting the root filesystem, so before this mount is made, the network bonding of four gigabit ethernet ports is done, via a script as part of the initramfs initialisation process.
+
+To allow the compute node to perform some local file modifications such as logging or temporary debugging, an overlayfs is constructed as part of the pivot process. This ensures that the root filesystem is read-write, whilst the actual underlying filesystem is read-only, preventing potential data corruption.
+
+The NFS server daemon is controlled from the masters via Pacemaker.
+
 ## Redundant filesystem - DRBD
 
 Another aspect of the redundant masters is that they have a replicated filesystem shared between them. This is enabled via the use of Distributed Redundant Block Devices (DRBD) that runs at kernel-level, intercepting all block writes and replicating those over the backend network to the other master, ensuring that the two keep in-sync at all times. This filesystem is configured into a two-primary state to allow it to remain instantly available should there be a hardware failure, the mounting of which controlled by Pacemaker to prevent corruption.
@@ -34,7 +44,7 @@ Docker is used to host the database and frontend elements - it is here that the
 
 The actual frontend configuration consists of three individual containers, set up via docker-compose from locally-built Dockerfiles. There is a cronjob running once a minute, attempting to pull any updates to the frontend git repo, and if there is one, updating the running containers using docker-compose.
 
-## Secondary master installation
+### Secondary master installation
 
 Once a new secondary master is entered into the database and the dhcpd config updated, when booted onto the network this machine will automatically get an SSH key from the current master and then SSH back into it as user master-builder, opening a socket file for itself there. By doing this, it exposes a shell on this new machine, with the initramfs as its base filesystem in memory. There is a second script that can then be run, which will install this second master via the socket file, allowing it to use that key to log in as root on the current master and clone the entire filesystem.