Building a high-performance homelab is rarely a simple journey. For many enthusiasts, it’s an evolutionary path defined by trial, error, and the relentless pursuit of stability and throughput. My journey has been a dramatic story of two distinct eras: the fragile era of USB networking torture and the glorious revolution of dedicated enterprise-grade silicon running over a 25GbE fabric.
Today, I’m excited to share the details of my new, arguably „overkill” 4-node VMware cluster built on Minisforum nodes, 25GbE MikroTik switches, and a potent vSAN + iSCSI hybrid storage architecture. It represents a giant revolution in my infrastructure, even if a slightly extravagant one for a „normal” enthusiast.
The Evolution: Leaving the USB Nightmare Behind
Before diving into the new cluster, we must appreciate how far I’ve come. My previous environment was a 3-node cluster built from HP EliteDesk G4 mini-PCs. While the CPUs were decent (Core i7-8700), the environment was severely limited by memory (64GB DDR4 per node) and storage (one NVMe vSAN drive and one OS drive).
However, the defining characteristic of that cluster was its networking: a TP-Link Omada SG3218XP-M2 switch connecting the nodes via 2.5GbE USB network adapters. I ran vSAN over two of these 2.5GbE USB adapters, and it was an absolute nightmare.
If a USB card accidentally disconnected or even just hung, it would trigger a Purple Screen of Death (PSOD) on the node, crashing the entire node and causing the vSAN cluster to collapse. The system was fragile, low-performance, and incredibly unstable. I lived in constant fear of a network failure bringing my digital world down. The jump to the new system was not just an upgrade; it was a revolution.
The Foundation: The 4-Node Minisforum MS-A2 Cluster
The core of the new system is defined by brute force and silicon stability. I have deployed a 4-node VMware ESXi cluster using Minisforum MS-A2 mini-PCs. These are powerhouses.
Each node boasts:
-
-
16 Cores / 32 Threads Processor: Massive multi-threading capability.
-
128GB RAM: Solves all the memory constraints of the previous generation.
-
2TB Lexar NM790 NVMe SSD: Dedicated entirely to the vSAN ESA datastore.
-
Rock-Solid Intel Networking: Integrated network cards replacing the volatile USB adapters.
-
The difference in stability and core compute is staggering. No more PSODs from a shaky network connection; just pure, sustained performance.
The Fabric: MikroTik-Powered 25GbE Dual-Network
To harness the power of four modern compute nodes and enterprise storage, I needed a massive network backbone. I chose MikroTik to handle the „Overkill Fabric.” I am running two separate MikroTik switches to segregate traffic and maximize performance.
1. The Core & iSCSI Network: CRS326-24S+2Q+RM
This switch handles the core connectivity and the dedicated iSCSI traffic to my shared storage NAS. Every iSCSI device and the main LAN is connected here via 10GbE SFP+.
2. The High-Performance Fabric: CRS518-16XS-2XQ-RM
This is the beast of the operation, dedicated to the most traffic-heavy components: vSAN, vMotion, and Node VLANs. This switch runs the cluster at 25GbE, providing a colossal pool of bandwidth.
On the nodes themselves, the networking breakdown is optimized using a mix of Intel controllers:
-
-
10GbE (Intel X710): Dedicated to iSCSI traffic (2x 10GbE ports).
-
25GbE (Intel E810-XXVDA4): Dedicated to vSAN traffic (2x 25GbE ports), vMotion traffic (1x 25GbE port), and generic Node/VLAN traffic (1x 25GbE port).
-
2.5GbE: Unused, native ports (2x 2.5GbE ports).
-
Moving the vSAN traffic from flaky 2.5GbE USB adapters to dedicated 25GbE Intel E810 controllers running over the CRS518 fabric has transformed the cluster’s response time and throughput. To further push the performance envelope, I’ve implemented RDMA (Remote Direct Memory Access) for the vSAN network. Leveraging the native RDMA capabilities of the Intel E810 cards, vSAN traffic can bypass the ESXi CPU stack, communicating directly between the nodes’ RAM. This minimizes latency and maximizes throughput, making the vSAN fabric truly elite.
The Storage: Hybrid vSAN + iSCSI with QTier Hero NAS
My storage architecture is a powerful hybrid designed to balance extreme vSAN performance with massive shared storage capacity for space-hungry VMs.
Most of my virtual machines live on the vSAN ESA datastore, utilizing the local NM790 Lexar drives in each node. ESA (Express Storage Architecture) is perfectly suited for these all-NVMe environments, providing incredibly fast storage directly to the VMs.
For VMs demanding significant storage space, I rely on a robust external NAS: a QNAP TS-855eU. This NAS is no slouch, equipped with 64GB ECC RAM and configured with an aggressive storage hierarchy managed by Qtier hero:
-
-
Shared Storage Fabric: Connected via 10GbE via the CRS326 switch.
-
Caching Tier: Two potent WD RED SN700 2TB NVMe SSDs.
-
Capacity Tier: Eight IronWolf Pro 16TB HDD disks configured in RAID 50.
-
I expose this massive capacity tier (using QTier for auto-tiering) to the cluster via iSCSI. The cluster currently uses three 10TB iSCSI datastores to host the largest virtual machines.
The Edge: Robust Security with FortiGate 90G
Every strong homelab network needs a resilient edge firewall. Sitting at the perimeter is a FortiGate 90G appliance. To match the speed of the rest of the network, the FortiGate is connected to the LAN via two 10GB SFP+ ports (utilizing the CRS326 core switch).
Equipped with a full ATP (Advanced Threat Protection) license, the FortiGate provides high-throughput inspection, IPS, malware prevention, and robust filtering, ensuring that all traffic—from internal vSAN communication to external WAN access—is securely managed without introducing bottlenecks.
The Transformation: Stability and Future Plans
The transition from the old HP cluster to this new Minisforum build is a night-and-day transformation. What used to be a source of constant performance throttles and instability (especially those PSOD-inducing USB adapters) is now a rock-solid foundation that I don’t even have to worry about. The giant revolution to 25GbE networking and dedicated compute silicon has delivered a quiet, stable, and incredibly potent datacenter-in-a-box.
Roadmap for Performance
The current bottleneck, if you can call it that, is the local vSAN NM790 Lexar drives. While fast, they are consumer drives. vSAN performance is sometimes throttled or „dampened” by the characteristics of these drives under sustained write operations.
To solve this, my future upgrade path is clear: I plan to change my vSAN drives to enterprise-grade Samsung 3.84TB PM9A3 server disks. This will not only increase capacity but significantly stabilize the IOPS and throughput, allowing vSAN ESA to operate at its absolute peak performance without drive-level dampening. This cluster is already a beast, but it’s still hungry for that true enterprise endurance.