Or go a little further and spend ~$200 on a used passive FDR Infiniband switch, $50 per used dual port FDR IB NIC, $40 for a 10 meter fibre optic cable including transceivers at each end.
Then run IP over IB on each host and you have a 56 Gbit network that all your applications will just see as another network interface on each host.
for home use I'd highly recommend sticking with just 10GbE because you're not locking yourself into a dead-end solution of used weird previous gen infiniband stuff.
if you get a $200 switch with a few 10GbE interfaces in it you can easily expand things in the future by trunking vlans to another newer 10GbE capable switch, or connecting to a switch that has multi-gig copper ports for access ports to 2.5/5GBaseT capable desktop PCs and laptops, etc.
$40 for a 10 meter fiber optic cable is a high price when you can buy LC-LC UPC 9/125 duplex 2 meter cables for $3.50 to $4.70 a piece (or a few cents more for additional meters) and connect them between $20 transceivers. no matter what route someone goes with would recommend buying $30-40 of basic fiber connector cleaning supplies.
if one wants to buy used weird previous gen dead-end stuff there are also tons of very cheap 40GbE mellanox ethernet adapters on ebay with the QSFP to go with them, and if you have a place to put a switch that doesn't matter if it's noisy like in a wiring closet somewhere, cheap 1U switches with 40GbE ethernet ports on them that can also be used as individual 10GbE when broken out.
Can you share what your experience with implementing IPoIB with used gear was? I'm asking mainly because I actually got interested recently with such setups however I got rather discouraged by the driver support.
It seems that some decent support only exists for more recent generations. The older ones like ConnectX-3 or earlier, which typically show up on ebay are either not supported any more or maybe available for older kernel versions and soon to be EOLed.
So do I understand it correctly that to use such adapters one has to actually downgrade to an older kernel version?
Or is there some basic support in the latest Linux kernels for older generations still?
Yes, if you want to use the officially supported driver for ConnectX-3 (mlx4_xxx kernel modules, LTS release of v4.9 available from Nvidia's page), you need to go with something like Ubuntu 20.04 LTS (which should be good until at least end of 2025). However, the latest Mellanox drivers (mlx5_xxx kernel modules) work just fine with the ConnectX-3, at least for basic functionality.
I've not actually used IPoIB on such gear myself, but we have been working quite a bit on reusing old/ancient HPC clusters with IB adapters, and you can generally make things work if you spend enough time on trial and error and you are not afraid of compiling code with complicated dependencies. As long as you can get the IB stuff talking, and the driver is using OFED, the IPoIB part should Just Work.
It is always going to be an adventure working with used gear. But HPC has such a high decommissioning tempo and low resale value that there will always be quite a few other enthusiasts toying about.
I bought brand new 10 Gbps NICs ($47/pcs) and switch ($120) for less than that and replacements are readily available. The 20m AOC cables were indeed $40.
A NAS home build of this size will never exceed 10 Gbps, I barely get ~ 2 Gbps out of the spinning disks.
the main application where one will see real 10Gbps speeds is if the NAS also has a fast NVME SSD used for smaller file size needs at high speeds...
for instance I have a setup which is meant for working with uncompressed raw yuv420 or yuv422p 1080p and 4K video, there's a 512GB NVME SSD and a 1TB SSD set up as individual JBOD and exposed to the network for video editing scratch file storage, and it will definitely saturate 10GbE.
this is actually needlessly complicated, if/when I build a more powerful desktop pc again I'm just going to put the same work file space storage on a 2TB NVME SSD directly stuck into the motherboard.
Very valid point, but I don't know how fast you will hit the chipset speed limits; that NIC is connected to the chipset that is connected to the CPU via a 4x PCIe link and from there to the nVME with 4x PCIe link. In theory you have 4 or 8 GB/sec max bandwidth, but the CPU-chipset link is not dedicated. If you go for Threadripper the math is very different, you have lots of direct CPU connections for (multiple) NIC and multiple nVME.
you may be referencing the original post's cpu/motherboard combo which is not the same as the pci-e bus/lanes, slot and setup on my home file server and desktop PC.
Then run IP over IB on each host and you have a 56 Gbit network that all your applications will just see as another network interface on each host.