Redundancy is not free, and anybody claiming otherwise is not doing it. Sending a job to five machines means paying for five machines. A decentralised network offering compute at the price of one centralised graphics card has either found something remarkable or is only sending your work to one machine. That cost is the product here, not an overhead on it.
It starts on two graphics cards on a desk. Genesis for the mesh is an RTX 3060 and an RTX 5070. That is not a figure of speech and it shapes real decisions: the first model we fine-tune is a small one, because it has to fit in twelve gigabytes. The larger variant the architecture wants becomes possible when enough other people have joined, which is a genuine dependency rather than a schedule.
Two mismatched cards are not a small cluster. Different generations, no direct interconnect. Using them together at all is real distributed-training engineering rather than a configuration file. We would rather say that than describe a devnet that sounds larger than it is.
And agreement is harder than it sounds. Comparing hashes only works if honest machines produce identical output, and floating-point work on different hardware does not do that by default. Pinned model and runtime versions and a fixed order of operations are part of every job for exactly this reason, and where bit-for-bit agreement turns out not to be achievable for some class of work, the honest fix is to change the rule rather than claim a precision the hardware does not have.