I let the router handle dhcp, while having it hand out each of the piholes IPs as DNS #1 and DNS #2. Those piholes settings are mirrored using Nebula-Sync, so any change made to one pihole gets copied over to the other automatically.

Both of those piholes are setup to wrap outbound DNS reqests in DOH as well, with the router blocking port 53 outbound. So all DNS traffic leaving the network has to be DOH/DOT or it just gets dropped. Currently using NextDNS and quad9 as my upstream DOH providers.

These were all running bcache on top of mdraid... One single mount.

Yes I understand how obnoxiously stupid it was to run RAID6 on a 48 disk volume. It was almost all just stuff I could re-acquire over time, not irreplaceable things.

I just HAD to solve this one though.

In come several spare chassis / mobo etc.... get a bunch of drives powered and on /dev/ , move to the next.

A couple spare gigabit switches...

several gigabit NICs...

two explicit paths for each machine...

a bit of iSCSI magic, and one machine now had the physical disks all exposed to it... mdadm --assemble blah blah, bit of UUID chaos...

It's surprising that while a bit speed limited (I think I got just around 110MB/sec reads), it was nicely performant for what a huge mess of wires and disks just strewn out around my rack.

Managed to evacuate all I needed without much issue once I got that going. Now, I try to keep my arrays under 16 drives at a time, or keep a very rigid policy of "I can lose this and don't care" vs "this box gets RAID10 and/or offsite backups nightly".

back up your critical stuff people!

I begin to try to diagnose from my phone with a hotspot to my computer. I’m getting warmer but some things just aren’t making sense. I go back and stare at the server running pi-hole. Then it fucking hits me.

I have the pretty odd and totally not ideal home network. I have plans to upgrade everything but currently I have two wifi networks tied to the same WAN. One network is run off of access points so I can get Ethernet to odd areas of the property. The Unraid server hosting pi-hole was connected to an AP and the Ethernet cable I used didn’t have a clip, so it just barely unseated itself from it, breaking the connection. The router+pi-hole config meant also breaking the internet if anything happened to pi-hole or that server. I swapped it for a better cable and it all instantly started working.

Spent that weekend sorting out the network so it doesn’t happen again. It was definitely a head-scratcher.

Is it a Threadripper? I built a couple of those for architectural rendering, such absolute beasts.

At the end, a strong unbranded Chinese double sided tape was better and still strong after all big western brand failed over time.

Maybe a mix of heat and uneven surface was the culprit, who knows.

After much investigation I discovered that restarting my router would solve it.

After some more weeks I discovered that the crawling bots were triggering some stateful tplink firewall that would close everything until rebooted.

So the router in the middle of the setup was the problem, not the edges.

I got sick of having to do this; so I replaced the power button with a transistor and a RPI. The PI would ping the server every 5min; if it failed to get a response 3 times in a row, it would trigger the transistor for 10sec, release for 3sec, then press and release again; forcing a poweroff then starting the machine again. It'd also write these events to a log file. I think it was around twice a week ish.

Never did figure out what the cause of the lockup was; but it stopped when I replaced windows with Debian.

Turns out it was in the basement. I forgot I moved it to be closer to my tilt Hydrometer.