Sample adventure

Still trying to sort this one out. So far no lightning I know of or chewed wires found.

First draft: to record details while getting things working again.

What happened:

• All of a sudden several cams (mainly connected to barn) go offline. 
• Go to try and reset power and find the WiFi plugs are offline. 
• Thinking main breaker in barn must have tripped but on inspection that is fine.
• Come back in and find home controller not responding, more cams offline and lots of red on MyMonitor screen. Can ping servers, all in next room, showing down but not talk to web interfaces on ones I try. Decide to sort barn first.
• Checking edge switches I can see the north and south walls of the barn are not talking
• Nothing appeared loose or chewed in exposed areas so ran temp line. (Note diff port on south wall switch so both lines available at north wall.)
• Most stuff came back online implying the line was bad in some way that was flooding the network with bad packets.
• Note barn east motion detector does not seem to be turning lights on.
• But the Etekcity plug powering cam6 in garage was still offline and needed repowered to work again. Getting plug online did not seem to sort cam6 though.
• The Inovelli dual plug switch controlling cams 15 and 58 seem to to have stop functioning. I can switch it manually, the status changes in screen and Z-seer says it is responding but it does not seem to respond to command via script or screen controls. 58 kept going off and on so moved off switch to see if that helped.
• IP cam 73 also on Inovelli dual plug switch in barn that also seems unresponsive to commands. Toggled power via the on button switch but cam still appeared down for several minutes then cam back. Oddly the status only updated for one of the plugs. The side that appears to still be supplying power. 

Some of this might be coincidence or even I just had not noticed but pretty sure cams 58 and 73 have been repowered via script recently. Both are temperamental though in the case of 58 it might be the switch. 73 is on WiFi and WiFi cams I've found just get confused and need resetting a lot.

I should note too even with all the stats in Unifi I'm still not seeing anything that would point to a packet storm though the way non barn things became unreachable till I unplugged the line in the barn would sure indicate that the bad line had something to do with the issue.

Note the final hump in the charts are where the issue happened. But the hump is there it is a good deal less than others shown. Though this is hourly summarized data. I wish I'd thought to grab 5 minute summarized data before it aged off. So I guess there could have been a spike of just few minutes that got normalized over the hour(s).

While writing this I've seen 73 go offline a few times while writing / updating this but it seems it did not connect to the closest AP for some reason so that might be the issue there. Forced to diff AP to see if that helps. 58 though seems more stable but still dropping offline some.

Looks like Netatmo stopped updating the server about the same time as the other mess too. Oddly some of the monitors built into Homeseer are still showing updates with no real pattern.  Rebooted the main indoor module and it seemed to sort itself out after a few minutes.

Update:
Still not sure what happened though I did experience something similar since on a smaller scale after a nearby lightning strike seemed to crash a few things despite them being on surge protectors. There too the Z-wave switch cam 58 was on seems to have died the day the above incident happened.

Update 3/18/2020:
I had replaced cam 58 with an upgrade and thought to use cam 58 to replace and older 1080p WiFi cam as it has been fairly stable running wired. Found if cam 58 runs with WiFi on it crashes after just a few minutes yet still works fine wired. Further implying some sort of EMP issue damaged some of its parts.

Lifx after a year

LIFX Mini White (A19) Wi-Fi Smart LED Light Bulb, Dimmable, Warm White, No Hub Required, Works with Alexa, Apple HomeKit and the Google Assistant

WiFi / Reliability

I bought one of these a year ago to see if it would work better in a area I have where RF devices seem to have issues. Hue bulbs and even Z-wave devices out there  Unfortunately the WiFi is so weak in these bulbs that it is no more reliable than the Hue bulb I had in the same place. Note Hue and WiFi operate in the same frequency range so strength of signal is the main deciding factor between them working reliably. Hue tends to operate at shorter distances but works in a mesh so the distance to the next bulb is more important than distance to the hub. How bad you ask. Here is the signal strength stats for it and a Wyze v2, which has the worse WiFi performance of any IP camera I've tried, sitting near each other. The Unifi Outdoor+ AP is less than 20 feet away though a single interior wall.

It also seems these bulbs can occasionally just lose WiFi connection and have to be setup again to get them back. Adding them back is fairly simple but you probably do not want to have deal with doing it without a serious feature to balance that out. Especially since you are paying extra.

Note after adding mine back in today I noticed there was a firmware update so I attempted update the bulb via the Android app. So far it has failed 4 times over 7 hours even with that 92% signal strength. I tried to get the Windows app too but it seems to be hanging at install. That might mean there are Lifx server issues. The bulb does seem to work fine from the Android app but not Homeseer till I also restarted the plugin.

Price

A 4 pack of Hue white bulbs will run you around $40 to $50.

As I write this the Mini White I have is going for $25 each. Even the cheapest used price is $14 so cost is not really a pro.

Oddly however if you want off white, LIFX Mini Day & Dusk (A19) Wi-Fi Smart LED Light Bulb is running a dollar cheaper right now.

I also thought about getting some of their LIFX + (BR30) Wi-Fi Smart LED Light Bulb with Infrared for Night Vision, Adjustable, Multicolor, Dimmable, No Hub Required, Works with Alexa, Apple HomeKit and Google Assistant bubs. In 4 packs they can be had for about $60 per. Last year they dropped as low as $46 per in a 4 pack. Still though unless you need color it is a hard sell given IR floods like the Univivi Infrared Illuminator, 850nm 6 LEDs 90 Degree Wide Angle IR Illuminator Night Vision,Waterproof LED Infrared Light IP Camera,CCTV Security Camera go for $23. If you are wanting color outdoor lighting though Lifx is competitive even if you do not need IR illumination. As a white flood / IR combo though a Z-wave switch on regular floods and adding the above IR flood would seem to be the most cost effective and reliable solution. Especially if you already have a home automation hub.

Running a camera on battery

What I wanted to achieve.

I wanted to get a cam on my mailbox which across the street. But the closest place to an outlet that could get line of sight is over 300 feet from any building and across the driveway. So we are talking major construction to run a line or POE cable out there. I already have a couple cams down by the creek running longer distances from my super AP ( a UniFi AP Outdoor+ with a Ubiquiti Airmax Omni AMO-2G10 10Dbi 2.4 GHz Rocket antenna) so I started to wonder what kind of battery would it take to power a camera.

Moved the body of this to my camera blog where it makes more sense after adding camera compare info. Leaving this here to make it easier to find.

IDS, IPS and general bad site blocking

Been a fun week. 

Been see a lot of warnings 

like this lately

Still not clear on who ml314.com is but their cert is expired and blocking them seems to have no affect so I did. Same with the other expired certs that came up. If you see a pop up telling you cert is iffy block it. If you have other blocking tools add that domain.

Then I saw a lot of warnings about a known JavaScript miner site being included in sites I'm visiting. Nothing that weird either. The site is t.hanstrackr.com After I explicitly blocked the JavaScript I got a warning it was trying an HTML method from my antivirus!

Then I see this article when looking for why the extension got disabled

 Google cuts fake ad blockers from Chrome Store: Were you among 20 million fooled?
Adremover, the one I was using, was the most downloaded of the ad blockers in the extension store. It was downloaded by 10 million! It also was blocking so much stuff that I had to keep turning it off to use a lot of sites. I'm not sure how long Adremover was off before I noticed but I think all the expired cert warning started about the same time. So I'm not sure calling it a "fake" ad blocker is exactly accurate.

As a stop gap I upped my OpenDNS filtering to include "Adware" and "Hate/Discrimination" though it does not seem to be stopping all that much.

I'm guessing that is because so many sites are moving code to sites on blocked lists and adding other blocker detector to try and force us to turn them off. Note I have no problem with ads as long as they are vetted for malware and do not slow the page loads to a crawl but responsible advertising seems to be mighty rare these days.

I upgraded my internet:

With just my workstation plugged into the modem I'm not really seeing much of an improvement in download speed but upload is better than expect and that was my bottleneck. The cable guy's meter was showing over 900 Mbps down but I did notice it jumped up from ~350 to over 900 close to the end of the test so it is probably a burst speed more than a solid speed. This seems to confirm what I'm seeing. New Spectrum GIG no where near 940 Mbps But then I do not see buffering watching HD content even while running speed tests and downloading updates all at the same time so I don't think download speed will prove to be an issue anytime soon.

There was a bit of a mix up and Starz was not in the bundle I got so had to add that again. Still ended up adding Showtime, Cinemax and Starz plus doubling my upload speed for less than $20 more a month. Not bad. One weird bit though is I seem to have lost channel 511, HBO HD east. They have no clue why. Error says temp issue so maybe will sort on its own. Since I almost never watch live, picking up on the west feed is not biggie.

The new internet setup is kind of strange too. They bring out a modem AND a router. Seems you do not need to use their router and it does not have the phone interface so you need to keep the old modem/router as well just for the phone. Instead of using their router I think I'll stick a pfSense box instead. I was planning on adding one anyway later. I'll need to rewire some stuff though so I'll be on and off probably for the rest of this weekend.

I put one of my old NETGEAR Nighthawk R7000 from before my Unifi conversion in place while I got OPNsense installed This turned out to be VERY bad for throughput but at least I had some protection while I got OPNsense ready.

So now I'm setting up OPNsense.

From what I'm reading in forums and groups, OPNsense is a fork / more user friendly version of pfSense  so I decide to go with it. Also see pfSense® vs OPNsense®: technical comparison and this compare from a guy that installed both. One of the co-founders of pfSense joined Unifi awhile back and the additions they are now making to Unifi fall somewhere in between OPNsense and pfSense from what I as a user see. As I mention below though it seems unlikely current Unifi hardware will be able handle the load of processing high bandwidth data so going with a standalone box makes more sense for probably at least the next year.

I'm using a Dell 745 with an Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz (2 cores), CDROM drive, 2 TB HD and 8 GB RAM. It is just spare I had been using initially for OpenShift testing but is not really powerful enough for any real projects I do these days but well beyond what a Unifi USG can do with a dual-core, 500 MHz processor, 512 MB DDR2 RAM and 2 GB of storage.

Install went smooth (the occasional label did not match the instructions but it was generally obvious what to do next) except for one thing. The DVD image failed half way though coming up because it seemed to be unable to find the USB DVD drive I had booted from. I had even less luck with a USB stick image as they would not even boot but that might have more to do with the age of the PC I'm using. Note when installer the new machine I found the installer went through without error the second time but got the same CD not found error the first time so this might just be an intermittent issue.

Even more detailed instructions can be found in Building a BSD home router (pt. 5): Installing OPNsense

From the picture above you can see It performed well as a router even after turning off hardware offloading but when I turned on full IPS it took a real hit.

The CPU usage is frequently hitting 100% though so I think that is at least part of the problem. I'll need to try swapping in a Dell 780 with a  Intel(R) Core(TM)2 Quad CPU Q9550  @ 2.83GHz box and see it that improves things. For now though it mainly seems to be affecting download and upload so not pressing. This is what the dashboard looks like (WAN IP blacked out here of course).

Just unchecking  IPS mode on the OPNsense Services: Intrusion Detection: Administration page, but leaving IDS enabled, pretty much gets us back to normal.

Having IPS on was also causing check for updates to fail.

Reports

Traffic

So far the traffic reports look about the same as what I'd get from Unifi though in my case it helps me to filter just what is talking or not talking to the outside world. Otherwise Unifi pretty much wins here.

IPS 

Now this is the bit I wanted OPNsense for. Unifi has started adding much of this in but with the CPU their routers have it just can't cope with processing this amount of data. Turning IPS on with the low end USG router people are reporting download speeds in the 50 Mbps range. As mentioned above even the Dual core I'm using for OPNsense is struggling with full IPS on (29905 rules) and drastically reduced max speeds. Granted that many rules is probably over kill. Especially when looking at the alert log and seeing almost all the hits were for this one rule

Unifi has a similar geo filter but currently you can only have on where OPNsense lets you have many.

Both let you get just alerts or drop packets. Note the above test had both set to alert instead of drop. Unifi set to alert only mode hardly affected throughput at all while block mode definitely does. With OPNsense it was not as clear because setting IPS mode on here

Seems drag down the system as if packets are being block even though the rules themselves have their "Action" set to Alert. Though looking at the above Unifi rule that would seem to imply it is blocking these packets yet OPNsense is still seeing them would imply it is working similar. But then too the only IPS alerts I've seen from Unifi since turning alerts on are:

  EVENT TIME

IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 191.101.167.73:46524, to: 10.10.2.48:8078, protocol: TCP, in interface: eth1 11:30 04/18/2018

IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 191.101.167.37:46430, to: 10.10.2.46:8076, protocol: TCP, in interface: eth1 19:12 04/11/2018

An IP that appears to be in Netherlands though the attached phone number and some of the contact info is Czech.

My Unifi IPS setting look like this

Console

Have to say I like the amount of stuff you can cram onto the console. Her I have it in 4 column mode and this is not even everything you can stick on there. Surprised there seems to be no way to hide or shrink the left nav menu as customizable as the rest is.

ssh access

Seems checking Permit password login does not seem to work. You still need to use ssh keys.

Adding speedtest to the OPNsesnse box

This should work but did not for me
curl -Lo speedtest-cli https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py
chmod +x speedtest-cli

So as a work around I copy pasted into vi from a browser. Note you will need to change the first line from python to python2.7 either way so it can find the python command.
Run it like this

For keeping track create a cron job that writes to a csv file

As root run (changing 16089 to the server ID you want to test against)
/root/speedtest-cli --server 16089 --csv-header > /usr/local/www/speedtest.csv
That sets the headers for you
Then set up a cron job to run with
crontab -e
Note it uses the vi editor so you need to know the right commands for that. Should look like this

Update it appears the crontab is getting overwritten so you may have to redo this from time to time.
To avoid this you need to create a /usr/local/opnsense/service/conf/actions.d/actions_speedtest.conf file and add

[test]
command:/root/speedtest-cli --server 16089 --csv >> /usr/local/www/speedtest.csv
description:Run a speed test
parameters:%s
type:script

message:running speed test %s

After saving run
service configd restart
to load action file changes then
configctl speedtest test
to test it. You should get an OK as a response. Then add as a cron job through the web GUI like this

This should add a line to the crontab but does not seem too. Nor did it run at first even after reboot but then started running later. No obvious reason why but the line I manually added to the crontab disappeared about the same time. Still not adding a line in crontab though so being handled else where.

Once it is running you can download the speedtests results file by pointing your browser to https://192.168.0.1/speedtest.csv (assuming your web interface it on 192.168.0.1 of course)

Odd bits

The LAN interface switched from 192.168.0.1, the static assigned, to 192.168.0.201, a DHCP address, at one point for no obvious reason.

Sometimes the Speedtest.net extension just seems to hang when run while on the dashboard page.

As mentioned above check updated fails with IPS on. It appeared it was because the Core(TM)2 CPU is just not powerful enough to handle both at the same time. But I'm seeing the same with the Quad Core and the CPU is under 40% max.

I found a good starting point for rules in this pfSense post
Basically this sets:

• emerging-drop
• emerging-botcc.portgrouped
• emerging-botcc
• emerging-ciarmy
• emerging-compromised
• emerging-dshield
• emerging-tor
• emerging-worm
• emerging-trojan
• emerging-mobile_malware
• emerging-malware
• snort

Note on Snort. There are a lot of Snort rules. I filtered the list for "snort", selected all then enabled selected which enabled about half of them. I then also enabled Snort VRT/blacklist.
After reenabling IPS mode speed test and load looks like this

Note the much improved CPU usage. That upload is way low but after turning off IPS I got an even worse run and then a slightly better one which makes me wonder if the speed tests are telling me anything. External factors must be throwing them off or something is causing flux in the OPNsense box I can't see.

Note the CPU usage has actually gone up even though IPS is now off which makes no sense at all.

Web GUI seems to have gotten slugish after enabling IPS even when traffic and CPU is low. eventually the dropped packets got so bad I rebooted but it did not seem to help so as a last ditch attempt I repovered the box after checking the cards were all well seated. This seems to have helped at least for now. Speedtest with IPS off

And Speedtest with IPS back on.

Doing more research Seeing 0% packet loss after repower but back to 20% and other weirdness within an hour.

Swapped out the 1 PCIe x1 network card (WAN side) and the 2 PCI cards (the slots the mother board had open) with a 4 port PCIe x4 card in place of the unneeded video card in the PCIe x16 slot and things have improved a lot! The next morning I was still seeing 0% packet loss and higher speeds than I had connected to the modem directly.

Last night I even hit over 40 Mbps up.

This is with these plugins installed

os-acme-client (orphaned)	1.13	221KiB	Let's Encrypt client
os-dyndns (orphaned)	1.6_1	134KiB	Dynamic DNS Support
os-intrusion-detection-content-snort-vrt (orphaned)	1.0	12.4KiB	IDS Snort VRT ruleset
os-smart (orphaned)	1.2	15.9KiB	SMART tools

And IPS on with the same ET and Snort rule sets as before.

Update: 5/17/2018 This are looking better though the results to vary a good bit.

Some of that might be from all that else is going on though. For instance here

There is about 40 mbps down and 5-6 mbps up going on besides the speed test. The fastest download test I was only seeing 8 mbps up because of how fast the security cams were triggering and transferring alert pics to the cloud that day.

I also added a Pi-Hole DNS filter which has shifted some load from Chrome and the OPNsense. To give you an idea here is the dashboard showing the number of DNS requests it is handling.

Simple to set up with these instructions. Just in time too as the ad blocker Chrome extension I had been using got caught doing stuff.

Short, quick and simplest version of install:

command line commands in bold-italics

Format SD card with SDFormatterv4
Get NOOBS OS installer as zip
Copy NOOBS zip contents to SD card
Boot rpi with SD
Install Raspbian
When done ssh from another PC
ssh pi@192.168.0.62 (password raspberry)
or open terminal window on the pi's console
Run command
sudo raspi-config 
Change the password
Optionally change the host name.
Switch to root user
sudo bash
and run these commands
apt-get update
apt-get upgrade
wget -O basic-install.sh https://install.pi-hole.net
bash basic-install.sh
Note password at end of line
   [i] Web Interface password: 
you will need it to login to the admin web interface.

Lastly you should backup your config any time you make changes like adding sites to a whitelist. If you are recovering from a crash (this has happened a couple times to me) you can also import your config from this screen (Settings->Teleporter) as well.

Update 7/7/2019 If you have Unifi
Check out [Step-By-Step Tutorial/Guide] Raspberry Pi with UniFi Controller and Pi-hole from scratch (headless)  Easy setup, about the same cost as a Cloudkey Gen 1,  About half that of a Gen2, works slick and can give you the space to turn on recording of Clients Historical Data that is tough on a Cloudkey Gen 1. My Raspberry Pi 3B+ seems to have no problem running both programs. I did bump the SD to 32GB. I've just turned on recording of Clients Historical Data so we'll have to see how much storage gets taken up by that.

Update 8/5:
After running the Unifi controller / Pi-hole combo for a couple weeks I'm getting a bit concerned. It has crashed twice now drastically degrading network performance since DNS goes offline. I had planned on taking the original Pi-hole offline but now I'm thinking I need to keep it up and config it as a backup.

Update 12/27:
Between logs, the Pi-Hole DB, the Unifi backups and the update files SD card keeps filling up regularly.
For example I had all these files over 100 MBs after doing some clean up
root@UnifiPi:/etc/pihole# find / -size +100M -ls |
   287694 138240 -rwxr-xr-x   1 root     root     141550912 Jul 10 07:27 /usr/lib/chromium-browser/chromium-browser
   398403 122876 -rw-r--r--   1 root     root     125817618 Oct 18 13:28 /usr/lib/jvm/java-11-openjdk-armhf/lib/modules
   278090 123076 -rw-r--r--   1 root     root     126023659 Oct 18 13:28 /usr/lib/jvm/java-11-openjdk-armhf/jmods/java.base.jmod
  1582114 10821252 -rw-r--r--   1 pihole   pihole   11080900608 Dec 27 20:03 /etc/pihole/pihole-FTL.db
  1835088   318560 -rw-r-----   1 root     adm        326197642 Dec  2 00:00 /var/log/daemon.log.1
  1835105   979680 -rw-r-----   1 root     adm       1003184946 Dec 27 19:43 /var/log/daemon.log
  1835084   277436 -rw-r--r--   1 pihole   pihole     284087304 Dec  1 00:05 /var/log/pihole.log.1
  1835011   287236 -rw-r-----   1 root     adm        294126053 Dec 27 20:00 /var/log/syslog
   662490   131076 -rw-------   1 unifi    unifi      134217728 Sep  5 09:38 /var/lib/unifi/db/_tmp_repairDatabase_last/ace_stat.3
   662484   131076 -rw-------   1 unifi    unifi      134217728 Oct 14 23:33 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_7/ace_stat.3
   662486   262144 -rw-------   1 unifi    unifi      268435456 Oct 14 23:33 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_7/ace_stat.4
   662483   524036 -rw-------   1 unifi    unifi      536608768 Oct 14 23:32 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_7/ace_stat.5
   662497   131072 -rw-------   1 unifi    unifi      134217728 Nov  4 03:47 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_5/ace_stat.3
   662499   262144 -rw-------   1 unifi    unifi      268435456 Nov  4 03:47 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_5/ace_stat.4
   662496   524032 -rw-------   1 unifi    unifi      536608768 Nov  4 03:46 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_5/ace_stat.5
   662501   262144 -rw-------   1 unifi    unifi      268435456 Sep  5 09:38 /var/lib/unifi/db/_tmp_repairDatabase_last/ace_stat.4
   662489   524036 -rw-------   1 unifi    unifi      536608768 Sep  5 09:37 /var/lib/unifi/db/_tmp_repairDatabase_last/ace_stat.5
   662506   131072 -rw-------   1 unifi    unifi      134217728 Dec 27 19:59 /var/lib/unifi/db/ace_stat.3
   662503   131072 -rw-------   1 unifi    unifi      134217728 Dec 27 20:02 /var/lib/unifi/db/ace.3
   662527   131076 -rw-------   1 unifi    unifi      134217728 Nov 25 13:15 /var/lib/unifi/db/_tmp_repairDatabase_1/ace_stat.3
   662529   262144 -rw-------   1 unifi    unifi      268435456 Nov 25 13:15 /var/lib/unifi/db/_tmp_repairDatabase_1/ace_stat.4
   662526   524036 -rw-------   1 unifi    unifi      536608768 Nov 25 13:13 /var/lib/unifi/db/_tmp_repairDatabase_1/ace_stat.5
   662520   262148 -rw-------   1 unifi    unifi      268435456 Dec 27 20:00 /var/lib/unifi/db/ace_stat.4
   662519   262148 -rw-------   1 unifi    unifi      268435456 Nov 25 23:59 /var/lib/unifi/db/ace.4
   662464   131072 -rw-------   1 unifi    unifi      134217728 Nov 22 00:04 /var/lib/unifi/db/_tmp_repairDatabase_0/ace_stat.3
   662466   262148 -rw-------   1 unifi    unifi      268435456 Nov 22 00:05 /var/lib/unifi/db/_tmp_repairDatabase_0/ace_stat.4
   662463   524036 -rw-------   1 unifi    unifi      536608768 Nov 22 00:04 /var/lib/unifi/db/_tmp_repairDatabase_0/ace_stat.5
   662505   524036 -rw-------   1 unifi    unifi      536608768 Dec 27 17:06 /var/lib/unifi/db/ace_stat.5
   662515   131072 -rw-------   1 unifi    unifi      134217728 Nov 29 00:05 /var/lib/unifi/db/_tmp_repairDatabase_2/ace_stat.3
   662517   262144 -rw-------   1 unifi    unifi      268435456 Nov 29 00:05 /var/lib/unifi/db/_tmp_repairDatabase_2/ace_stat.4
   662514   524036 -rw-------   1 unifi    unifi      536608768 Nov 29 00:03 /var/lib/unifi/db/_tmp_repairDatabase_2/ace_stat.5
   662675   262144 -rw-------   1 mongodb  nogroup    268435456 Jun 30  2019 /var/lib/mongodb/journal/prealloc.2
   662677   262144 -rw-------   1 mongodb  nogroup    268435456 Jun 30  2019 /var/lib/mongodb/journal/prealloc.1
   662676   262144 -rw-------   1 mongodb  nogroup    268435456 Dec 27 18:37 /var/lib/mongodb/journal/j._0
  4194815   178776 -rw-r--r--   1 root     root       183066184 Oct 22 23:08 /var/cache/apt/archives/openjdk-11-jdk-headless_11.0.5+10-1~deb10u1_armhf.deb

  4194330   172596 -rw-r--r--   1 root     root       176736120 Jun 25  2019 /var/cache/apt/archives/openjdk-11-jdk-headless_11.0.3+7-5_armhf.deb

So I'm swapping the SD to a 128GB one using these instructions. Note if you installed via nobs like I did you will need to use the instructions to make a new partition instead of expanding root. I used these variations:
parted
mkfs.ext4 /dev/mmcblk0p3
e2label /dev/mmcblk0p3 var
vi /etc/fstab
/dev/mmcblk0p3  /var2            ext4    defaults          1       2
mkdir /var2
mount /var2
vi /etc/fstab
/dev/mmcblk0p3  /var            ext4    defaults          1       2
mkdir /vb
systemctl stop unifi.service
/etc/init.d/pihole-FTL stop
cd /var
find . | cpio -pvmd /var2
cd /
mv /var /vb
mkdir /var
mount /var
Then just to be sure
df
New FS should show up as /var
reboot
Everything should not restart OK and you can start deleting stuff from /vb

This gets Unifi and Pi-Hole in separate file systems. Mine came out like this
Filesystem     1K-blocks     Used            Available Use% Mounted on 
/dev/root              28776316 25388648    1902868  94% /                   
devtmpfs                  469544               0      469544   0% /dev             
tmpfs                        474152         1580      472572   1% /dev/shm     
tmpfs                        474152         6488      467664   2% /run             
tmpfs                            5120               4          5116   1% /run/lock     
tmpfs                        474152               0      474152   0% /sys/fs/cgroup
/dev/mmcblk0p3  91611060 12665804  74248568  15% /var             
/dev/mmcblk0p6      258094       53035      205060  21% /boot           
tmpfs                          94828               0        94828   0% /run/user/1000
tmpfs                          94828               0        94828   0% /run/user/999  

Update 3/6/2020: Still running out of space on / which causes the Pi to crash

This is caused by /etc/pihole/pihole-FTL.db getting huge. Mine was 15,561 MB when the system ran out of space.
By default the DB keeps up to a years worth of data. You can either reduce this by adding MAXDBDAYS to /etc/pihole/pihole-FTL.conf to reduce the days or probably better solution moving the db onto the /var partition created earlier. To do that add DBFILE=/var/pihole/pihole-FTL.db to /etc/pihole/pihole-FTL.conf then run
mkdir /var/pihole
/etc/init.d/pihole-FTL stop
cp /etc/pihole/pihole-FTL.db /var/pihole/

Note this takes awhile. (My 15 GB file took about 30 minutes!) Once it is done:

/etc/init.d/pihole-FTL stop
ls -lapi /var/pihole/
Once you see it is updating the DB you can remove the old DB with
rm /etc/pihole/pihole-FTL.db

Now I have a lot more space on root and /var hardly sees the increase.

Update 7/23/2020: Still running out of space the old PiHole so stuck a 128 in there too.

But then tried to update and discovered the OS was too old. Fortunately it is not that hard, though lengthy, process to upgrade the OS in place. Here are the instructions I used.

Link aggregation on Windows 10?

Basically this is to increase the throughput to a a higher end NAS like the Synology 12 bay NAS DiskStation DS2415+ (Diskless) I use. By adding a multi port board like Intel's Pro1000 (EXPI9404PTL) and using what Windows calls teaming you are supposed to be able to increase bandwidth between your workstation and NAS up to the number of ports linked on each end. For example with all 4 ports of the Synology linked and the 4 ports of the Intel card linked you should have a theoretical max bandwidth of 400 Gb/s which is almost local SATA speeds. But there are some catches not the least of which seems to be that Microsoft seems to be trying to force you to upgrade to server versions of their OS in order to use teaming. This means you have to hack around them to get it set up and upgrades may undo you workaround and have to be done again. This happened to me just this morning.

Doing some more research however shows that usually even with 4 ports available on both ends the link aggregation algorithm will only use one. A good explanation can be found here. To confirm this I set up my Synology with 2 ports linked on each of 2 networks and the workstation with 1 on one network (mounted as drive T)  and 4 teamed on the other (mounted as drive S).  LAN Speed Test showed no difference between using either network connection.

Though it also returned these results for a local 8 TB drive like is in the NAS (D:) and a Samsung 840 EVO mount locally as E: so I'm not sure that means much.

A better test might be doing a copy to from one folder on the NAS to another via the single workstation port

And via the 4 teamed ports

Note those are Bytes not bits per second. Not that much faster peaks though more consistent.

Just to confirm, after linking all 4 of the Synology ports together we do not see any more improvement in speed either because 4 port link aggregation really only provides 4 times the bandwidth to 4 or more devices.

Note the theoretical max for a  1 Gb/s port is 128 MB/s. 

So this seems to confirm teaming 2 connections might help some but it is unlikely you will get anything near a 2X bump. A third is unlikely to help at all unless you have high volume going to more than 2 places at once. And so on.

Just for fun I ran Crystal Disk Mark to get some comparative read/write times.
A Samsung Electronics 840 EVO-Series 250GB SSD

WD Red 8TB NAS Hard Disk Drive (what is in the NASs)

Synology 12 bay NAS DiskStation DS2415+ in SHR-2 mode

Drobo5n in SHR mode

Instructions for setting up teaming on Windows 10

So if after reading all the above you still want to team your ports here is what you do.

1) Download ProSet (PROwinx64.exe) Intel download site. Lookup “Intel Network Adapter Driver for windows 10” currently ver. 22.10

2) Run and install the exe (keep the file, you might need it again).

3) Find the temp folder the the drivers where put in (install does NOT install the ones we want) by looking for PROUnstl.exe in your users temp folder (C:\Users\{your username}\AppData\Local\Temp)

Use that to local the PRO1000 folder. Copy the PRO1000 folder to some where safe as you will be needing it often.

Adding teaming to Pro1000:

1) For each Ethernet port, go to properties->configure->driver and click “update driver button”

Note yours will not mention teaming if this is your first time through this process.

2) Choose “browse my computer for driver software” then “let me pick from a list…” then “Have desk”

3) Browse to where you copied the PROWin64 files and browse to \PRO1000\Winx64\NDIS65

4) You will see about 5 inf file there. Choose the 1st one (you will need to keep track of which one you used)

5) From the list of adapters scroll down and choose 82579V Gigabit Network Connection.

Note ignore this warning

6) Finally, repeat steps 3-8 for each of the 4 ports.

7) You’ll probably need to reboot at this point, after that you will see new teaming tabs in the Ethernet port properties.

8) For first Ethernet port, go to properties->configure->driver and click “update driver button”
You should now have a teaming tab. Select new Team is one is not already defined from previous run and select all the ports you want in the team.

You now have your ports teamed on your Windows 10 PC but as I said at the start you may need to come back and do this all over again after an update. Also this assumes you have also linked the ports on switch as well. For Unifi this would look like this.