Sample adventure

Still trying to sort this one out. So far no lightning I know of or chewed wires found.

First draft: to record details while getting things working again.

What happened:

• All of a sudden several cams (mainly connected to barn) go offline. 
• Go to try and reset power and find the WiFi plugs are offline. 
• Thinking main breaker in barn must have tripped but on inspection that is fine.
• Come back in and find home controller not responding, more cams offline and lots of red on MyMonitor screen. Can ping servers, all in next room, showing down but not talk to web interfaces on ones I try. Decide to sort barn first.
• Checking edge switches I can see the north and south walls of the barn are not talking
• Nothing appeared loose or chewed in exposed areas so ran temp line. (Note diff port on south wall switch so both lines available at north wall.)
• Most stuff came back online implying the line was bad in some way that was flooding the network with bad packets.
• Note barn east motion detector does not seem to be turning lights on.
• But the Etekcity plug powering cam6 in garage was still offline and needed repowered to work again. Getting plug online did not seem to sort cam6 though.
• The Inovelli dual plug switch controlling cams 15 and 58 seem to to have stop functioning. I can switch it manually, the status changes in screen and Z-seer says it is responding but it does not seem to respond to command via script or screen controls. 58 kept going off and on so moved off switch to see if that helped.
• IP cam 73 also on Inovelli dual plug switch in barn that also seems unresponsive to commands. Toggled power via the on button switch but cam still appeared down for several minutes then cam back. Oddly the status only updated for one of the plugs. The side that appears to still be supplying power. 

Some of this might be coincidence or even I just had not noticed but pretty sure cams 58 and 73 have been repowered via script recently. Both are temperamental though in the case of 58 it might be the switch. 73 is on WiFi and WiFi cams I've found just get confused and need resetting a lot.

I should note too even with all the stats in Unifi I'm still not seeing anything that would point to a packet storm though the way non barn things became unreachable till I unplugged the line in the barn would sure indicate that the bad line had something to do with the issue.

Note the final hump in the charts are where the issue happened. But the hump is there it is a good deal less than others shown. Though this is hourly summarized data. I wish I'd thought to grab 5 minute summarized data before it aged off. So I guess there could have been a spike of just few minutes that got normalized over the hour(s).

While writing this I've seen 73 go offline a few times while writing / updating this but it seems it did not connect to the closest AP for some reason so that might be the issue there. Forced to diff AP to see if that helps. 58 though seems more stable but still dropping offline some.

Looks like Netatmo stopped updating the server about the same time as the other mess too. Oddly some of the monitors built into Homeseer are still showing updates with no real pattern.  Rebooted the main indoor module and it seemed to sort itself out after a few minutes.

Update:
Still not sure what happened though I did experience something similar since on a smaller scale after a nearby lightning strike seemed to crash a few things despite them being on surge protectors. There too the Z-wave switch cam 58 was on seems to have died the day the above incident happened.

Update 3/18/2020:
I had replaced cam 58 with an upgrade and thought to use cam 58 to replace and older 1080p WiFi cam as it has been fairly stable running wired. Found if cam 58 runs with WiFi on it crashes after just a few minutes yet still works fine wired. Further implying some sort of EMP issue damaged some of its parts.

Be careful out there!

I ran across something this week you should probably be aware of. I was checking my Spectrum modem was not set to default passwords. Spectrum seems to have stopped setting them up with the factory default passwords. (While bad for people like me that like to monitor signal strength and know the password is changed it is probably a plus over all given how few seem to change their modems and routers from the defaults.) Anyway I googled the common ones used to see if the one (or ones) Spectrum is using might be online. The first link returned took me a page that looked like useful logins to try. But then I started getting warnings that this page was trying to probe my network. Without a paranoid level of network threat management I might not have even known. Note one of the things that set off alarms in my network was it was trying to use https to avoid packet inspection which, due to cert mismatch, triggered warnings. Another was one of addresses it tried to probe was the router for a test network, that while configured, has no devices on it at the moment.

Who cares? Well this is how some newer threats are getting past your firewall to your networked devices. They run probes on your network looking for devices with known exploits that they can then turn into a back doors into your network to take control of even more of your devices and even set them up to monitor your traffic. Then of course there are the malware cryptominers that that suck all your CPU. For an example of the level of adventure see https://www.grc.com/SN/SN-675-Notes.pdf

Link aggregation on Windows 10?

Basically this is to increase the throughput to a a higher end NAS like the Synology 12 bay NAS DiskStation DS2415+ (Diskless) I use. By adding a multi port board like Intel's Pro1000 (EXPI9404PTL) and using what Windows calls teaming you are supposed to be able to increase bandwidth between your workstation and NAS up to the number of ports linked on each end. For example with all 4 ports of the Synology linked and the 4 ports of the Intel card linked you should have a theoretical max bandwidth of 400 Gb/s which is almost local SATA speeds. But there are some catches not the least of which seems to be that Microsoft seems to be trying to force you to upgrade to server versions of their OS in order to use teaming. This means you have to hack around them to get it set up and upgrades may undo you workaround and have to be done again. This happened to me just this morning.

Doing some more research however shows that usually even with 4 ports available on both ends the link aggregation algorithm will only use one. A good explanation can be found here. To confirm this I set up my Synology with 2 ports linked on each of 2 networks and the workstation with 1 on one network (mounted as drive T)  and 4 teamed on the other (mounted as drive S).  LAN Speed Test showed no difference between using either network connection.

Though it also returned these results for a local 8 TB drive like is in the NAS (D:) and a Samsung 840 EVO mount locally as E: so I'm not sure that means much.

A better test might be doing a copy to from one folder on the NAS to another via the single workstation port

And via the 4 teamed ports

Note those are Bytes not bits per second. Not that much faster peaks though more consistent.

Just to confirm, after linking all 4 of the Synology ports together we do not see any more improvement in speed either because 4 port link aggregation really only provides 4 times the bandwidth to 4 or more devices.

Note the theoretical max for a  1 Gb/s port is 128 MB/s. 

So this seems to confirm teaming 2 connections might help some but it is unlikely you will get anything near a 2X bump. A third is unlikely to help at all unless you have high volume going to more than 2 places at once. And so on.

Just for fun I ran Crystal Disk Mark to get some comparative read/write times.
A Samsung Electronics 840 EVO-Series 250GB SSD

WD Red 8TB NAS Hard Disk Drive (what is in the NASs)

Synology 12 bay NAS DiskStation DS2415+ in SHR-2 mode

Drobo5n in SHR mode

Instructions for setting up teaming on Windows 10

So if after reading all the above you still want to team your ports here is what you do.

1) Download ProSet (PROwinx64.exe) Intel download site. Lookup “Intel Network Adapter Driver for windows 10” currently ver. 22.10

2) Run and install the exe (keep the file, you might need it again).

3) Find the temp folder the the drivers where put in (install does NOT install the ones we want) by looking for PROUnstl.exe in your users temp folder (C:\Users\{your username}\AppData\Local\Temp)

Use that to local the PRO1000 folder. Copy the PRO1000 folder to some where safe as you will be needing it often.

Adding teaming to Pro1000:

1) For each Ethernet port, go to properties->configure->driver and click “update driver button”

Note yours will not mention teaming if this is your first time through this process.

2) Choose “browse my computer for driver software” then “let me pick from a list…” then “Have desk”

3) Browse to where you copied the PROWin64 files and browse to \PRO1000\Winx64\NDIS65

4) You will see about 5 inf file there. Choose the 1st one (you will need to keep track of which one you used)

5) From the list of adapters scroll down and choose 82579V Gigabit Network Connection.

Note ignore this warning

6) Finally, repeat steps 3-8 for each of the 4 ports.

7) You’ll probably need to reboot at this point, after that you will see new teaming tabs in the Ethernet port properties.

8) For first Ethernet port, go to properties->configure->driver and click “update driver button”
You should now have a teaming tab. Select new Team is one is not already defined from previous run and select all the ports you want in the team.

You now have your ports teamed on your Windows 10 PC but as I said at the start you may need to come back and do this all over again after an update. Also this assumes you have also linked the ports on switch as well. For Unifi this would look like this.

Unifi Cloud Key not updating firmware?

Was trying to update from 5.22 to 5.29 and it kept returning without updating or saying what happened anywhere I could find. After Googling for a bit I found a few posts suggesting to move of trim the DB along the lines of this
The problem with disk space is usually because the DB is growing too large for the original install location. Since the Cloud Key unit ships with an 8GB SD card which mounts as /data (as you pointed out) I was able to solve this problem by moving the db folder to the SD card and then setting up a symbolic link to the new location. This provides much more storage space for the DB to grow and should free up space to handle future firmware updates. Here are the steps to do this:

## STOP SERVICES
sudo /etc/init.d/unifi stop
sudo /etc/init.d/cloudkey-webui stop


## MOVE DB AND SETUP SYMBOLIC LINK
sudo mv /srv/unifi/data/db /data
sudo ln -s /data/db/ /srv/unifi/data/db

## START SERVICES
sudo /etc/init.d/unifi start
sudo /etc/init.d/cloudkey-webui start

I then also ran the script located here to clean out older data from the DB:
https://help.ubnt.com/hc/en-us/articles/204911424-UniFi-Remove-prune-older-data-and-adjust-mongo-database-size


I am thinking about setting this script as a cron job to run periodically on its own, but I haven't done that yet. So far everything seems stable but your mileage may vary. Be sure to check the log files for any errors.

This got me thinking so I checked the available space:
ssh ubnt@keyIP (password is your admin user password or default ubnt)
df -h
But the only even close to full filesystem was .rofs
Filesystem                     Size  Used Avail Use% Mounted on
aufs-root                      1.1G  319M  735M  31% /
udev                            10M     0   10M   0% /dev
tmpfs                          201M  372K  201M   1% /run
/dev/disk/by-label/userdata    1.1G  319M  735M  31% /mnt/.rwfs
/dev/disk/by-partlabel/rootfs  291M  291M     0 100% /mnt/.rofs
tmpfs                          503M     0  503M   0% /dev/shm
tmpfs                          5.0M     0  5.0M   0% /run/lock
tmpfs                          503M     0  503M   0% /sys/fs/cgroup
tmpfs                          503M   96K  502M   1% /tmp
/dev/mmcblk0p7                 1.5G  1.2G  321M  78% /srv
/dev/mmcblk1p1                 7.2G  1.6G  5.7G  22% /data

So I tried just compacting the DB instead.

Note much of a change but it did the trick

Filesystem                     Size  Used Avail Use% Mounted on
aufs-root                      1.1G  319M  735M  31% /
udev                            10M     0   10M   0% /dev
tmpfs                          201M  372K  201M   1% /run
/dev/disk/by-label/userdata    1.1G  319M  735M  31% /mnt/.rwfs
/dev/disk/by-partlabel/rootfs  291M  291M     0 100% /mnt/.rofs
tmpfs                          503M     0  503M   0% /dev/shm
tmpfs                          5.0M     0  5.0M   0% /run/lock
tmpfs                          503M     0  503M   0% /sys/fs/cgroup
tmpfs                          503M   96K  502M   1% /tmp
/dev/mmcblk0p7                 1.5G 1000M  449M  70% /srv
/dev/mmcblk1p1                 7.2G  1.6G  5.7G  22% /data

Arris cable modem issue / securing your home network

If you have an Arris cable modem, especially if you have AT&T internet you should read this. It shows why you ALWAYS want a router between you and your cable modem.BTW TWC uses Arris too but unclear at this time is have same bugs.

To check to see if your modem is exposed even it not an Arris

The easiest way to check is to go to Steve Gibson's ShieldsUP and see if any ports are responding on your cable modem.
Check "All service ports" then a custom port scan on port 49955 61001 49152 8080

You want to see all green

Stuff to fix / check

Lastly if you have never changed your routers password or see ports open in the above tests from a browser go to the Arris web interface
Change the password if still using the default. (Basic settings -> login settings). Make it something random and at least 12 characters.

On LAN Settings Enable UPnP should be unchecked. (Allows stuff to open holes in your firewall.)

I would turn off the WiFi on the modem if you have any other access points. All WiFi should be encrypted no matter what provides it or be ahead of any router you use to protect your devices.

If you have something you need to access outside the home like a webcam first find and add your router to the reserved IP list so its address will not change.

Add a port forward at the modem to the router the cam (or device / PC) is on

If you have multiple cams / devices / ports you can forward a range like this.

You will then need to forward each port to the correct cam / device on your router. That is diff depending on brand and even version so you will need to goggle that bit if it is not obvious from the router's interface. Look for something that says port forwarding. For instance on a Unifi router the instructions look like this.

One last thing

You actually want to have a couple routers. One for your PCs and phones and one for all the rest of your stuff like Blu-ray players, cams, TiVos, home automation hubs .... All the things that might not be all that secure. As an added measure you set that router's firewall to only allow those things you KNOW need to call out to the internet to do so. There have been a lot of reports lately for instance of webcams being shipped with malware on them. If they can not call home or be connected to directly from outside they can not be used in bot nets or as platforms to attack your other devices.

Nest / WiFi Adventure

I've been merging my IoT and AV class C networks into 1 class B network so the automation can talk to the home entertainment devices easier and bring back order to my IP addresses for better filter of what has access to what. For the most part it has just been tedious updating the info close to 200 devices. While I was reconfiguring the house WiFi APs (Access Points) , I set them to enable roaming (AP swap you off to closest AP automatically) which in the case of Unifi APs assumes WAP2 encryption. Most devices handled this just fine but others had a real problem with it. For instance my wired (AC powered) Nest Protect continued to work fine but the battery powered one and the thermostat, in the house, both switched to the shop network I had not changed yet. So technically they were connected but not with enough signal to actually work. Some devices just reconnected no problem while others required a reboot, reconfig or, as in the case of the Nests, flat out refused to connect. So I've had to forget the roaming / zero hand off setup and use a configuration that support multiple encryption methods.