First draft: to record details while getting things working again.
What happened:
- All of a sudden several cams (mainly connected to barn) go offline.
- Go to try and reset power and find the WiFi plugs are offline.
- Thinking main breaker in barn must have tripped but on inspection that is fine.
- Come back in and find home controller not responding, more cams offline and lots of red on MyMonitor screen. Can ping servers, all in next room, showing down but not talk to web interfaces on ones I try. Decide to sort barn first.
- Checking edge switches I can see the north and south walls of the barn are not talking
- Nothing appeared loose or chewed in exposed areas so ran temp line. (Note diff port on south wall switch so both lines available at north wall.)
- Most stuff came back online implying the line was bad in some way that was flooding the network with bad packets.
- Note barn east motion detector does not seem to be turning lights on.
- But the Etekcity plug powering cam6 in garage was still offline and needed repowered to work again. Getting plug online did not seem to sort cam6 though.
- The Inovelli dual plug switch controlling cams 15 and 58 seem to to have stop functioning. I can switch it manually, the status changes in screen and Z-seer says it is responding but it does not seem to respond to command via script or screen controls. 58 kept going off and on so moved off switch to see if that helped.
- IP cam 73 also on Inovelli dual plug switch in barn that also seems unresponsive to commands. Toggled power via the on button switch but cam still appeared down for several minutes then cam back. Oddly the status only updated for one of the plugs. The side that appears to still be supplying power.
Some of this might be coincidence or even I just had not noticed but pretty sure cams 58 and 73 have been repowered via script recently. Both are temperamental though in the case of 58 it might be the switch. 73 is on WiFi and WiFi cams I've found just get confused and need resetting a lot.
I should note too even with all the stats in Unifi I'm still not seeing anything that would point to a packet storm though the way non barn things became unreachable till I unplugged the line in the barn would sure indicate that the bad line had something to do with the issue.
Note the final hump in the charts are where the issue happened. But the hump is there it is a good deal less than others shown. Though this is hourly summarized data. I wish I'd thought to grab 5 minute summarized data before it aged off. So I guess there could have been a spike of just few minutes that got normalized over the hour(s).
While writing this I've seen 73 go offline a few times while writing / updating this but it seems it did not connect to the closest AP for some reason so that might be the issue there. Forced to diff AP to see if that helps. 58 though seems more stable but still dropping offline some.
Looks like Netatmo stopped updating the server about the same time as the other mess too. Oddly some of the monitors built into Homeseer are still showing updates with no real pattern. Rebooted the main indoor module and it seemed to sort itself out after a few minutes.
Update:
Still not sure what happened though I did experience something similar since on a smaller scale after a nearby lightning strike seemed to crash a few things despite them being on surge protectors. There too the Z-wave switch cam 58 was on seems to have died the day the above incident happened.
Update 3/18/2020:
I had replaced cam 58 with an upgrade and thought to use cam 58 to replace and older 1080p WiFi cam as it has been fairly stable running wired. Found if cam 58 runs with WiFi on it crashes after just a few minutes yet still works fine wired. Further implying some sort of EMP issue damaged some of its parts.
No comments:
Post a Comment