Been a fun week.
Been see a lot of warningslike this lately
Still not clear on who ml314.com is but their cert is expired and blocking them seems to have no affect so I did. Same with the other expired certs that came up. If you see a pop up telling you cert is iffy block it. If you have other blocking tools add that domain.
Then I see this article when looking for why the extension got disabledGoogle cuts fake ad blockers from Chrome Store: Were you among 20 million fooled?
Adremover, the one I was using, was the most downloaded of the ad blockers in the extension store. It was downloaded by 10 million! It also was blocking so much stuff that I had to keep turning it off to use a lot of sites. I'm not sure how long Adremover was off before I noticed but I think all the expired cert warning started about the same time. So I'm not sure calling it a "fake" ad blocker is exactly accurate.
As a stop gap I upped my OpenDNS filtering to include "Adware" and "Hate/Discrimination" though it does not seem to be stopping all that much.
I upgraded my internet:With just my workstation plugged into the modem I'm not really seeing much of an improvement in download speed but upload is better than expect and that was my bottleneck. The cable guy's meter was showing over 900 Mbps down but I did notice it jumped up from ~350 to over 900 close to the end of the test so it is probably a burst speed more than a solid speed. This seems to confirm what I'm seeing. New Spectrum GIG no where near 940 Mbps But then I do not see buffering watching HD content even while running speed tests and downloading updates all at the same time so I don't think download speed will prove to be an issue anytime soon.
There was a bit of a mix up and Starz was not in the bundle I got so had to add that again. Still ended up adding Showtime, Cinemax and Starz plus doubling my upload speed for less than $20 more a month. Not bad. One weird bit though is I seem to have lost channel 511, HBO HD east. They have no clue why. Error says temp issue so maybe will sort on its own. Since I almost never watch live, picking up on the west feed is not biggie.
The new internet setup is kind of strange too. They bring out a modem AND a router. Seems you do not need to use their router and it does not have the phone interface so you need to keep the old modem/router as well just for the phone. Instead of using their router I think I'll stick a pfSense box instead. I was planning on adding one anyway later. I'll need to rewire some stuff though so I'll be on and off probably for the rest of this weekend.
I put one of my old NETGEAR Nighthawk R7000 from before my Unifi conversion in place while I got OPNsense installed This turned out to be VERY bad for throughput but at least I had some protection while I got OPNsense ready.
So now I'm setting up OPNsense.From what I'm reading in forums and groups, OPNsense is a fork / more user friendly version of pfSense so I decide to go with it. Also see pfSense® vs OPNsense®: technical comparison and this compare from a guy that installed both. One of the co-founders of pfSense joined Unifi awhile back and the additions they are now making to Unifi fall somewhere in between OPNsense and pfSense from what I as a user see. As I mention below though it seems unlikely current Unifi hardware will be able handle the load of processing high bandwidth data so going with a standalone box makes more sense for probably at least the next year.
I'm using a Dell 745 with an Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz (2 cores), CDROM drive, 2 TB HD and 8 GB RAM. It is just spare I had been using initially for OpenShift testing but is not really powerful enough for any real projects I do these days but well beyond what a Unifi USG can do with a dual-core, 500 MHz processor, 512 MB DDR2 RAM and 2 GB of storage.
Install went smooth (the occasional label did not match the instructions but it was generally obvious what to do next) except for one thing. The DVD image failed half way though coming up because it seemed to be unable to find the USB DVD drive I had booted from. I had even less luck with a USB stick image as they would not even boot but that might have more to do with the age of the PC I'm using. Note when installer the new machine I found the installer went through without error the second time but got the same CD not found error the first time so this might just be an intermittent issue.
Even more detailed instructions can be found in Building a BSD home router (pt. 5): Installing OPNsense
From the picture above you can see It performed well as a router even after turning off hardware offloading but when I turned on full IPS it took a real hit.
The CPU usage is frequently hitting 100% though so I think that is at least part of the problem. I'll need to try swapping in a Dell 780 with a Intel(R) Core(TM)2 Quad CPU Q9550 @ 2.83GHz box and see it that improves things. For now though it mainly seems to be affecting download and upload so not pressing. This is what the dashboard looks like (WAN IP blacked out here of course).
TrafficSo far the traffic reports look about the same as what I'd get from Unifi though in my case it helps me to filter just what is talking or not talking to the outside world. Otherwise Unifi pretty much wins here.
Now this is the bit I wanted OPNsense for. Unifi has started adding much of this in but with the CPU their routers have it just can't cope with processing this amount of data. Turning IPS on with the low end USG router people are reporting download speeds in the 50 Mbps range. As mentioned above even the Dual core I'm using for OPNsense is struggling with full IPS on (29905 rules) and drastically reduced max speeds. Granted that many rules is probably over kill. Especially when looking at the alert log and seeing almost all the hits were for this one rule
Unifi has a similar geo filter but currently you can only have on where OPNsense lets you have many.
Both let you get just alerts or drop packets. Note the above test had both set to alert instead of drop. Unifi set to alert only mode hardly affected throughput at all while block mode definitely does. With OPNsense it was not as clear because setting IPS mode on here
Seems drag down the system as if packets are being block even though the rules themselves have their "Action" set to Alert. Though looking at the above Unifi rule that would seem to imply it is blocking these packets yet OPNsense is still seeing them would imply it is working similar. But then too the only IPS alerts I've seen from Unifi since turning alerts on are:
IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 22.214.171.124:46524, to: 10.10.2.48:8078, protocol: TCP, in interface: eth1 11:30 04/18/2018
IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 126.96.36.199:46430, to: 10.10.2.46:8076, protocol: TCP, in interface: eth1 19:12 04/11/2018
An IP that appears to be in Netherlands though the attached phone number and some of the contact info is Czech.
My Unifi IPS setting look like this
Have to say I like the amount of stuff you can cram onto the console. Her I have it in 4 column mode and this is not even everything you can stick on there. Surprised there seems to be no way to hide or shrink the left nav menu as customizable as the rest is.
ssh accessSeems checking Permit password login does not seem to work. You still need to use ssh keys.
Adding speedtest to the OPNsesnse boxThis should work but did not for me
curl -Lo speedtest-cli https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py
chmod +x speedtest-cli
So as a work around I copy pasted into vi from a browser. Note you will need to change the first line from python to python2.7 either way so it can find the python command.
Run it like this
For keeping track create a cron job that writes to a csv fileAs root run (changing 16089 to the server ID you want to test against)
/root/speedtest-cli --server 16089 --csv-header > /usr/local/www/speedtest.csv
That sets the headers for you
Update it appears the crontab is getting overwritten so you may have to redo this from time to time.
To avoid this you need to create a /usr/local/opnsense/service/conf/actions.d/actions_speedtest.conf file and add
command:/root/speedtest-cli --server 16089 --csv >> /usr/local/www/speedtest.csv
description:Run a speed test
message:running speed test %s
After saving run
service configd restart
to load action file changes then
configctl speedtest test
to test it. You should get an OK as a response. Then add as a cron job through the web GUI like this
Sometimes the Speedtest.net extension just seems to hang when run while on the dashboard page.
As mentioned above check updated fails with IPS on. It appeared it was because the Core(TM)2 CPU is just not powerful enough to handle both at the same time. But I'm seeing the same with the Quad Core and the CPU is under 40% max.
I found a good starting point for rules in this pfSense post
Basically this sets:
Note on Snort. There are a lot of Snort rules. I filtered the list for "snort", selected all then enabled selected which enabled about half of them. I then also enabled Snort VRT/blacklist.
After reenabling IPS mode speed test and load looks like this
Note the CPU usage has actually gone up even though IPS is now off which makes no sense at all.
Web GUI seems to have gotten slugish after enabling IPS even when traffic and CPU is low. eventually the dropped packets got so bad I rebooted but it did not seem to help so as a last ditch attempt I repovered the box after checking the cards were all well seated. This seems to have helped at least for now. Speedtest with IPS off
And Speedtest with IPS back on.
Doing more research Seeing 0% packet loss after repower but back to 20% and other weirdness within an hour.
Swapped out the 1 PCIe x1 network card (WAN side) and the 2 PCI cards (the slots the mother board had open) with a 4 port PCIe x4 card in place of the unneeded video card in the PCIe x16 slot and things have improved a lot! The next morning I was still seeing 0% packet loss and higher speeds than I had connected to the modem directly.
Last night I even hit over 40 Mbps up.
This is with these plugins installed
|os-acme-client (orphaned)||1.13||221KiB||Let's Encrypt client|
|os-dyndns (orphaned)||1.6_1||134KiB||Dynamic DNS Support|
|os-intrusion-detection-content-snort-vrt (orphaned)||1.0||12.4KiB||IDS Snort VRT ruleset|
|os-smart (orphaned)||1.2||15.9KiB||SMART tools|
And IPS on with the same ET and Snort rule sets as before.
Update: 5/17/2018 This are looking better though the results to vary a good bit.
Update: 5/17/2018 This are looking better though the results to vary a good bit.
Some of that might be from all that else is going on though. For instance here
I also added a Pi-Hole DNS filter which has shifted some load from Chrome and the OPNsense. To give you an idea here is the dashboard showing the number of DNS requests it is handling.
instructions. Just in time too as the ad blocker Chrome extension I had been using got caught doing stuff.