IDS, IPS and general bad site blocking

Been a fun week. 

Been see a lot of warnings 

like this lately

Still not clear on who ml314.com is but their cert is expired and blocking them seems to have no affect so I did. Same with the other expired certs that came up. If you see a pop up telling you cert is iffy block it. If you have other blocking tools add that domain.

Then I saw a lot of warnings about a known JavaScript miner site being included in sites I'm visiting. Nothing that weird either. The site is t.hanstrackr.com After I explicitly blocked the JavaScript I got a warning it was trying an HTML method from my antivirus!

Then I see this article when looking for why the extension got disabled

 Google cuts fake ad blockers from Chrome Store: Were you among 20 million fooled?
Adremover, the one I was using, was the most downloaded of the ad blockers in the extension store. It was downloaded by 10 million! It also was blocking so much stuff that I had to keep turning it off to use a lot of sites. I'm not sure how long Adremover was off before I noticed but I think all the expired cert warning started about the same time. So I'm not sure calling it a "fake" ad blocker is exactly accurate.

As a stop gap I upped my OpenDNS filtering to include "Adware" and "Hate/Discrimination" though it does not seem to be stopping all that much.

I'm guessing that is because so many sites are moving code to sites on blocked lists and adding other blocker detector to try and force us to turn them off. Note I have no problem with ads as long as they are vetted for malware and do not slow the page loads to a crawl but responsible advertising seems to be mighty rare these days.

I upgraded my internet:

With just my workstation plugged into the modem I'm not really seeing much of an improvement in download speed but upload is better than expect and that was my bottleneck. The cable guy's meter was showing over 900 Mbps down but I did notice it jumped up from ~350 to over 900 close to the end of the test so it is probably a burst speed more than a solid speed. This seems to confirm what I'm seeing. New Spectrum GIG no where near 940 Mbps But then I do not see buffering watching HD content even while running speed tests and downloading updates all at the same time so I don't think download speed will prove to be an issue anytime soon.

There was a bit of a mix up and Starz was not in the bundle I got so had to add that again. Still ended up adding Showtime, Cinemax and Starz plus doubling my upload speed for less than $20 more a month. Not bad. One weird bit though is I seem to have lost channel 511, HBO HD east. They have no clue why. Error says temp issue so maybe will sort on its own. Since I almost never watch live, picking up on the west feed is not biggie.

The new internet setup is kind of strange too. They bring out a modem AND a router. Seems you do not need to use their router and it does not have the phone interface so you need to keep the old modem/router as well just for the phone. Instead of using their router I think I'll stick a pfSense box instead. I was planning on adding one anyway later. I'll need to rewire some stuff though so I'll be on and off probably for the rest of this weekend.

I put one of my old NETGEAR Nighthawk R7000 from before my Unifi conversion in place while I got OPNsense installed This turned out to be VERY bad for throughput but at least I had some protection while I got OPNsense ready.

So now I'm setting up OPNsense.

From what I'm reading in forums and groups, OPNsense is a fork / more user friendly version of pfSense  so I decide to go with it. Also see pfSense® vs OPNsense®: technical comparison and this compare from a guy that installed both. One of the co-founders of pfSense joined Unifi awhile back and the additions they are now making to Unifi fall somewhere in between OPNsense and pfSense from what I as a user see. As I mention below though it seems unlikely current Unifi hardware will be able handle the load of processing high bandwidth data so going with a standalone box makes more sense for probably at least the next year.

I'm using a Dell 745 with an Intel(R) Core(TM)2 CPU 6400 @ 2.13GHz (2 cores), CDROM drive, 2 TB HD and 8 GB RAM. It is just spare I had been using initially for OpenShift testing but is not really powerful enough for any real projects I do these days but well beyond what a Unifi USG can do with a dual-core, 500 MHz processor, 512 MB DDR2 RAM and 2 GB of storage.

Install went smooth (the occasional label did not match the instructions but it was generally obvious what to do next) except for one thing. The DVD image failed half way though coming up because it seemed to be unable to find the USB DVD drive I had booted from. I had even less luck with a USB stick image as they would not even boot but that might have more to do with the age of the PC I'm using. Note when installer the new machine I found the installer went through without error the second time but got the same CD not found error the first time so this might just be an intermittent issue.

Even more detailed instructions can be found in Building a BSD home router (pt. 5): Installing OPNsense

From the picture above you can see It performed well as a router even after turning off hardware offloading but when I turned on full IPS it took a real hit.

The CPU usage is frequently hitting 100% though so I think that is at least part of the problem. I'll need to try swapping in a Dell 780 with a  Intel(R) Core(TM)2 Quad CPU Q9550  @ 2.83GHz box and see it that improves things. For now though it mainly seems to be affecting download and upload so not pressing. This is what the dashboard looks like (WAN IP blacked out here of course).

Just unchecking  IPS mode on the OPNsense Services: Intrusion Detection: Administration page, but leaving IDS enabled, pretty much gets us back to normal.

Having IPS on was also causing check for updates to fail.

Reports

Traffic

So far the traffic reports look about the same as what I'd get from Unifi though in my case it helps me to filter just what is talking or not talking to the outside world. Otherwise Unifi pretty much wins here.

IPS 

Now this is the bit I wanted OPNsense for. Unifi has started adding much of this in but with the CPU their routers have it just can't cope with processing this amount of data. Turning IPS on with the low end USG router people are reporting download speeds in the 50 Mbps range. As mentioned above even the Dual core I'm using for OPNsense is struggling with full IPS on (29905 rules) and drastically reduced max speeds. Granted that many rules is probably over kill. Especially when looking at the alert log and seeing almost all the hits were for this one rule

Unifi has a similar geo filter but currently you can only have on where OPNsense lets you have many.

Both let you get just alerts or drop packets. Note the above test had both set to alert instead of drop. Unifi set to alert only mode hardly affected throughput at all while block mode definitely does. With OPNsense it was not as clear because setting IPS mode on here

Seems drag down the system as if packets are being block even though the rules themselves have their "Action" set to Alert. Though looking at the above Unifi rule that would seem to imply it is blocking these packets yet OPNsense is still seeing them would imply it is working similar. But then too the only IPS alerts I've seen from Unifi since turning alerts on are:

  EVENT TIME

IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 191.101.167.73:46524, to: 10.10.2.48:8078, protocol: TCP, in interface: eth1 11:30 04/18/2018

IPS Alert 2: Misc Attack. Signature ET DROP Dshield Block Listed Source group 1. From: 191.101.167.37:46430, to: 10.10.2.46:8076, protocol: TCP, in interface: eth1 19:12 04/11/2018

An IP that appears to be in Netherlands though the attached phone number and some of the contact info is Czech.

My Unifi IPS setting look like this

Console

Have to say I like the amount of stuff you can cram onto the console. Her I have it in 4 column mode and this is not even everything you can stick on there. Surprised there seems to be no way to hide or shrink the left nav menu as customizable as the rest is.

ssh access

Seems checking Permit password login does not seem to work. You still need to use ssh keys.

Adding speedtest to the OPNsesnse box

This should work but did not for me
curl -Lo speedtest-cli https://raw.githubusercontent.com/sivel/speedtest-cli/master/speedtest.py
chmod +x speedtest-cli

So as a work around I copy pasted into vi from a browser. Note you will need to change the first line from python to python2.7 either way so it can find the python command.
Run it like this

For keeping track create a cron job that writes to a csv file

As root run (changing 16089 to the server ID you want to test against)
/root/speedtest-cli --server 16089 --csv-header > /usr/local/www/speedtest.csv
That sets the headers for you
Then set up a cron job to run with
crontab -e
Note it uses the vi editor so you need to know the right commands for that. Should look like this

Update it appears the crontab is getting overwritten so you may have to redo this from time to time.
To avoid this you need to create a /usr/local/opnsense/service/conf/actions.d/actions_speedtest.conf file and add

[test]
command:/root/speedtest-cli --server 16089 --csv >> /usr/local/www/speedtest.csv
description:Run a speed test
parameters:%s
type:script

message:running speed test %s

After saving run
service configd restart
to load action file changes then
configctl speedtest test
to test it. You should get an OK as a response. Then add as a cron job through the web GUI like this

This should add a line to the crontab but does not seem too. Nor did it run at first even after reboot but then started running later. No obvious reason why but the line I manually added to the crontab disappeared about the same time. Still not adding a line in crontab though so being handled else where.

Once it is running you can download the speedtests results file by pointing your browser to https://192.168.0.1/speedtest.csv (assuming your web interface it on 192.168.0.1 of course)

Odd bits

The LAN interface switched from 192.168.0.1, the static assigned, to 192.168.0.201, a DHCP address, at one point for no obvious reason.

Sometimes the Speedtest.net extension just seems to hang when run while on the dashboard page.

As mentioned above check updated fails with IPS on. It appeared it was because the Core(TM)2 CPU is just not powerful enough to handle both at the same time. But I'm seeing the same with the Quad Core and the CPU is under 40% max.

I found a good starting point for rules in this pfSense post
Basically this sets:

• emerging-drop
• emerging-botcc.portgrouped
• emerging-botcc
• emerging-ciarmy
• emerging-compromised
• emerging-dshield
• emerging-tor
• emerging-worm
• emerging-trojan
• emerging-mobile_malware
• emerging-malware
• snort

Note on Snort. There are a lot of Snort rules. I filtered the list for "snort", selected all then enabled selected which enabled about half of them. I then also enabled Snort VRT/blacklist.
After reenabling IPS mode speed test and load looks like this

Note the much improved CPU usage. That upload is way low but after turning off IPS I got an even worse run and then a slightly better one which makes me wonder if the speed tests are telling me anything. External factors must be throwing them off or something is causing flux in the OPNsense box I can't see.

Note the CPU usage has actually gone up even though IPS is now off which makes no sense at all.

Web GUI seems to have gotten slugish after enabling IPS even when traffic and CPU is low. eventually the dropped packets got so bad I rebooted but it did not seem to help so as a last ditch attempt I repovered the box after checking the cards were all well seated. This seems to have helped at least for now. Speedtest with IPS off

And Speedtest with IPS back on.

Doing more research Seeing 0% packet loss after repower but back to 20% and other weirdness within an hour.

Swapped out the 1 PCIe x1 network card (WAN side) and the 2 PCI cards (the slots the mother board had open) with a 4 port PCIe x4 card in place of the unneeded video card in the PCIe x16 slot and things have improved a lot! The next morning I was still seeing 0% packet loss and higher speeds than I had connected to the modem directly.

Last night I even hit over 40 Mbps up.

This is with these plugins installed

os-acme-client (orphaned)	1.13	221KiB	Let's Encrypt client
os-dyndns (orphaned)	1.6_1	134KiB	Dynamic DNS Support
os-intrusion-detection-content-snort-vrt (orphaned)	1.0	12.4KiB	IDS Snort VRT ruleset
os-smart (orphaned)	1.2	15.9KiB	SMART tools

And IPS on with the same ET and Snort rule sets as before.

Update: 5/17/2018 This are looking better though the results to vary a good bit.

Some of that might be from all that else is going on though. For instance here

There is about 40 mbps down and 5-6 mbps up going on besides the speed test. The fastest download test I was only seeing 8 mbps up because of how fast the security cams were triggering and transferring alert pics to the cloud that day.

I also added a Pi-Hole DNS filter which has shifted some load from Chrome and the OPNsense. To give you an idea here is the dashboard showing the number of DNS requests it is handling.

Simple to set up with these instructions. Just in time too as the ad blocker Chrome extension I had been using got caught doing stuff.

Short, quick and simplest version of install:

command line commands in bold-italics

Format SD card with SDFormatterv4
Get NOOBS OS installer as zip
Copy NOOBS zip contents to SD card
Boot rpi with SD
Install Raspbian
When done ssh from another PC
ssh pi@192.168.0.62 (password raspberry)
or open terminal window on the pi's console
Run command
sudo raspi-config 
Change the password
Optionally change the host name.
Switch to root user
sudo bash
and run these commands
apt-get update
apt-get upgrade
wget -O basic-install.sh https://install.pi-hole.net
bash basic-install.sh
Note password at end of line
   [i] Web Interface password: 
you will need it to login to the admin web interface.

Lastly you should backup your config any time you make changes like adding sites to a whitelist. If you are recovering from a crash (this has happened a couple times to me) you can also import your config from this screen (Settings->Teleporter) as well.

Update 7/7/2019 If you have Unifi
Check out [Step-By-Step Tutorial/Guide] Raspberry Pi with UniFi Controller and Pi-hole from scratch (headless)  Easy setup, about the same cost as a Cloudkey Gen 1,  About half that of a Gen2, works slick and can give you the space to turn on recording of Clients Historical Data that is tough on a Cloudkey Gen 1. My Raspberry Pi 3B+ seems to have no problem running both programs. I did bump the SD to 32GB. I've just turned on recording of Clients Historical Data so we'll have to see how much storage gets taken up by that.

Update 8/5:
After running the Unifi controller / Pi-hole combo for a couple weeks I'm getting a bit concerned. It has crashed twice now drastically degrading network performance since DNS goes offline. I had planned on taking the original Pi-hole offline but now I'm thinking I need to keep it up and config it as a backup.

Update 12/27:
Between logs, the Pi-Hole DB, the Unifi backups and the update files SD card keeps filling up regularly.
For example I had all these files over 100 MBs after doing some clean up
root@UnifiPi:/etc/pihole# find / -size +100M -ls |
   287694 138240 -rwxr-xr-x   1 root     root     141550912 Jul 10 07:27 /usr/lib/chromium-browser/chromium-browser
   398403 122876 -rw-r--r--   1 root     root     125817618 Oct 18 13:28 /usr/lib/jvm/java-11-openjdk-armhf/lib/modules
   278090 123076 -rw-r--r--   1 root     root     126023659 Oct 18 13:28 /usr/lib/jvm/java-11-openjdk-armhf/jmods/java.base.jmod
  1582114 10821252 -rw-r--r--   1 pihole   pihole   11080900608 Dec 27 20:03 /etc/pihole/pihole-FTL.db
  1835088   318560 -rw-r-----   1 root     adm        326197642 Dec  2 00:00 /var/log/daemon.log.1
  1835105   979680 -rw-r-----   1 root     adm       1003184946 Dec 27 19:43 /var/log/daemon.log
  1835084   277436 -rw-r--r--   1 pihole   pihole     284087304 Dec  1 00:05 /var/log/pihole.log.1
  1835011   287236 -rw-r-----   1 root     adm        294126053 Dec 27 20:00 /var/log/syslog
   662490   131076 -rw-------   1 unifi    unifi      134217728 Sep  5 09:38 /var/lib/unifi/db/_tmp_repairDatabase_last/ace_stat.3
   662484   131076 -rw-------   1 unifi    unifi      134217728 Oct 14 23:33 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_7/ace_stat.3
   662486   262144 -rw-------   1 unifi    unifi      268435456 Oct 14 23:33 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_7/ace_stat.4
   662483   524036 -rw-------   1 unifi    unifi      536608768 Oct 14 23:32 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_7/ace_stat.5
   662497   131072 -rw-------   1 unifi    unifi      134217728 Nov  4 03:47 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_5/ace_stat.3
   662499   262144 -rw-------   1 unifi    unifi      268435456 Nov  4 03:47 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_5/ace_stat.4
   662496   524032 -rw-------   1 unifi    unifi      536608768 Nov  4 03:46 /var/lib/unifi/db/_tmp_repairDatabase_last/_tmp_repairDatabase_5/ace_stat.5
   662501   262144 -rw-------   1 unifi    unifi      268435456 Sep  5 09:38 /var/lib/unifi/db/_tmp_repairDatabase_last/ace_stat.4
   662489   524036 -rw-------   1 unifi    unifi      536608768 Sep  5 09:37 /var/lib/unifi/db/_tmp_repairDatabase_last/ace_stat.5
   662506   131072 -rw-------   1 unifi    unifi      134217728 Dec 27 19:59 /var/lib/unifi/db/ace_stat.3
   662503   131072 -rw-------   1 unifi    unifi      134217728 Dec 27 20:02 /var/lib/unifi/db/ace.3
   662527   131076 -rw-------   1 unifi    unifi      134217728 Nov 25 13:15 /var/lib/unifi/db/_tmp_repairDatabase_1/ace_stat.3
   662529   262144 -rw-------   1 unifi    unifi      268435456 Nov 25 13:15 /var/lib/unifi/db/_tmp_repairDatabase_1/ace_stat.4
   662526   524036 -rw-------   1 unifi    unifi      536608768 Nov 25 13:13 /var/lib/unifi/db/_tmp_repairDatabase_1/ace_stat.5
   662520   262148 -rw-------   1 unifi    unifi      268435456 Dec 27 20:00 /var/lib/unifi/db/ace_stat.4
   662519   262148 -rw-------   1 unifi    unifi      268435456 Nov 25 23:59 /var/lib/unifi/db/ace.4
   662464   131072 -rw-------   1 unifi    unifi      134217728 Nov 22 00:04 /var/lib/unifi/db/_tmp_repairDatabase_0/ace_stat.3
   662466   262148 -rw-------   1 unifi    unifi      268435456 Nov 22 00:05 /var/lib/unifi/db/_tmp_repairDatabase_0/ace_stat.4
   662463   524036 -rw-------   1 unifi    unifi      536608768 Nov 22 00:04 /var/lib/unifi/db/_tmp_repairDatabase_0/ace_stat.5
   662505   524036 -rw-------   1 unifi    unifi      536608768 Dec 27 17:06 /var/lib/unifi/db/ace_stat.5
   662515   131072 -rw-------   1 unifi    unifi      134217728 Nov 29 00:05 /var/lib/unifi/db/_tmp_repairDatabase_2/ace_stat.3
   662517   262144 -rw-------   1 unifi    unifi      268435456 Nov 29 00:05 /var/lib/unifi/db/_tmp_repairDatabase_2/ace_stat.4
   662514   524036 -rw-------   1 unifi    unifi      536608768 Nov 29 00:03 /var/lib/unifi/db/_tmp_repairDatabase_2/ace_stat.5
   662675   262144 -rw-------   1 mongodb  nogroup    268435456 Jun 30  2019 /var/lib/mongodb/journal/prealloc.2
   662677   262144 -rw-------   1 mongodb  nogroup    268435456 Jun 30  2019 /var/lib/mongodb/journal/prealloc.1
   662676   262144 -rw-------   1 mongodb  nogroup    268435456 Dec 27 18:37 /var/lib/mongodb/journal/j._0
  4194815   178776 -rw-r--r--   1 root     root       183066184 Oct 22 23:08 /var/cache/apt/archives/openjdk-11-jdk-headless_11.0.5+10-1~deb10u1_armhf.deb

  4194330   172596 -rw-r--r--   1 root     root       176736120 Jun 25  2019 /var/cache/apt/archives/openjdk-11-jdk-headless_11.0.3+7-5_armhf.deb

So I'm swapping the SD to a 128GB one using these instructions. Note if you installed via nobs like I did you will need to use the instructions to make a new partition instead of expanding root. I used these variations:
parted
mkfs.ext4 /dev/mmcblk0p3
e2label /dev/mmcblk0p3 var
vi /etc/fstab
/dev/mmcblk0p3  /var2            ext4    defaults          1       2
mkdir /var2
mount /var2
vi /etc/fstab
/dev/mmcblk0p3  /var            ext4    defaults          1       2
mkdir /vb
systemctl stop unifi.service
/etc/init.d/pihole-FTL stop
cd /var
find . | cpio -pvmd /var2
cd /
mv /var /vb
mkdir /var
mount /var
Then just to be sure
df
New FS should show up as /var
reboot
Everything should not restart OK and you can start deleting stuff from /vb

This gets Unifi and Pi-Hole in separate file systems. Mine came out like this
Filesystem     1K-blocks     Used            Available Use% Mounted on 
/dev/root              28776316 25388648    1902868  94% /                   
devtmpfs                  469544               0      469544   0% /dev             
tmpfs                        474152         1580      472572   1% /dev/shm     
tmpfs                        474152         6488      467664   2% /run             
tmpfs                            5120               4          5116   1% /run/lock     
tmpfs                        474152               0      474152   0% /sys/fs/cgroup
/dev/mmcblk0p3  91611060 12665804  74248568  15% /var             
/dev/mmcblk0p6      258094       53035      205060  21% /boot           
tmpfs                          94828               0        94828   0% /run/user/1000
tmpfs                          94828               0        94828   0% /run/user/999  

Update 3/6/2020: Still running out of space on / which causes the Pi to crash

This is caused by /etc/pihole/pihole-FTL.db getting huge. Mine was 15,561 MB when the system ran out of space.
By default the DB keeps up to a years worth of data. You can either reduce this by adding MAXDBDAYS to /etc/pihole/pihole-FTL.conf to reduce the days or probably better solution moving the db onto the /var partition created earlier. To do that add DBFILE=/var/pihole/pihole-FTL.db to /etc/pihole/pihole-FTL.conf then run
mkdir /var/pihole
/etc/init.d/pihole-FTL stop
cp /etc/pihole/pihole-FTL.db /var/pihole/

Note this takes awhile. (My 15 GB file took about 30 minutes!) Once it is done:

/etc/init.d/pihole-FTL stop
ls -lapi /var/pihole/
Once you see it is updating the DB you can remove the old DB with
rm /etc/pihole/pihole-FTL.db

Now I have a lot more space on root and /var hardly sees the increase.

Update 7/23/2020: Still running out of space the old PiHole so stuck a 128 in there too.

But then tried to update and discovered the OS was too old. Fortunately it is not that hard, though lengthy, process to upgrade the OS in place. Here are the instructions I used.

Note to self, NEVER make changes within an hour of bed time.

Went to move my house Z-NET onto the new network as the last thing before bed. Almost 3 hours later I'm back online. 😞 Part of the problem was according the the user forum all I needed to do to update the IP was restart Homeseer. When I did that it asked to install and update which I decided to do. Unfortunately I moved my instance to a SD awhile back to improve performance (the logging was killing the drive that also was recording security cam footage) and it seems the upgrade was confused and wiped most of my install and put a new instance on the old drive. Fortunately having a paranoid bent I backup my Homeseer instance every time the server is restarted so I was able to get a list of the files still in the folder newer than my back up saved off. Then I copied the files in form the new instance and finally the saved files back in. That sorted. I think at least. But it did not find the new IP for the Z-NET. Not surprising since I'm moving from 192.168.* IPs to 10.* IPs. After a bit of file scanning I found the IP needs changed in \Program Files (x86)\HomeSeer HS3\Config\Z-Wave.ini so after changing that and a restart I'm finally back in biz. Time for bed.

iRedMail on CentOS 8 as aggregating mail server

CPU fan went on my home mail server (that pulls in email from all my accounts) and since it is so old (a Core 2 Duo) I decided to just use one of the old security camera servers left over from an upgrade instead.

Things assumed you know

Note I use vi to edit files since it is including in almost every Linux and Unix install. If you are not familiar with any available editors look at this vi Cheat Sheet for help.

To see active lines in move config files you can use the command

grep -v "#" /etc/postfix/main.cf | grep -v "^[[:space:]]*$"

where /etc/postfix/main.cf  is the config file you want to see the active lines of. 

For the examples below I'm using these values. Where you see them replace with your own.

• host=mail
• domain=dea42.us
• full qualified name=mail.dea42.us
• server IP=10.10.2.200
• end email address = deabigt@dea42.us
• placeholder for passwords = P@$$w0rd

Unless otherwise mentioned all commands should be run as root (or add sudo in front of them)

A typo can cause emails to be lost so you will want to have a server to test fetching emails from that you can limit what is in the "inbox". Either an account that has separate imapped folders (not gmail) or test account you can send test emails to like a gmail account you do not use for anything important. Once you think you have things working you can try a few emails a time with you test account and even your main account if it has separate folders.

And things you should

I tried to go down the road of installing CentOS 7 but iRedMail makes that virtually impossible. You would have to hack it so much you might as well just install it manually.

The only web server option is Nginx. If you do not select it you will not get web mail.

Steps:

Get CentOS 8 Stream

Note the "DVD" ISOs will not fit on a DVD. You will need a Blu-Ray or USB stick 16 GB or higher. 

If you need help with that see How to Burn an ISO File to a USB Drive. Be sure to create a user other than root and put them in the wheel group.

Note after install you will want to turn on your network interface as it is off by default. Also the name local.localhost for the computer name is misleading. If you enter name.domain as in mail.dea42.us it changes the name to mail-dea42-us.

the simple fix is to run the command

hostname -b mail.dea42.us

Install iRedMail

iRedMail gets you a web mail server without a lot of config file editing but note the "And things you should" above.

The above link is for iRedMail 1.4 but has some commands missing in the official doc.

Ignore the implied CentOS 8 option. As of this writing CentOS 8 is EOL and your only non stream option is 7.9.

Note some stuff is in CentOS 8 stream so ignore any already installed "errors".

SOGO is not selected by default. I added it though I doubt I'll need it. Note running yum update will get errors trying to update this due to a library conflict. 

Note I have a thing about putting large amounts of expanding data in the / partition so for mail storage I used /home/var/vmail instead of /var/vmail.

Once done, go add your user to the mail system with the iRedMail panel by pointing your browser to https://10.10.2.200/iredadmin (note I'm using the IP here since it is on my local network).

To start you can only log into the admin panel and web mail as postmaster@dea42.us. As the last bit of the install says you really want to read iRedMail.tips and move it to a secure location since it contains login info in plain text.

Bringing over old data.

Once you have you web mail users created you can tar up your old emails in the user's Maildir folder. May be under user's home or some convoluted  vmail path. For instance for deabigt@dea42.us on the old server was /home/var/vmail/vmail1/dea42.us/d/e/a/deabigt-2018.03.23.20.35.01/Maildir and on the new server /home/var/vmail/vmail1/dea42.us/d/e/a/deabigt-2022.03.22.17.01.28/Maildir. I moved the Maildir folder to Maildir.bak just to be safe but no really needed.

You can also export and import your filters but that is it without access to the DB. To get indenties for example you will need to export tables, possibly modify the SQL and then import in to the new DB.

Install phpMyAdmin

Note iRedMail sets up the html folder to be the more standard /var/www/html instead of Nginx's normal  /usr/share/nginx/html/

Note use this Generate blowfish secret link to generate a random encrypt key of the 32 characters needed. 

Also the old php.info() is disabled by default. I usually change it to a honeypot anyway so try making /var/www/html/info.php this instead to test php and log any malware looking for it.

<html>

<head><title>honeypot Access logger</title></head>

<body><h1>Honeypot logger</h1>

<?php

$log="/var/log/nginx/honeypot.log";

$el = date("Y-m-d h:i:s",time())." ".$_SERVER['REQUEST_URI'] . " accessed from " . $_SERVER['REMOTE_ADDR'] . " via " . $_SERVER['SERVER_PROTOCOL'] . "- " . $_SERVER['REQUEST_METHOD'] . " with query " . $_SERVER['QUERY_STRING'];

$hl = error_log($el.PHP_EOL, 0);

echo "<br>Logged $el - $hl <br><br>";

foreach (getallheaders() as $name => $value) {

    echo "$name: $value<br>";

    $el =  $el . " :" . $name .": " . $value;

}

foreach ($_REQUEST as $name => $value) {

    echo "$name: $value<br>";

    $el =  $el . " :" . $name .": " . $value;

}

$hl = error_log($el, 3, $log);

?>

</body>

</html>

You can then add a cron script to notify you if something hits this and you will get more info than you probably want about what accessed it. For example run

crontab -e

and add this to the bottom

# Honeypot checker

15 * * * * /bin/grep -v 10.10.2.183 /var/log/nginx/honeypot.log

Every hour at a quarter past if there is anything in there not from 10.10.2.183 the user (root assumed) will get and email with the info till you empty the log. I'm assuming a small network here where this is unlikely to get many hits. Something exposed to the internet should of course be a lot more complex.

Now you should be able to export and import data from your old system DB into the new one. Though you might need to edit user IDs if not added in same order. 

Fetchmail

Man page GitLab repo

This is the aggregating bit. It will go get mail from other servers and deliver it here. Webmin has a simple web interface for this but iRedMail does not. Installing both could cause conflicts but the free iRedMail admin panel seems to do so little it is probably worth the risk.

Install Webmin 

Getting to actually work

After all that it appears it did not get setup correctly since fetchmail retrieved email but postfix then tried to forward them but could not connect to localhost according to logs.

Issues (found) and fixed while debugging:

• Run cp /etc/hosts /var/spool/postfix/etc/hosts (logs said they were out of sync)
• Run cp /etc/localtime /var/spool/postfix/etc/localtime (logs said they were out of sync)
• On Roundcube check folders are all subscribed. (I found many of mine were not so were hidden.)
• Add no sslcertck to my user's ~/.fetchmailrc since Webmin does not seem to have a way to add it. (Testing server had a self signed cert.)
• Add Delivered-To ignore to header checks. (Fixes issues with postfix forwarding to the original to.) Details on how to do that here. 
• Add  --sslproto~TLS1.2+  to fetchmail command in Webmin. (Test server did not do TLS1.3)

https://10.10.2.200:10000/config.cgi?module=fetchmail&section=line1&search=&nnext=Next&xnavigation=1

I sorted the above by going back to basics, testing each link in the chain. To start testing open an extra window and as root run

tail -f /var/log/maillog

to see any errors while running mail tests.

Test Postfix

For this to work you may need create the tmp, new and cur email folder for the test user. I'm using deabigt for my username which Roundcube had already created them for. Type the stuff in bold, stuff in grey are the responses you should see (or something close to).

telnet localhost smtp (you will be asked if you want to install if you have not already) 

Trying ::1...

Connected to localhost.

Escape character is '^]'.

220 localhost.localdomain ESMTP Postfix

ehlo localhost

250-localhost.localdomain

250-PIPELINING

250-SIZE 10240000

250-VRFY

250-ETRN

250-ENHANCEDSTATUSCODES

250-8BITMIME

250 DSN

mail from:<deabigt@dea42.us>

250 2.1.0 Ok

rcpt to:<deabigt@dea42.us>

250 2.1.5 Ok

data

354 End data with <CR><LF>.<CR><LF>

This is a test email from telnet

.

250 2.0.0 Ok: queued as 62AD46011817

quit

221 2.0.0 Bye

Connection closed by foreign host.

If you get the queued line you know it tool it OK.

Testing Dovecot

Next we want to check we can read our emails from via Dovecot.

Run

netstat -tunap | grep dovecot

to see what port(s) dovecot is listening on

Short version. Run

openssl s_client -connect 10.10.2.200:993

or

openssl s_client -connect 10.10.2.200:143 -starttls imap

You should see bunch of SSL handshake stuff then 

. OK Pre-login capabilities listed, post-login capabilities have more.

as the last line.

Send (with username a password for user you have set up).

a login "deabigt@dea42.us" "P@$$w0rd"

and you should see

* CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS BINARY MOVE SNIPPET=FUZZY SPECIAL-USE

a OK Logged in

Just to be sure send

c list "" *

and you see something like this (depending of what folders you created)

* LIST (\HasNoChildren \Sent) "." Sent

* LIST (\HasNoChildren \Drafts) "." Drafts

* LIST (\HasNoChildren \Trash) "." Trash

* LIST (\HasNoChildren) "." INBOX

c OK List completed (0.001 + 0.000 + 0.001 secs).

Then logout with

e logout

and see

* BYE Logging out

e OK Logout completed (0.001 + 0.000 secs).

closed

If you get and error like:

system library:connect:Connection refused:crypto/bio/b_sock2.c

Dovecot is not running or you IP address or port is wrong.

Check sieve

In my case messages where not getting delivered so the next step is to check the sieve log. For my test user it is at 

/home/var/vmail/vmail1/dea42.us/d/e/a/deabigt-2022.03.22.17.01.28/sieve/dovecot.sieve.log

In there I found 

error: from=postmaster@dea42.us, envelope_sender=MAILER-DAEMON, subject=Rejected: , msgid=<dovecot-1648226282-16727-0@mail-dea42-us>, size=3642, delivery_time=42ms, failed to store into mailbox 'INBOX': Quota exceeded (mailbox for user is full).

Since the email was too and from me it went nowhere. From Webmin it appeared my quota should be unlimited but digging I found in /etc/dovecot/dovecot.conf that no default quota set as these lines were commented out.

    quota_rule = *:storage=1G

    quota_rule2 = *:messages=0

    quota_rule3 = Trash:storage=1G

    quota_rule4 = Junk:ignore

so I uncommented them and restarted Dovecot. (I bumped the top rule to 100G since I had the space).

That sorted that. It appears my sieve rule to send emails without a subject to Junk even worked.

To be sure though you should run sieve-filter with some test emails.

sieve-filter -D -v -W -C -u "deabigt@dea42.us" "/home/var/vmail/vmail1/dea42.us/d/e/a/deabigt-2022.03.22.17.01.28/sieve/dovecot.sieve" "AAA_filterTest" keep

Note AAA_filterTest is an email folder I created to stick test emails in.

That should have showed me how the emails in that folder would be filtered / sorted but instead I got

sieve-filter(deabigt@dea42.us): Error: user deabigt@dea42.us: Auth USER lookup failed

sieve-filter(root): Fatal: Internal error occurred. Refer to server log for more information.

Where is a whole mix of perms to get around for this to work. How I got it to finally work was to

chmod 7755 /usr/bin/sieve-filter

chmod 711 /home/deabigt

then as user deabigt create a script chkRules.sh with this line

sieve-filter -D -v -W  "/home/var/vmail/vmail1/dea42.us/d/e/a/deabigt-2022.03.22.17.01.28/sieve/dovecot.sieve" "AAA_filterTest" keep

and another script runRules.sh with this line

sieve-filter -e -D -v -W  "/home/var/vmail/vmail1/dea42.us/d/e/a/deabigt-2022.03.22.17.01.28/sieve/dovecot.sieve" "AAA_filterTest" keep

Make both executable with

chmod 744 *Rules.sh

Pigeonhole Sieve rule examples

Enable fetchmail schedule

Once you have everything tested (including options below) the last thing you should do it setup and enable fetchmail's schedule in the Webmin interface. This will setup cron job(s) to pulls from the servers. 

Optional

A few other things I wanted for my setup 

UPS monitor

Install apcupsd to shutdown server when UPS is ready to quit.

VNC

I want root DB access only from the console but the server will be in another building so VNC should let me run a browser there remotely.

Note VNC is built in for CentOS 8 Stream. If you install and enable something like TigerVNC server you may not be able to login on the console much less remotely with a VNC viewer.

Also RealVNC appears to still not support TLS so make sure to follow instructions to the bottom.

It also seems that the user must be logged in on the console to connect as well. Given that and I could not seem to get past the black screen issue either so I moved on for now. And setup XMing instead. Though I used SecureCRT instead of Putty.

Top 25 Nginx Web Server Best Security Practices

Getting more Roundcube options to work

The basic / free version of iRedmail does not install the mark junk and spell check options / plugins I want installed so I needed to run installer manually. 

To be safe, make backups of /opt/www/roundcubemail/composer.json and /opt/www/roundcubemail/config/config.inc.php before proceeding.

To enable it you first need to enable the installer again. They locked it down multiple ways. To reverse this you need to:

• chmod 777 /opt/www/roundcubemail/installer
• chmod 555 /opt/www/roundcubemail/config/config.inc.php
• ln -s  /opt/www/roundcubemail/installer/ /opt/www/roundcubemail/public_html/installer
• edit /etc/nginx/templates/roundcube.tmpl and comment out (add # at beginning of line) to the location ~ ^/mail/(bin|config|installer|logs|SQL|temp|vendor)($|/.*) { deny all; } line.
• systemctl restart nginx
• Add $config['enable_installer'] = true; to the end of  /opt/www/roundcubemail/config/config.inc.php

You can point your browser at https://10.10.2.200/mail/installer.

Step 1 

Shows you what support structure is installed. In this case just imagick and some unneeded DBs should be missing. If you want imagick installed see Amazon Linux 2: Installing ImageMagick for PHP 7.4 even though you probably have php version 7.2 installed these instructions will still work if you sub nginx for httpd in the restart services step.

Refreshing the browser window where you have the installer should now look like this.

Roundcube installer step 1

Since you already have a config.inc.php installed the Next button will take you straight to step 3. Instead click on Create config (step 2).

Step 2

This is where all the options are. The one not enabled by iRedMail that I enabled were (from top to bottom of page).

• Enabled enable_spellcheck
• I changed smtp_server to my hosted email system since most email systems these days block email coming from "residential IPs". 
• Changed htmleditor from never to on reply to HTML message only
• Enabled additional_message_headers, archive, emoticons, identity_select, jqueryui, markasjunk, newmail_notifier, show_additional_headers, subscriptions_option, userinfo and vcard_attachments.

Check UPDATE_CONFIG button at bottom of page and it saves your changes and reloads the page. Click CONTINUE button at top of page to go to step 3.

Step 3

Mainly shows you that installer did not seriously mess up the config file. It also give you a quick chance to check connections to the smtp and Dovecot servers. So you are done with the installer and can disable it again. Again personally I'd just remove the link, /opt/www/roundcubemail/public_html/installer and replace it with another honeypot script like /var/www/html/info.php but that does not actually work with the way Ngnix works. What you can do is in /etc/nginx/templates/roundcube.tmpl replace the lines

# Block access to default directories and files under these directories

location ~ ^/mail/(bin|config|installer|logs|SQL|temp|vendor)($|/.*) { deny all; }

with

location ~ ^/mail/installer(.*) {

    include /etc/nginx/templates/hsts.tmpl;

    include /etc/nginx/templates/fastcgi_php.tmpl;

    fastcgi_param SCRIPT_FILENAME /var/www/html/info.php;

}

# Block access to default directories and files under these directories

location ~ ^/mail/(bin|config|logs|SQL|temp|vendor)($|/.*) { deny all; }

and restart Ngnix

systemctl restart nginx

Note after disabling the installer you may need to clear your cache or restart your browser to see the change.

Step 4

Not part of the installer but a check to see what actually changed. My back up of  /opt/www/roundcubemail/config/config.inc.php was /opt/www/roundcubemail/config/config.inc.php.iredmail so I did a diff removing comments and blank lines with 

grep "=" config.inc.php.iredmail | sort -b > config.inc.php.iredmail.srt

grep "=" config.inc.php| sort -b > config.inc.php.srt

diff -B -b -w config.inc.php.srt config.inc.php.iredmail.srt | grep -v ". // "

Which yielded

6c6

< $config['db_dsnw'] = 'mysql://roundcube:P@$$w0rd@127.0.0.1/roundcubemail';

---

> $config['db_dsnw'] = 'mysqli://roundcube:P@$$w0rd@127.0.0.1:3306/roundcubemail';

10a11

> $config['default_port'] = 143;

13d13

< $config['enable_spellcheck'] = true;

15d14

< $config['htmleditor'] = 2;

20c19

< $config['language'] = 'en_US';

---

28c27

< $config['plugins'] = ['additional_message_headers', 'archive', 'emoticons', 'identity_select', 'jqueryui', 'managesieve', 'markasjunk', 'newmail_notifier', 'password', 'show_additional_headers', 'subscriptions_option', 'userinfo', 'vcard_attachments', 'zipdownload'];

---

> $config['plugins'] = array('managesieve', 'password', 'zipdownload');

33,35c32,36

< $config['smtp_server'] = 'tls://mail.dea42.org';

< $config['spellcheck_engine'] = 'enchant';

< $config['support_url'] = '';

---

> $config['smtp_pass'] = '%p';

> $config['smtp_port'] = 587;

> $config['smtp_server'] = 'tls://127.0.0.1';

> $config['smtp_user'] = '%u';

> $config['spellcheck_engine'] = 'pspell';

37c38

< $config['useragent'] = 'Roundcube Webmail';

---

> $config['useragent'] = 'Roundcube Webmail'; // Hide version number

39,42c40,41

<   'ssl' => 

<   'ssl' => 

---

>     'ssl' => array(

>     'ssl' => array(

I color coded to make seeing the diff stand out. Colors are added stuff, changed stuff and removed stuff. The removed stuff is the worrying bit though the DB line does not seem to matter. The smtp stuff though will unless you are using a smtp server that does not need login info to send emails. Since this an aggregating server using a remote smtp server I would need to override smtp_user and smtp_pass at minimum anyway. Per https://github.com/roundcube/roundcubemail/wiki/Configuration

Create an outgoing email account on your hosted site
Get client connection info as in

Secure SSL/TLS Settings (Recommended)

Username:	smtp@example.com
Password:	Use the email account’s password.
Incoming Server:	mail.example.com IMAP Port: 993     POP3 Port: 995
Outgoing Server:	mail.example.com SMTP Port: 465
IMAP, POP3, and SMTP require authentication.

Edit /var/www/roundcubemail/config/config.inc.php to match these
edit the smtp section from
// SMTP
$config['smtp_server'] = 'tls://127.0.0.1';
$config['smtp_port'] = 587;
$config['smtp_user'] = '%u';
$config['smtp_pass'] = '%p';
$config['smtp_auth_type'] = 'LOGIN';
// Required if you're running PHP 5.6 or later
$config['smtp_conn_options'] = array(
    'ssl' => array(
        'verify_peer'      => false,
        'verify_peer_name' => false,
    ),
);

To
// SMTP
$config['smtp_server'] = 'ssl://mail.example.com';
$config['smtp_port'] = 465;
$config['smtp_user'] = 'smtp@example.com';
$config['smtp_pass'] = 'P@$$w0rd';
$config['smtp_auth_type'] = 'LOGIN';
// Required if you're running PHP 5.6 or later
$config['smtp_conn_options'] = array(
    'ssl' => array(
        'verify_peer'      => false,
        'verify_peer_name' => false,
    ),

Don't add Procmail for spam filtering

Has been unmaintained for years and is not longer considered good practice even though still in Webmin and most Dovecot plus SpamAssassin searches will show using Promail.

Add SpamAssassin to learn from emails marked as spam and not spam. 

Add SpamAssassin

Follow the steps in the link till you get to the Move Spam into the Junk Folder bit. Though tweak the rules and scores as you see fit. I left most at the defaults for now.  Also you can safely ignore the OpenDKIM and OpenDMARC bits if like me you are only fetching emails and not receiving them directly at this server.

Move Spam into the Junk Folder

The iRedMail install stuck all the Dovecot config into the main file /etc/dovecot/dovecot.conf so the files in the conf.d folder are ignored.  Plus all the stuff it tells you to add should have already been done. Note the sieve_before script is at /home/var/vmail/sieve/dovecot.sieve

Also from article comments, it appears in /etc/sysconfig/spamassassin 

razor-home-dir='/var/lib/razor/

ought to be

razor-home-dir='/var/spool/amavisd/.razor/

Set Message Maximum Size

I left the default here as well till I see it is an issue to reduce load. Especially since I get a lot of security cam alerts with picture attached.

How to Configure Individual User Preferences

Probably could skip this unless you have more than one account and you want them to detect differently. The instructions are not very clear though so you are probably better looking at the official docs. Note if you do want to add user_prefs it should be in a .spamassassin folder placed in the same folder as you Maildir. In my example case /home/var/vmail/vmail1/dea42.us/d/e/a/deabigt-2022.03.22.17.01.28/.spamassassin/user_prefs

After testing it appears this may not work in the virtual users context. You will want to add your customizations to /etc/mail/spamassassin/local.cf Note this is not the normal location and the more common /usr/share/spamassassin/ files exist but are ignored.

Learn spam

The Roundcube Webmail MarkAsJunk Plugin docs imply that it can be configured to learn / unlearn when to mark an email as junk / not junk but the bits just to change the Subject do not seem to get triggered by the button. All the button appears to do is move emails to the Junk or Inbox folders.

I prefer a command I can run manually and or from a cron job anyway. Again the virtual mail setup complicates this. I started with this info, this and this.

First I made a couple new folders. AAA_junk, AAA_NotJunk and AAA_filterTest. They could be named anything. The AAA_ keeps them at the top on my folder list which is VERY long.

Next I created a script learnSpam.sh in my users home, /home/deabigt since it needs user specific info. In this case my script looks like this

mf=/home/var/vmail/vmail1/dea42.us/d/e/a/deabigt-2022.03.22.17.01.28/

sa-learn -p ${mf}.spamassassin/user_prefs --spam ${mf}Maildir/.AAA_Junk/{cur,new,tmp}

sa-learn -p ${mf}.spamassassin/user_prefs --ham ${mf}Maildir/.AAA_NotJunk/{cur,new,tmp}

sa-learn --dump magic

Note the . in front of the folder names.

Here is the tricky bit. Again it needs to run as vmail so you need to run

chown vmail.vmail ~deabigt/learnSpam.sh

chmod 7755 ~deabigt/learnSpam.sh

Now stick a few spam emails in AAA_Junk and non spam in AAA_NotJunk and run ~deabigt/learnSpam.sh as that user or root.

You will see output like this

Learned tokens from 390 message(s) (392 message(s) examined)

Learned tokens from 1 message(s) (14 message(s) examined)

0.000          0          3          0  non-token data: bayes db version

0.000          0        601          0  non-token data: nspam

0.000          0         31          0  non-token data: nham

0.000          0      82589          0  non-token data: ntokens

0.000          0 1380809146          0  non-token data: oldest atime

0.000          0 1648659657          0  non-token data: newest atime

0.000          0          0          0  non-token data: last journal sync atime

0.000          0          0          0  non-token data: last expiry atime

0.000          0          0          0  non-token data: last expire atime delta

0.000          0          0          0  non-token data: last expire reduction count

If that looks OK, set it up as a cron job be running (as root)

crontab -e

and adding

# learn spam from staged folders

00 1 * * * /home/deabigt/learnSpam.sh | mailx -s "Learn Spam" deabigt@dea42.us

That will run the script each night at 1 AM and send the user and email with the output.

Note this has to be run as root despite permission appear that running as anyone should work since the script assumes the id of vmail while running but it does not work and will not be able to read the email folders.

I check my Junk folder like this

Search for emails with SPAM

Do a quick look for anything that might be legit.

Move anything that looks legit to AAA_NotJunk

Delete "SPAM" emails since the filter already knows them.

Show all emails in Junk

Move anything that looks legit to AAA_NotJunk

Move rest to AAA_Junk

Run ~deabigt/learnSpam.sh

Delete emails in AAA_Junk

Move emails in AAA_NotJunk to AAA_filterTest

Now you can run ~deabigt/chkRules.sh to see where these emails would be filed or ~deabigt/runRules.sh to file them.

Sort any new rules as needed. Do not forget to check the Junk folder if any of you rules move emails to Junk.

About here is where you wish X-Spam-Score was a column option in Roundcube.

What I did to work around this was to create a script /home/deabigt/rateJunk.sh containing

for l in `grep X-Spam-Score /home/var/vmail/vmail1/dea42.us/d/e/a/deabigt-2022.03.22.17.01.28/Maildir/.AAA_Junk/{cur,new,tmp}/* 2>/dev/null | sed -e "s/ /:/g"`

do

        f=`echo $l | cut -f1-2 -d':'`

        s=`echo $l | cut -f5 -d':'`

        sub=`grep Subject $f`

        echo ${s}:${sub}

done

Make it executable with

chmod 744 /home/deabigt/rateJunk.sh

and run (as root) as

/home/deabigt/rateJunk.sh | sort -nr

the r on the end makes the sort order high to low so the emails with the lowest spam scores at the bottom. The output looks something like this

2.613:Subject: AUTUMN SALE SEASON STARTED! GRAB EXTRA 25% OFF

2.613:Subject: APPROVED CANADIAN HEALTHCARE

2.6:Subject: Webstore 100% Pure Pharmacy.

2.598:Subject: We're More Than Just Your Local Webstore, We're Your Friends, Dqkaujtgfsvn

2.582: h=Date:From:To:Subject:MIME-Version:Content-Type:List-Unsubscribe:Message-ID; i=renewal-by-andersen@meanvilee.cam; Subject: Get ready for the Spring with Renewal by Andersen

2.572:Subject: Seen this Crazy Heart Health Secret?

2.572:Subject: Odd Toxin The Cause Of Your Small Member?

2.32: h=Date:From:To:Subject:MIME-Version:Content-Type:List-Unsubscribe:Message-ID; i=christina_allmand@buyers2022.sbs; Subject: $1OO Visa Reward-Card from ADT with install of Home Security #84358969

2.252:Subject: High-grade MEDICATIONS For a Cheap Price, Becker Aaron ..

2.213:Subject: HIGH-QUALITY DRUGS for the LOWEST PRICE

2.213:Subject: BEST MEDS for the LOWEST PRICE, Ghixbbxigrgv

2.114:Subject: SCORE HUGE SAVINGS on the BEST MEDS, Dilmnbu

2.114:Subject: High-grade MEDICATIONS For a Cheap Price, Fxivjmtxedld

2.114:Subject: Canadian Medicine Shop. The Pharmacy America Trusts Iqzixeuqiox

2.113:Subject: YOUR HEALTH is OUR MAIN CONCERN, Jmleobsiqazn

2.113:Subject: World Best DRUGS Mall For a Reasonable Price, Cnqzjcipgi

2.113:Subject: World Best DRUGS Mall For a Reasonable Price

2.113:Subject: World Best DRUGS Mall For a Reasonable Price

2.113:Subject: When It Comes Healthcare, Nothing Beats a Hometown Advantage Stephen Randall

2.113:Subject: When It Comes Healthcare, Nothing Beats a Hometown Advantage Brickman Hope ..

2.113:Subject: What a Webstore Was Meant to Be Jmhxht

2.113:Subject: What a Webstore Was Meant to Be, Capricee

2.113:Subject: What a Medicine Shop Was Meant to Be, Kennedy Alexis..

2.113:Subject: We're More Than Just Your Local Pharmacy, We're Your Friends

2.113:Subject: We're More Than Just Your Local Medicine Shop, We're Your Friends Ckomoajpfaqp

2.113:Subject: We're More Than Just Your Local Drug Mall, We're Your Friends Mercer Elsa

2.113:Subject: TRUSTED CANADIAN HEALTHCARE Lawman Ernest .

2.113:Subject: TOP-QUALITY DRUGS at DISCOUNT PRICES

2.113:Subject: TOP-GRADE MEDICATIONS at DISCOUNT PRICES Farmer Erik

2.113:Subject: Top-grade Medications at Discount Prices Ellington Becky

2.113:Subject: The Medicine Shop with a Tender Loving Touch, Chesterton Adam

2.113:Subject: The Drugstore with a Tender Loving Touch Lmjpemzpn !!

2.113:Subject: The Drug Mall with a Tender Loving Touch

2.113:Subject: The Biggest DRUGSTORE Mall, Dilmnbu ...

2.113:Subject: SCORE HUGE SAVINGS on the BEST MEDICATIONS

2.113:Subject: SCORE HUGE SAVINGS on the BEST DRUGS, Inhbe

2.113:Subject: Not Just a Pharmacy, But a Family Higgins Foster!!

2.113:Subject: Not Just a Medicine Shop, But a Family Dice

2.113:Subject: High-grade MEDICATIONS For a Cheap Price

2.113:Subject: For the Best Healthcare in Your Hometown .. Count on Us, Creighton Millard.

2.113:Subject: EXTRA 25% OFF AUTUMN SALE Ivvvneooz

2.113:Subject: Drug Mall, Holiday Ted ! 100% Pure Pharmacy .

2.113:Subject: CVS/Medicine Shop ! Expect Something Extra, Htbtgckpy..

2.113:Subject: CVS/Drug Mall .. Expect Something Extra, Fpabyoyfcvgbdf ..

2.113:Subject: CVS/Drug Mall Expect Something Extra Chapman Ann

2.113:Subject: CANADIAN PHARMACY EXCLUSIVE STORE ... 19% OFF Carrington Yvette .

2.113:Subject: Canadian Drugstore The Pharmacy America Trusts Dqolk .

2.113:Subject: Canadian Drug Mall The Pharmacy America Trusts

2.113:Subject: Approved Canadian Healthcare Dea Myheritage

2.113:Subject: Any Meds For a Reasonable Price Fondgaxsuctbh

2.113:Subject: Any Meds For a Reasonable Price Dqolk

2.113:Subject: Any Medications For a Reasonable Price Fraser Georgia .

2.113:Subject: A Hometown Medicine Shop with World Class Service, Gilmore Greg

2.113:Subject: A Hometown Drugstore with World Class Service, Dhsocjc..

2.113:Subject: A Hometown Drug Mall with World Class Service, Julian Alsopp .

2.113:Subject: ADDITIONAL 16% OFF AUTUMN SALE, Dowman Doug..

2.103:Subject: Canadian Medicine Shop Exclusive Mall . 26% Off

2.095:Subject: THE BEST MEDICATIONS at DISCOUNT PRICES Dea Scam!!

There are some obvious things to turn into new Spamassassin rules here.

For instance

header    SUBJECT_DRUGS   Subject =~ /(Pharmacy|MEDICATIONS|DRUG|CANADIAN|Medicine|Meds)/i

describe  SUBJECT_DRUGS   Subject contains drugs

score     SUBJECT_DRUGS   2.0

header    SUBJECT_DRUGS2   Subject =~ /(Healthcare|Hometown|Mall|Best)/i

describe  SUBJECT_DRUGS2   Subject contains drugs

score     SUBJECT_DRUGS2   2.0

header    SUBJECT_CVS   Subject =~ /CVS/i

describe  SUBJECT_CVS   Subject contains CVS

score     SUBJECT_CVS   2.0

You can then test those changes by finding the email with

grep "Subject: THE BEST MEDICATIONS at DISCOUNT PRICES" /home/var/vmail/vmail1/dea42.us/d/e/a/deabigt-2022.03.22.17.01.28/Maildir/.AAA_Junk/{cur,new,tmp}/*

Then rerun it manually with

spamassassin -D < /home/var/vmail/vmail1/dea42.us/d/e/a/deabigt-2022.03.22.17.01.28/Maildir/.AAA_Junk/cur/1648672590.M85185P2023089.mail-dea42-us,S=13452,W=13691:2,| grep X-Spam

If you just want the scoring and not all the debug, leave off the -D

Note you may want to backup the DB with

sa-learn --backup > /home/backups/spamassassin.bak

if you are worried about making things worse. You can restore with

sa-learn --restore < /home/backups/spamassassin.bak

Forcing SpamAssassin To Add The X-Spam-Status Header To Ham For Debugging and tweaking bounce.

The controls for this are in /etc/amavisd/amavisd.conf The defaults are

$sa_tag_level_deflt  = 2.0;  # add spam info headers if at, or above that level

$sa_tag2_level_deflt = 6.2;  # add 'spam detected' headers at that level

$sa_kill_level_deflt = 6.9;  # triggers spam evasive actions (e.g. blocks mail)

$sa_dsn_cutoff_level = 10;   # spam level beyond which a DSN is not sent

$sa_crediblefrom_dsn_cutoff_level = 18; # likewise, but for a likely valid From

#$sa_quarantine_cutoff_level = 25; # spam level beyond which quarantine is off

I always want to know why so to always get the X-Spam-Status header set

$sa_tag_level_deflt  = -9999

I do not want to bounce spam emails for lots of reasons so I replaced all the D_DISCARD values with D_PASS so nothing is bounced. Then I just add a sieve rule to move spam above a score directly to Trash. As in

# rule:[Spam10]

if header :contains "x-spam-level" "**********"

{

fileinto "Trash";

stop;

}

will send to Trash any email with a score of 10 or more. (Actually 9 or more given how the pattern works.)

$sa_tag2_level_deflt sets the level at which spam is tagged in the subject line of the message.

Note this overrides what spamassassin adds so you may see headers added in the spamassassin tests above only to have amavisd remove them and insert its own.

If you want to add the score to the subject change the line

$sa_spam_subject_tag = '[SPAM ] ';

to

$sa_spam_subject_tag = 'SPAM _SCORE_:';

Note not using [] so is sortable more or less by score in Roundcube

And then restart the main services just to be sure with

systemctl restart postfix spamassassin amavisd

More on the SpamAssassin as a Learning System

Add headers in sieve rules

This can be real useful for debugging rule order

To the bottom of /etc/dovecot/dovecot.conf add

plugin {

  # Use editheader

  sieve_extensions = +editheader

  # Header fields must not exceed one kilobyte

  sieve_editheader_max_header_size = 1k

  # Protected special headers

  sieve_editheader_forbid_add = X-Verified

  sieve_editheader_forbid_delete = X-Verified X-Seen

}

Add editheader to the require line of you sieve rule file. It should look like this

require ["copy","enotify","fileinto","imap4flags","regex","variables","editheader"];

You can then add a header with a rule action like

addheader "X-Test-Header" "This is a test header.";

Or as fancy as

# Match/select your message as you see fit

if header :contains "List-Id" ["<foo.lists.example.net>"]

{

    # Match the entire subject ...

    if header :matches "Subject" "*" {

        # ... to get it in a match group that can then be stored in a variable:

        set "subject" "${1}";

    }

    # We can't "replace" a header, but we can delete (all instances of) it and

    # re-add (a single instance of) it:

    deleteheader "Subject";

    # Append/prepend as you see fit

    addheader :last "Subject" "[Foo-List] ${subject}";

    # Note that the header is added ":last" (so it won't appear before possible

    # "Received" headers).

}

Changing the subject header method did not work for me but addheader does.

Add support for amavisd to scan rar files

yum install php72-php-pecl-rar -y

systemctl restart amavisd

Add Spamassassin rule for from == to

A fair bit of spam I see has the to and from the same so you get double hit with it if it bounces. Unfortunately there appears to be no way to compare the fields in Spamassassin but you can see if they are from the same domain with something like this which adds 2 points if they are.

header AVATAR42_TO   To =~ /avatar42.us/

header AVATAR42_FROM   From =~ /avatar42.us/

meta AVATAR42_FROM_TO (AVATAR42_FROM && AVATAR42_TO)

score AVATAR42_FROM_TO 2

TBD

Stuff I'm still looking at but does not seem to be an issue

• plugin: eval failed: bayes: (in learn) locker: safe_lock: cannot create lockfile /var/vmail/dea42.us/deabigt/spamassassin/bayes.mutex: No such file or directory
• razor2: razor2 check failed: No such file or directory razor2: Can't read: /var/lib/razor/ at /usr/share/perl5/vendor_perl/Mail/SpamAssassin/Plugin/Razor2.pm line 331
• amavis[2548656]: No ext program for   .lrz, tried: lrzip -q -k -d -o -, lrzcat -q -k
• amavis still bouncing emails despite setting all D_DISCARD to D_PASS