Capturing IP tables logs

Overview

CentOS7’RHEL7 – I needed to log dropped packets form IPtables to a separate file using rsyslog, I like my logs in separate files, and then rotate them. I read several online guides and most worked but ended up with logs going to a separate file just fine, but they were still going to /var/log/messages, I did not require the double logging, I did set “& ~” in rsyslog.conf but it just didn’t work, after a bit of experimenting the following worked great.

iptables

I only needed to log dropped packets coming in, so I added the following to iptables, they are appended after the last “INPUT” lines, the format of the text “Dropped: ” is important, as it changes the way it’s filtered and displayed.

-A INPUT -j LOGGING
-A LOGGING -m limit --limit 2/min -j LOG --log-prefix "Dropped: " --log-level 4
-A LOGGING -j DROP
-A OUTPUT -m state --state ESTABLISHED -j ACCEPT 
-A OUTPUT -o lo -j ACCEPT

That’s it for iptables

Create a file for the logs

 touch /var/log/iptables-drop

rsyslog changes

I created a new conf file under /etc/rsyslog.d/ called iptables-drop.conf, if you have quite a few conf here you may want to give the file a numerical ID as well. Also check that the rsyslog.conf file has the line $IncludeConfig /etc/rsyslog.d/*.conf uncommented, it usually is by default though.

I then added to iptables-drop.conf, note, that many of the guides use “& ~” but this is deprecated and will tell you so in logs, so use “& stop“.

:msg, contains, "Dropped: "        -/var/log/iptables-drop
& stop

Restart services

Now restart rsyslogd and iptables and you should be good to go. You can check by using tail and sending some rogue traffic

tail -f /var/log/iptables-drop

Rotate logs

All I did for this was add it to /etc/logrotate.d/syslog

/var/log/cron
/var/log/maillog
/var/log/messages
/var/log/secure
/var/log/spooler
/var/log/daemon.log
/var/log/kern.log
/var/log/iptables-drop
{
    missingok
    sharedscripts
    postrotate
        /bin/kill -HUP `cat /var/run/syslogd.pid 2> /dev/null` 2> /dev/null || true
    endscript
}

Notes

I first added the lines in iptables-drop.conf to the main rsyslog.conf, but it didn’t seem to work well, could be an ordering issue, but the correct ways is to use rsyslog.d so that is the way to go.

I also tried the often recommended “:msg, startswith …. ” but that did not work well for me, changing “startswith” to “contains” got it all working great.

There is another easier way of doing all of this as well, if you don’t require a separate log you can use journald to do much of the heavy lifting, all you need to do is add the lines to iptables and you are good to go, no rsyslog configuration required at all. To view them just use the following.

journalctl -k
journalctl -k -f

 

Posted in Uncategorized | Tagged , , , , | Leave a comment

Migrate and upgrade Tenable Security Center from RHEL6/CentOS6 to RHEL7/CentOS7

Overview

I had Tenable Security Center running on a RHEL6 VM which scans around 1000 hosts several times a week. REHL6 is getting a bit long-in-the-tooth and also there was a need to upgrade to the latest and greatest version of SC and RHEL, those being at the time SecurityCenter-5.8.0-el7.x86_64 and RHEL 7.6.

In theory, this should work with other Linux distros as well, there will be some subtle changes, but conceptually you should be good to go.

The user of SC wanted to keep all the custom dashboards, plugins and scan data, so this meant that the entire SC configuration and associated database needed to be migrated, there was only a 24hr window to do this as the scans are used almost daily.

Plan (Simplified)

  • Stand-up and new RHEL7 VM with a temporary IP and name that is different to the original SC, I named it SC-new
  • Stop all SC services on the current SC (named SC-old) and use rsync over SSH to copy everything over
  • Once data has been copied, power off SC-old
  • Change the identity of SC-new to be the same as SC-old, so, same IP host name etc – this is crucial for the migration to work, the host name is tied to the license keys.
  • Install the same version of SC as it had on the old one, once this is done and configured SC should work just like the old one
  • Next, upgrade the SC to the new version
  • Finally patch the latest version
  • Test

Assumptions

I’m assuming you have full access to both hosts and have the firewall etc all configured, have stood-up a new RHEL7 host, and have some Linux command line experience. I have successfully migrated using this process but use at your own risk, if you have never done anything like this before I would seek professional advice before going any further.

I also had the luxury of taking VM snapshots for the entire process, so could revert back if anything what wrong.

Environment

  • VMs on VSphere, 12 cores 8GB RAM
  • Separate 500GB logical volume for /opt – which is the location SC installs
  • Migrating from RHEL6.10 to RHEL7.6
  • Security Center from SecurityCenter-5.6.2.1-es7.x86_64 to SecurityCenter-5.8.0-el7.x86_64

Ports

Incoming TCP Port 443 – User Interface
Incoming TCP Port 22 – Remote Repository Sync with other Tenable.sc(s)
Outgoing TCP Port 443 – Plugins update and Nessus Cloud Manager Communication
Outgoing TCP Port 25 – SMTP Email Notification
Outgoing UDP Port 53 – DNS Resolving

Tools used

  • Ensure you have rsync installed on both hosts. I used rsync to transfer data. The advantage of using rsync is that if your transfer gets interrupted you can resume it, also you can transfer the bulk of the data, say in the evening, and then just copy over the changes another day. For me, the entire sc directory was 200GB in size and took around 1hr to transfer, a few days later, when I ran another sync it only had to copy 6GB of data over so took minutes, this is a huge time saver.
  • On the new host for SC, check you have the following installed; zip, unzip and java-11-openjdk.x86_64, at this point you DO NOT INSTALL any Security Center products.

Prep

  • Ensure new host has been stood-up, with a different name and IP,  is ready to accept transfer, has zip, unzip and java-11-openjdk.x86_64 installed
  • yum install zip unzip java-11-openjdk.x86_64
  • You have configured enough disk for the transfer
  • Inform users there maybe some downtime

The process

Current hosts – SC-old

  • Stop all Security Center services before rsyncing the data over
service SecurityCenter stop

Check it has stopped, there should no output

ps -fu tns
  • On sc-old – copy/rsync the entire contents of /opt/sc over to the new host, it may take several hours so plan accordingly.
rsync -avr --stats --progress -e ssh /opt/sc user1@SC-new:/opt/
  • You are now done with the old Security Center, so power if off, if all goes well, you never need this again.

New host – SC-new

  • Check that you now have a /opt/sc directory and all looks good, currently, as Security Center is not installed, the directory and contents are owned by the user used to copy the data over, we need to fix this later, the user and group will eventually be tns.
pwd
/opt
ls -ltr
drwxr-x--- 16 user1 users 4096 Jan 23 12:20 sc
  • Now we are going to change the identity of the new host to be the same as the old, this is crucial, without doing it will complain about licensing and fail to work.
hostnamectl set-hostname sc-old
  • Now edit /etc/host
vi /etc/hosts
10.1.2.3 SC-old SC-old.mydomain
  • Change IP by editing ifcfg-xxx file, or by what ever method you are used to.
vi /etc/sysconfig/network-scripts/ifcfg-ens192
  • For me, this was all I had to change, but YMMV, so check.
  • Reboot for the identity change to kick-in.
  • Now install the same version of Security Center that was on the old on, for me it was SecurityCenter-5.6.2.1-es7.x86.
rpm ivh SecurityCenter-5.6.2.1-es7.x86_64.rpm
Preparing...                          ################################# [100%]
/usr/bin/chown: invalid user: ‘tns:tns’
Backing up previous application files ... complete.
Updating / installing...
  • You can see it checked and found that the tns user was not there, don’t worry, it creates this for you during the re-install, a long with the new systemd type service start-up instead of the old initd.
  • Now cd into /opt, when installing I noticed that it didn’t recursively set all the permissions up, to fix do
chown -R tns:tns sc
  • At this point you are ready to test and see if you old version of SC works on the new OS, before upgrading to the new version of SC.
  • I started the SecurityCenter service, it started but there were a few errors, so I rebooted and checked
systemctl status SecurityCenter

….and all was good

[user@SC-old opt]# systemctl status SecurityCenter 
● SecurityCenter.service - SYSV: Starts and stops the Lightning daemons
Loaded: loaded (/etc/rc.d/init.d/SecurityCenter; bad; vendor preset: disabled)
Active: active (running) since Tue 2019-01-22 10:50:42 CST; 4s ago
Docs: man:systemd-sysv-generator(8)
Process: 7119 ExecStop=/etc/rc.d/init.d/SecurityCenter stop (code=exited, status=0/SUCCESS)
Process: 7158 ExecStart=/etc/rc.d/init.d/SecurityCenter start (code=exited, status=0/SUCCESS)
CGroup: /system.slice/SecurityCenter.service
├─7169 /opt/sc/support/bin/php -f /opt/sc/daemons/Jobd.php
├─7175 /opt/sc/support/bin/httpd -k start
├─7182 /opt/sc/support/bin/httpd -k start
├─7183 /opt/sc/support/bin/httpd -k start
└─7184 /opt/sc/support/bin/httpd -k start

Jan 22 10:50:41 SC-old systemd[1]: Starting SYSV: Starts and stops the Lightning daemons...
Jan 22 10:50:42 SC-old SecurityCenter[7158]: Starting SecurityCenter services: [ OK ]
Jan 22 10:50:42 SC-old systemd[1]: Started SYSV: Starts and stops the Lightning daemons.
  • Next I got the SC user to login to the admin web GUI and runs some scans and jobs etc to test that the old version was working before upgrading to the new version of SC, after a day of use they reported it was working great, so now we can upgrade and patch.
  • To upgrade, it only took a few minutes
rpm Uvh SecurityCenter-5.8.0-el7.x86_64.rpm
  • Restart SC for good measure and get the SC user to test again
systemctl start SecurityCenter
  • Now the last thing to do is patch
tar zxf SC-201812.1-5.8.0.tgz
cd SC-201812.1-5.8.0
./install.sh
  • That’s it all done!

 

Posted in Linux & Solaris | Tagged , , , , , , | Leave a comment

How to prevent audit logs appearing in /var/log/messages – RHEL7

Overview

For RHEL7.

This took more work than I anticipated.

My goal was this:

  1. Write audit logs to /var/log/audit
  2. Forward audit and syslog to central logging server
  3. Audit logs to NOT appear in /var/log/messages

The items 1 and 2 were easy, number 3, not so much.

audisp, rsyslogd and journald

After using rsyslog to send logs onto a central log collector, I noticed that all the audit logs are still going to ”/var/log/messages” as well as ”/var/log/audit”, not ideal, the only solution that worked, out of the many I tested, is detailed below, the audit logs still go to /var/log/audit (separate LV of course), /var/log/messages doesn’t fill up with audit logs, and all the logs get forwarded to a central log collector. The following configs below seem to work in general, but you may have to tweak them for your application, specifically the rate and burst limiting for journald.

Files of interest are:

  • /etc/audisp/plugins.d/syslog.conf – the plugin that forwards audit logs.
  • /etc/systemd/journald.conf – journald daemon config, change default values for rate-limiting otherwise message maybe be dropped.
  • /etc/rsyslog.conf – configure forwarding and exclude local0
  • /etc/auditd/rules.d/audit.rules – I needed exclude ”/var/lib/rsyslog” to stop logging loop, you may not have to do this.

Configure syslog plugin /etc/audisp/plugins.d/syslog.conf to be like the below

active = yes
direction = out
path = builtin_syslog
type = builtin 
#args = LOG_INFO
args = LOG_LOCAL0
format = string

Modify rsyslog.conf, the key change was adding the local0.none

.info;mail.none;authpriv.none;cron.none;local0.none      /var/log/messages

# Forward logs using TCP to central log collector

 *.* @@xxx.xxx.xxx.xxx:10514

And now journald.conf, the default is ”RateLimitInterval=30s” and ”RateLimitBurst=1000”, I had to change this as I was seeing dropped journalctl messages – ”journal: Suppressed 10361 messages from”. I had to exclude ”/var/lib/rsyslog” in audit.rules as it was creating thousands of messages, i.e. a logging loop. It is possible that rsyslog can drop messages as well, they will show up as ”imjournal: begin to drop messages due to rate-limiting”, to rate-limit rsyslog you make changes the rsyslog.conf file.

 RateLimitInterval=15s
 RateLimitBurst=3000

If you need to rate-limit rsyslog to can add the following, however, I’ve not tested this.

Old style syslog format

# File to store the position in the journal
$IMJournalStateFile imjournal.state
$imjournalRatelimitInterval 300
$imjournalRatelimitBurst 30000

Or if you want the new style, all one line.

module(load="imjournal" StateFile="/var/lib/rsyslog/imjournal.state" ratelimit.interval="300" ratelimit.burst="30000")

Testing rate-limiting

Tail your log file

tail -f /var/log/messages

From a second shell

journalctl -f

From a third shell generate some traffic

seq 1 3000 | logger
Posted in Linux & Solaris | Tagged , , , , | Leave a comment

Adding a custom unit file RHEL7, for swatch

swatch is a Linux tool for watching (it uses ”tail -f”) log files and then demonizes it.

I had a need to know when a host needed rebooting after yum-cron had updated the kernel, rather than logging into each host I decided I could make use of ”swatch”, in my case, it watches the ”yum.conf” file for the string ”Installed: kernel”, if the string appears in the log it sends an email out.

I have not found away for a single daemon instance to watch more than a single log file, there are some examples online showing this configuration, but when I tested it only ever worked on a single file being watched. After further reading and research this is the expected behavior.

Installing swatch, configuring start-up to watch a single log source

  • First install it

yum install swatch

  • Next create a swatch directory

mkdir -p /etc/swatch

  • Create a ”swatch” conf file for what you want to monitor

vi swatch-yum.conf
watchfor /Installed: kernel/
echo bold
mail=root@localhost, subject=”New kernel has been installed – reboot required”

  • Now create a ”.swatchrc file in ”/root”

touch /root/.swatchrc

  • You can now test the configuration, you might want to change it to something that is easy to test and validate, ssh logins or something.

swatch –config-file=/etc/swatch/swatch-yum.conf –tail-file=/var/log/yum.log –daemon

  • Check it started

ps -ef | grep swatch

root 1187 1 0 12:59 ? 00:00:00 /usr/bin/swatch –config-file=/etc/swatch/swatch-yum.conf –tail-file=/var/log/yum.log –pid-file=/var/run/sw

  • Ok, so we know it works, so go ahead and kill it to stop it as we need to create a custom unit file (service) so it can start at boot.

kill -15 1187

  • Because this is systemd you do not need to touch init scripts or anything, but you do need to create a custom unit file, cd to ”/etc/systemd/system”
  • Next, create your unit file using an editor

vi swatch.service
[Unit]
Description=Swatch Log Monitoring Daemon
After=syslog.target network.target auditd.service sshd.service

[Service]
ExecStart=/usr/bin/swatch –config-file=/etc/swatch/swatch-yum.conf –tail-file=/var/log/yum.log –pid-file=/var/run/swatch.pid –daemon
ExecStop=/usr/bin/kill -s KILL $(cat /var/run/swatch.pid)
Type=forking
PIDFile=/var/run/swatch.pid

[Install]
WantedBy=multi-user.target

  • Create PIDs

touch /var/run/swatch-secure.pid

touch /var/run/swatch-secure.pid

  • Now reload it, start it and enabled and check its status.

systemctl daemon-reload
systemctl start swatch.service
systemctl enable swatch.service
systemctl status swatch.service

If it doesn’t work , try rebooting

  • If all has gone well it should look like this
  • systemctl status swatch.service
    ● swatch.service – Swatch Log Monitoring Daemon
    Loaded: loaded (/etc/systemd/system/swatch.service; enabled; vendor preset: disabled)
    Active: active (running) since Wed 2017-10-11 12:59:05 CDT; 39min ago
    Main PID: 1187 (/usr/bin/swatch)
    CGroup: /system.slice/swatch.service
    ├─1187 /usr/bin/swatch –config-file=/etc/swatch/swatch-yum.conf –tail-file=/var/log/yum.log –pid-file=/var/run/sw
    └─1188 /usr/bin/tail -n 0 -F /var/log/yum.log
    Oct 11 12:59:03 rhel7-.local systemd[1]: Starting Swatch Log Monitoring Daemon…
    Oct 11 12:59:05 rhel7-.local systemd[1]: Started Swatch Log Monitoring Daemon.

Configuring swatch to monitor more than a single log file source

This is pretty much the same as the above, but I split things up a little, it makes it easier to manage. I’ve assumed you’ve done the initial install, also note that the file paths have changed in the service start-up files to reflect having to run more than a single instance of swatch.

  • Create your swatch.conf files, in this case I’m going watch /var/log/yum.log for kernel updates and /var/log/secure for invalid user and failed password attempts.

vi swatch-yum.conf
watchfor /Installed: kernel/
echo bold
mail=root@localhost, subject=”New kernel has been installed – reboot required”

vi swatch-secure.conf
watchfor /Invalid user/
echo bold
mail=root@localhost, subject=”Invalid user”

watchfor /Failed password/
echo bold
mail=root@localhost, subject=”Failed password”

  • Next you need to create 2 custom unit file (service), one for each file we want to monitor. Note that we have to created two separate PID files now.

vi swatch-yum.service
[Unit]
Description=Swatch Log Monitoring Daemon
After=syslog.target network.target auditd.service sshd.service

[Service]
ExecStart=/usr/bin/swatch –config-file=/etc/swatch/swatch-yum.conf –tail-file=/var/log/yum.log –pid-file=/var/run/swatch-yum.pid –daemon
ExecStop=/usr/bin/kill -s KILL $(cat /var/run/swatch-yum.pid)
Type=forking
PIDFile=/var/run/swatch-yum.pid

[Install]
WantedBy=multi-user.target

And..

vi swatch-secure.service
[Unit]
Description=Swatch Log Monitoring Daemon
After=syslog.target network.target auditd.service sshd.service

[Service]
ExecStart=/usr/bin/swatch –config-file=/etc/swatch/swatch-secure.conf –tail-file=/var/log/yum.log –pid-file=/var/run/swatch-secure.pid –daemon
ExecStop=/usr/bin/kill -s KILL $(cat /var/run/swatch-secure.pid)
Type=forking
PIDFile=/var/run/swatch-secure.pid

[Install]
WantedBy=multi-user.target

  • Now reload it, start it and enabled and check its status.

systemctl daemon-reload
systemctl start swatch-yum
systemctl enable swatch-yum
systemctl start swatch-secure
systemctl enable swatch-secure

You can test it by echoing the watch text to the log

echo Failed password >> /var/log/secure

Posted in Linux & Solaris | Tagged , , , , , , | Leave a comment

2008 Kia Sedona – A/C stops working.

2008 Kia Sedona LX, A/C was working great, we stop, go into a store, come out and now the A/C does not work. Blowers still blow, and there have been no unusual noises from the engine compartment.

The 7.5A fuse was blown, so replaced, and on turning the A/C on it blows again.

Here is how I diagnosed and fixed it.

A/C was working fine
No mechanical noises
A/C suddenly ceases to work
A/C 7.5 fuse was blown
A/C fuse replaced, immediately blows the fuse
A/C relay checks out good
Removed A/C relay and measured from pin/socket 87 to ground, reads open-circuit
Removed old field coil, it measures between 1.4 and 1.7 ohms
New coil measured between 3.4 and 3.5 ohms
Disconnect battery
Removed under tray/splash shielding from underneath minivan
Removed pulley wheel from A/C compressor
Install new field coil
Re-assemble pulley etc
A/C now works

The correct part is Kia K97641-4D900 A/C Field Coil, cost $53.50 + tax from a Kia main dealer.

Posted in Vehicle maintenance | Leave a comment

Adding vmware-tools to Centos5 or RHEL

Adding vmware-tools to RHEL6 or greater is easy, assuming you have the EPEL repo enabled all you do is:

yum install open-vm-tools.x86_64

That’s it your done.

Things are a tad more complicated for lower version of RHEL, for example RHEL5, some of this is down to the way VMWare have changed the way they support vmware-tools on Linux, they scrapped having the vm-tools.rpm the linux.iso which you use to push out from the vSpehre client,  they now advise you to use your disto’s repos.

Here’s how you do for RHEL5.

Check your version/architecture etc

[root@foo ~]# cat /etc/redhat-release 
 CentOS release 5.6 (Final)
 [root@foo ~]# uname -a
 Linux foo.local 2.6.18-308.20.1.el5 #1 SMP Tue Nov 13 10:15:12 EST 2012 x86_64 x86_64 x86_64 GNU/Linux

Create repo file

touch /etc/yum.repos.d/vmware-tools.repo
Add this and adjust for your release/architecture, tip, DO NOT USE the ''latest'' repo, it has caused issues, always go for a point/named release

 [vmware-tools]
 name=VMware Tools
 #baseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel6/x86_64 # DO NOT USE
 #baseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel6/i386
 #baseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel5/i386
 #baseurl=http://packages.vmware.com/tools/esx/5.1latest/rhel5/x86_64
 baseurl=http://packages.vmware.com/tools/esx/5.5u2/rhel5/x86_64   <====== THIS WORKS, NOTE THE PATH
 #baseurl=http://packages.vmware.com/tools/esx/4.0latest/rhel6/x86_64
 #baseurl=http://packages.vmware.com/tools/esx/4.0latest/rhel5/i686
 #baseurl=http://packages.vmware.com/tools/esx/4.0latest/rhel6/i686
 enabled=1
 gpgcheck=1

Download the keys, you may have to export proxy settings

 wget http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-DSA-KEY.pub
 wget http://packages.vmware.com/tools/keys/VMWARE-PACKAGING-GPG-RSA-KEY.pub

Import the keys

 rpm --import ./VMWARE-PACKAGING-GPG-RSA-KEY.pub
 rpm --import ./VMWARE-PACKAGING-GPG-DSA-KEY.pub

Now you are ready to install – IMPORTANT, install ”vmware-tools-esx-kmods.x86_64” first
yum install vmware-tools-esx-kmods.x86_64 – and it can take some time.

 yum install vmware-tools-esx-nox.x86_64

”vmware-tools-esx-nox.x86_64” is for the non GUI version or headless, which is what we want for us as we do not run GNOME etc on our servers.

Amongst other things, having vmware-tools installed allows the full potential of the vmxnet3 NIC to be exploited, and allows you to shutdown the guest from with the vSphere client.

Posted in Linux & Solaris | Tagged | 4 Comments

2001 GE Spacemaster XL1800 over-stove microwave oven repair

Microwave went faulty a few days ago, with the following symptoms, play close attention to the symptoms, as this was leading me to think it was a cap going out, and nothing that serious as the oven was still heating. There are quite a few articles out there on the web where people have replaced transformers, megatron’s, controller boards etc (each costing anywhere from $25 – $100 each), when all they needed to replace was 10 cent cap.

  • Garbled buzzing/beeping sound when the door opened, or any cooking function was selected
  • The buzzing/beeping sound changes when any of the buttons are pressed, it seemed unwanted electrical signals were getting to the piezo speaker/buzzer
  • The oven light flickered in time with the buzzing sound
  • Turntable not turning – I recently replaced the turntable motor so I knew it was ok
  • It would still heat food even though it was doing all of the above

So, disconnect power, then take the oven off the wall, remove the outer case, then remove the circuit board that hosts the display and control panel.

A guy on appliancepartspros.com forum said cap C1 50v 220uF goes out, so I take a look at that one and the others nearby, they all look ok.  I decided to remove C1 and the other 3 that make up a block of 4, C2 50v 47uF, C3 16v 470uF and C4 2200uF 16v. I tested C1 with my multimeter and sure enough it’s dud, just for good measure I also replaced the others in that block of 4, even though they tested ok.

I then decided to test the other caps next to this block, this column of caps are all the same physical size, unlike the first block of 4 which have various sizes – they are C5 100uF 10v, C6 22uF 16v, C14 22uF 16v, C15 4.7uF 50v. Glad I did, as found that C5 was also dud.

Resistor R51 39k ohm also looked as if it had got hot, which is possible as it works along with the relay which has been chattering away, so I replaced that as well, it did test ok though.

So, to summarized, replace 2 faulty caps and a resistor

C1 50v 220uF

C5 100uF 10v replaced with 100uF 16v, remember it’s usually ok to replace with a higher voltage cap, never lower though.

R51 39k 1% resistor

I leave up to you to decide if you want to replace the other caps that test ok.

oven-pcb-small

Posted in Electronics & Gadgets | Tagged , , | Leave a comment