Sysstat

For disk I/O trending there are a few options. the sar command from sysstat. By default, it gives output like this:

# sar
Linux 2.6.32-642.11.1.el6.x86_64 (test.geekdecoder.com)    01/26/2017      _x86_64_        (2 CPU)

12:00:01 AM     CPU     %user     %nice   %system   %iowait    %steal     %idle
12:10:01 AM     all      0.03      0.00      0.03      0.09      0.00     99.85
12:20:01 AM     all      0.27      0.00      0.04      0.00      0.00     99.69
12:30:01 AM     all      0.33      0.00      0.05      0.00      0.00     99.62
12:40:01 AM     all      1.01      0.00      0.18      1.18      0.00     97.63
12:50:01 AM     all      0.15      0.00      0.04      0.00      0.00     99.80
01:00:01 AM     all      0.16      0.00      0.03      0.00      0.00     99.80
01:10:01 AM     all      0.22      0.00      0.04      0.00      0.00     99.74
01:20:01 AM     all      0.36      0.00      0.05      0.00      0.00     99.59
01:30:01 AM     all      0.23      0.00      0.04      0.00      0.00     99.74
01:40:01 AM     all      0.70      0.00      0.25      6.23      0.00     92.82
01:50:01 AM     all      0.13      0.00      0.03      0.00      0.00     99.83
02:00:01 AM     all      0.16      0.00      0.04      0.00      0.00     99.80
02:10:01 AM     all      0.21      0.00      0.04      0.00      0.00     99.75
02:20:01 AM     all      0.30      0.00      0.04      0.00      0.00     99.66
02:30:01 AM     all      0.14      0.00      0.03      0.00      0.00     99.83
02:40:01 AM     all      0.20      0.00      0.03      0.00      0.00     99.76
02:50:01 AM     all      0.32      0.00      0.05      0.00      0.00     99.63
03:00:01 AM     all      0.21      0.00      0.03      0.00      0.00     99.76
03:10:01 AM     all      0.19      0.00      0.04      0.00      0.00     99.77
03:20:01 AM     all      0.12      8.44      2.69     10.79      0.00     77.96
03:30:01 AM     all      0.13      4.52      8.36      0.30      0.00     86.69
03:40:01 AM     all      0.48      0.00      0.05      0.02      0.00     99.45
03:50:01 AM     all      0.15      0.00      0.04      0.01      0.00     99.80
04:00:01 AM     all      0.13      0.00      0.03      0.00      0.00     99.84
04:10:01 AM     all      0.48      0.00      0.06      0.04      0.00     99.43
04:20:01 AM     all      0.17      0.00      0.05      0.11      0.00     99.67
04:30:01 AM     all      0.22      0.00      0.04      0.00      0.00     99.74
04:40:01 AM     all      0.89      0.00      0.08      0.16      0.00     98.87
04:50:01 AM     all      0.55      0.00      0.07      0.05      0.00     99.34
05:00:01 AM     all      0.11      0.00      0.04      0.01      0.00     99.85
05:10:01 AM     all      0.09      0.00      0.03      0.00      0.00     99.87
05:20:01 AM     all      0.17      0.00      0.04      0.00      0.00     99.80
05:30:01 AM     all      0.27      0.00      0.05      0.08      0.00     99.60

Read More

If your site is hacked, the check for edited files. Find all files modified within 30 days and then parsed that for base64 decodes (used to encrypt/hide coding)

Find the coding itself

# cd /home/domain/public_html
# find . -type f -ctime -30 -iname "*.php" -exec grep "base64_decode" {} \;

Results

# find . -type f -ctime -30 -iname "*.php" -exec grep "base64_decode" {} \;
        $buf .= base64_decode($util->GetRandom($bytes, 0));
                                $data = base64_decode($data);
                        $data = base64_decode( $data );
                        $data = base64_decode($data);
                $value = base64_decode($this->_currentTagContents);
                $challenge = base64_decode(substr($this->last_reply, 4));
                $decoded = base64_decode( $value['encoded_serialized_instance'], true );
                                        $flac->setStringMode(base64_decode($ThisFileInfo_ogg_comments_raw[$i]['value']));
                                        $data = base64_decode($ThisFileInfo_ogg_comments_raw[$i]['value']);
    $decoded_sig = base64_decode($signature);
        $uncompressed =$this->_uncompress(base64_decode(strtr($compressed, '-_', '+/')));
                        return base64_decode($value);
                        return base64_decode($value);
                $this->_accountKey = base64_decode($accountKey);
                $this->_accountKey = base64_decode($value);
                                        base64_decode((string)$xmlMessages[$i]->MessageText)
            return base64_decode($sessionRecord->serializedData);
        $incomingSignature = base64_decode($message->get('Signature'));
            if (base64_decode($token,true)){
                $decoded_token = base64_decode($token,true);
            $raw_data = base64_decode($data);
    return base64_decode($b64);
                $challenge = base64_decode($challenge);
                $challenge = base64_decode(substr($this->last_reply, 4));
    return base64_decode($b64);
eval(base64_decode('TWFuIGlzIGRpc3Rpbmd1aXNoZWQsIG5vdCBvbmx5IGJ5IGhpcyByZWFzb24sIGJ1dCBieSB0aGlz'.
                $expected_raw_md5 = base64_decode( $expected_md5 );

As we can see – here is a excerpt from the base64 code. Now, lets find the file. To find the base64 coding, copy a bit of the base64 and run;


# find . -type f -ctime -30 -iname "*.php" -exec grep -l "TWFuIGlzIGR" {} \;
./basecode.php


Where that gibberish in the grep is a small tidbit of the base64 coding
The first one finds the coding itself. The second finds the file containing the coding.

Change to root

$ su -
Password: 

adduser mynewuser

Now set the password for the new user:


passwd mynewuser

Step 2: Grant Root Privileges to the User
For a refresher on editing files with vim see: New User Tutorial: Overview of the Vim Text Editor


visudo

Find the following code:


## Allow root to run any commands anywhere
root ALL=(ALL) ALL

In this case, we’re granting root privileges to the user mynewuser . Add the following below that code:


mynewuser ALL=(ALL) ALL

nstall geoiplookup on Debian, Ubuntu or Linux Mint:


$ sudo apt-get install geoip-bin

To install geoiplookup on Fedora:


$ sudo yum install geoip

To install geoiplookup on CentOS, first enable EPEL repository

$ sudo yum install epel-release

then use yum command:


$ sudo yum install geoip

The default installation of geoiplookup comes with GeoIP.dat database file which is located in /usr/share/GeoIP. With this database, you can look up the country information only.


$ geoiplookup 23.66.166.151
GeoIP Country Edition: US, United States

You can download additional GeoIP databases from MaxMind, which give you more detailed information about IP addresses beyond country info. You can also download more up-to-date GeoIP.dat from the site. This is recommended because GeoIP.dat may have already been outdated by the time you install it from Linux repositories. The GeoIP databases available on MaxMind website are updated every month.

To install additional GeoIP databases from MaxMind, do the following. You may want to set up a monthly cronjob to automate this process.


$ wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz
$ wget http://geolite.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz
$ wget http://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz
$ gunzip GeoIP.dat.gz
$ gunzip GeoIPASNum.dat.gz
$ gunzip GeoLiteCity.dat.gz
$ sudo cp GeoIP.dat GeoIPASNum.dat GeoLiteCity.dat /usr/share/GeoIP/

Now if you re-run geoiplookup, you will see the additional AS number information of an IP address. This basically tells you which administrative domain the IP address belongs to.


$ geoiplookup 128.112.119.209
GeoIP Country Edition: US, United States
GeoIP ASNum Edition: AS88 Princeton University

When run without any parameter, geoiplookup tool automatically uses GeoIP.dat and GeoIPASNum.dat only, but not use GeoLiteCity.dat. The latter can give you city-level information.

To obtain city-level geolocation information, explicitly tell geoiplookup to use GeoLiteCity.dat database.


$ geoiplookup -f /usr/share/GeoIP/GeoLiteCity.dat 23.66.166.151
GeoIP City Edition, Rev 1: US, MA, Cambridge, 02142, 42.362598, -71.084297, 506, 617

The output includes state, city, zipcode, latitude and longitude. The accuracy of the inferred location varies across different countries and networks. For example, the geolocation result tends to be more accurate for broadband IP addresses, but not as accurate for mobile networks.

You can try ipinfo.io online service. Unlike other services, ipinfo.io provides JSON-based geolocation API, so you can easily look up geolocation from the command line, using tools like curl.


$ curl ipinfo.io/23.66.166.151

CentOS 7 minimal systems, use the commands “ip addr” and “ip link” to find the details of a network interface card. To know the statistics use “ip -s link”.

To view the details of the network interface cards, enter the following commands:

# ip addr

To view the statistics of your network interfaces, enter the command:


# ip link

OR


# ip -s link

How do I enable and use “ifconfig” Command in CentOS 7 minimal servers?


# yum provides ifconfig


# yum install net-tools

Timestamped bash history logging may be a great idea for logging changes. An example would look like this:

 
Mar 28 15:08:26 database root: root@192.168.1.1 [10882]: shutdown -r now [0]
Mar 28 15:08:31 database root: root@192.168.1.1 [10882]: cd /var/log [0]

This can be done by adding 2 lines to the root .bashrc file or as a custom file in /etc/profile.d/ folder:

 
whoami="$(whoami)@$(echo $SSH_CONNECTION | awk '{print $1}')"
export PROMPT_COMMAND='RETRN_VAL=$?;logger -p local6.debug "$whoami [$$]: $(history 1 | sed "s/^[ ]*[0-9]\+[ ]*//" ) [$RETRN_VAL]"'

then a single line added to /etc/rsyslog.conf:

 
local6.*                /var/log/bash.log

After this all commands will be logged to the /var/log/bash.log file.

My VM on ESXi have all different dates. Here is how toset them the same.

date

First, you’ll need to install NTP. For Debian or Ubuntu, that would be this command:


# sudo apt-get install ntp

For Redhat or CentOS, you’ll need to use this one:


# yum install ntp

Then you’ll want to edit the /etc/ntp.conf file, which is quite possibly already filled out for you.


# nano /etc/ntp.conf

You can get a list of server addresses at www.pool.ntp.org. Here are the North America ones:


server 0.north-america.pool.ntp.org
server 1.north-america.pool.ntp.org
server 2.north-america.pool.ntp.org
server 3.north-america.pool.ntp.org

Set it to run at boot

# chkconfig --list
ntpd            0:off   1:off   2:off   3:off   4:off   5:off   6:off

For Redhat and CentOS 6

chkconfig ntpd on
# ntpd            0:off   1:off   2:on    3:on    4:on    5:on    6:off

For Redhat 7 and CentOS 7

# systemctl enable ntpd.service

CentOS 6 restart or start the NTPD service:


# /etc/init.d/ntpd restart

CentOS 7 Restart

# /bin/systemctl start  ntpd.service

If you want to update the time right now, you can stop the NTP service and then run the following command, swapping out your preferred server for pool.ntp.org.


/usr/sbin/ntpdate pool.ntp.org

Check your Site at https://drownattack.com/#test

or run the following:

http (replace IP with your server ip)

openssl s_client -connect 192.168.1.1:443 -ssl2

postfix or other email MTA

openssl s_client -connect xxxxxxxxxxxxx.com:25 -starttls smtp -ssl2

Check your version

# openssl version
OpenSSL 1.0.1e-fips 11 Feb 2013

To protect against DROWN, server operators need to ensure that their private keys are not used anywhere with server software that allows SSLv2 connections. This includes web servers, SMTP servers, IMAP and POP servers, and any other software that supports SSL/TLS. You can use the form above to check whether your server appears to be exposed to the attack.

Disabling SSLv2 can be complicated and depends on the specific server software. We provide instructions here for several common products:
OpenSSL: OpenSSL is a cryptographic library used in many server products.

For users of OpenSSL, the easiest and recommended solution is to upgrade to a recent OpenSSL version. OpenSSL 1.0.2 users should upgrade to 1.0.2g. OpenSSL 1.0.1 users should upgrade to 1.0.1s. Users of older OpenSSL versions should upgrade to either one of these versions.

More details can be found in this OpenSSL blog post.

Postfix

add the following to your main.cf config file:

smtpd_tls_security_level = may
smtpd_tls_auth_only = yes
smtpd_tls_protocols = SSLv3, TLSv1, !SSLv2
smtpd_tls_cipherlist = ALL:!ADH:RC4+RSA:+HIGH:+MEDIUM:-LOW:-SSLv2:-EXP:-eNULL

then just do a postfix reload. eg.

# /etc/init.d/postfix reload

to check it’s actually disabled use the following openssl command

openssl s_client -connect xxxxxxxxxxxxx.com:25 -starttls smtp -ssl2

which should give you something like this:

Code:
CONNECTED(00000003)
write:errno=104

as opposed to the SSL3 test

openssl s_client -connect xxxxxxxxxxxxx.com:25 -starttls smtp -ssl3
CONNECTED(00000003)

....

SSL-Session:
    Protocol  : SSLv3
    Cipher    : DHE-RSA-AES256-SHA
    Session-ID: AB6C68095ADFA60119F4845485D840A62DEB5B519E803510692F1BBCD71199CD
    Session-ID-ctx:
    Master-Key: 8BA2691B5EEEA9AE6752D804F0B0700C0792E7AD6BC6D19416B819EF5014FA80FAC51E124DFFB083C70A547AF522C149
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1292001315
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
---
220 mail.xxxxxxxxx.net ESMTP Postfix