Change OpenSSH Port CentOS

Want to change the ssh port for ssh? Here is a good tutorial.
(Note: If you are making these changes – ssh to the server and keep the terminal open as you make the changes. Test with a new terminal. This way if something is amiss – you are not locked out.)

Edit /etc/ssh/sshd_config, enter:

# vi /etc/ssh/sshd_config

Note:
The strategy used for options in the default sshd_config shipped with OpenSSH is to specify options with their default value where possible, but leave them commented. Uncommented options change a default value.

Uncomment the following and edit to set the port to 10221:

Port 10221

ListenAddress option

Note: If you have multiple IP address on the server, add you IP addresses.

ListenAddress as follows :

## bind sshd to two ip address on a non-standard port ##
ListenAddress 192.168.1.5:10221
ListenAddress 203.1.2.3:10221

Save and close the file.

Before you restart or reload sshd server. You need to update SELinux configuration or Firewall settings (iptables).

You also need to update firewall settings so that users can login using TCP # 10221. Edit,

/etc/sysconfig/iptables and open sshd port 10221:
# vi /etc/sysconfig/iptables

Edit/append as follows:

 
## delete or comment out port 22 line ##
## -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
## open port 22
-A INPUT -m state --state NEW -m tcp -p tcp --dport 10221 -j ACCEPT

Save and close the file. If you are using IPv6, edit /etc/sysconfig/ip6tables file too. Temporally, stop the firewall so that you will not loose the connectivity to the server:

# service iptables stop
# service ip6tables stop

Restart sshd on a CentOS

# service sshd restart

Verify new port settings with the following netstat command:

# netstat -tulpn | grep sshd

Finally, start the firewall:

# service iptables start
## IPv6 ##
# service ip6tables start

Now, login with a different terminal to check the settings.

Categories SSH

Apache Compromise

Your server is running very slow.

Top looks Okay. Loads are not insane. Ram is high, but not out of bounds. sar shows high i/o wait times. Swap usage is not an issue. MySql process show hundreds of sleeping processes. Restarting MySql or Apache clears them, but then they start right back up.

WHAT? This make no sense!

tail /var/www/vhost/domain.com/statistics/log/access_log

[07/Dec/2013:17:08:17 -0700] “GET /local/image_product480000_1/mlomeupenvtb2012tb201212tb201212044071d032736e44d9b3e5b914d378f9e2jpg.jpg HTTP/1.0″ 200 16322 “-” “-”
[07/Dec/2013:17:08:17 -0700] “GET /local/image_product480000_1/pics2dsstaticcomprodimg165178300jpg.jpg HTTP/1.0″ 200 12690 “-” “-”
[07/Dec/2013:17:08:17 -0700] “GET /local/image_product480000_1/slimagesmacyscomisimageMCYproducts4optimized515264fpxtif.jpg HTTP/1.0″ 200 10497 “-” “-”
[07/Dec/2013:17:08:17 -0700] “GET /local/image_product480000_1/plefuxcom6120111219A0361000WNipadiphonebatteriesexternal5000mah3751965bigjpg.jpg HTTP/1.0″ 200 9638 “-” “-”
[07/Dec/2013:17:08:17 -0700] “GET /local/image_product480000_1/taylorgiftscomimagesp43126500jpg.jpg HTTP/1.0″ 200 59977 “-” “-”

Notice how these connections are coming from the server itself instead of from an external IP.

Now look at who is connecting to the server:

netstat -nat | grep :80 | gawk '{ print $5; }' | gawk -F: '{ print $1 }' | sort | uniq -c | sort -n

2 66.249.73.222
3 157.55.32.143
3 199.30.20.68
3 199.30.20.76
4 131.253.24.85
4 199.30.20.106
4 23.67.252.11
4 65.55.55.229
5 174.125.28.4
12 23.67.252.59
325 64.150.184.165

Again, all coming from the server. The solution to the problem was discovered in /tmp

ls -la /tmp

total 44532
drwxrwxrwx 4 root root 3522560 Dec 7 17:12 .
drwxr-xr-x 24 root root 4096 Dec 6 13:03 ..
drwx–x–x 2 apache apache 4096 Feb 29 2012 .bash
-rw-r–r– 1 apache apache 37281 Oct 13 10:21 .dsf
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.1
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.2
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.3
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.4
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.5
-rw-r–r– 1 apache apache 37287 Oct 13 17:46 .dsf.6
-rw-r–r– 1 apache apache 37281 Oct 13 18:18 .dsf.7
-rw-r–r– 1 apache apache 37281 Oct 13 18:18 .dsf.8

now,

ls -la /tmp/.bash

total 27392
drwx–x–x 2 apache apache 4096 Feb 29 2012 .
drwxrwxrwx 4 root root 3522560 Dec 7 17:14 ..
-rwx–x–x 1 apache apache 146 Nov 12 2012 1
-rwxr-xr-x 1 apache apache 323 Jan 13 2011 autorun
-rwx–x–x 1 apache apache 8922 Jan 23 2006 b
-rwx–x–x 1 apache apache 19557 May 9 2005 b2
-rwxr-xr-x 1 apache apache 11445 Jan 5 2011 bang
-rwxr-xr-x 1 apache apache 12321980 Feb 29 2012 bangnew
-rwxr-xr-x 1 apache apache 11824732 Jan 23 2011 bangold
-rw-r–r– 1 apache apache 44 Aug 3 03:28 cron.d
-rwx–x–x 1 apache apache 14679 Nov 2 2005 f4
-rwxr-xr-x 1 apache apache 15988 Sep 7 2002 juno
-rw-r–r– 1 apache apache 11 Aug 3 03:28 mech.dir
-rwx–x–x 1 apache apache 566 Jan 20 2013 mech.set
-rwxr-xr-x 1 apache apache 27 Jan 11 2011 run
-rwx–x–x 1 apache apache 152108 Jan 11 2011 sshd:
-rwxr-xr-x 1 apache apache 17 Nov 5 2008 start
-rwxr-xr-x 1 apache apache 8231 Feb 29 2012 std
-rwxr-xr-x 1 apache apache 13399 Aug 6 2000 stealth
-rwx–x–x 1 apache apache 8790 Jan 23 2006 stream
-rwxr-xr-x 1 apache apache 17690 Feb 6 1996 synk
-rwxr-xr-x 1 apache apache 6442 Jun 23 2011 talk
-rwxr–r– 1 apache apache 166 Aug 3 03:28 update
-rwx–x–x 1 apache apache 14841 Jul 22 2005 v
-rwxr-xr-x 1 apache apache 14911 Mar 6 2002 v2

End Result

End result: This server ahs been root compromised. The only solution is to reinstall and slave drive the existing compromised drive.

Verisign SSL Certificates

Verisign SSL Certificates

You have generated a certificate request (CSR and private key) using plesk. You would like to know how to complete the certificate request process. The Verisign digital certificate can be downloaded in the X.509 format as three files. The three files are designated as:

1) End Entity Certificate

2) First Intermediate Certificate

3) Second Intermediate Certificate

When the Plesk CSR function is submitted, it prompts for only two values:

1) Certificate

2) CA certificate

How are these values determined since there are 3 files returned, but Plesk prompts for 2 values?

1) indicate which Verisign file is mapped to the Certificate field.

2) indicate which Verisign file is mapped to the CA certificate field.

Answer:

The End Entity Certificate is your SSL created to match your public key. The contents of the entity certificate should be placed into the Certificate field of Plesk.

The contents of both the First Intermediate and Second Intermediate will need to be placed into Plesk’s CA Certificate field. These should be pasted in order to create a two part chain certificate.

You should be able to open all of these in the notepad or wordpad programs to view the plain text contents of each certifcate. This will facilitate copy/pasting of the content into Plesk for all certificate fields.

Migrate MySQL from Slaved Drive

Migrate MySQL from Slaved Drive

1. Mount the slave drive. We’ll assume you mounted it at /media/slave

Find the drive:

# fdisk -l
Disk /dev/sda: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000374d4

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           1          64      512000   83  Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2              64         587     4194304   82  Linux swap / Solaris
Partition 2 does not end on cylinder boundary.
/dev/sda3             587      121602   972054528   83  Linux

Disk /dev/sdb: 1000.2 GB, 1000204886016 bytes
255 heads, 63 sectors/track, 121601 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1   *           1          62      497983+  83  Linux
/dev/sdb2              64         584     4184932+  82  Linux swap / Solaris
/dev/sdb3             585      121598   972044955   82  Linux swap / Solaris

sdb is the drive – the slaved drive after the reinstall
Check to see if it is ext3 or ext4

# blkid /dev/sdb3
/dev/sdb3: UUID="52721885-a9af-45e9-89f5-5f26ffca55dd" TYPE="ext3"

Mount according to ext3 or ext4

mount -t ext3 /dev/sdb3 /media/slave

Add to fstab

# nano /etc/fstab
/dev/sdb3   /media/slave   ext3   default 0   1 

2. Edit the MySQL config file to point to the slave’s mysql databases

nano /etc/my.cnf

# datadir = /var/lib/mysql
datadir = /media/slave/var/lib/mysql


service mysqld restart

3. Export the required database


mysqldump -u root -p[root_password] [database_name] > dumpfilename.sql

for plesk:


mysqldump -u admin -p`cat /etc/psa/.psa.shadow` [database_name] > dumpfilename.sql

4. Reset the Mysql path and import the file


nano /etc/my.cnf


datadir = /var/lib/mysql
# datadir = /media/slave/var/lib/mysql


service mysqld restart


mysql -u root -p[root_password] [database_name] < dumpfilename.sql

Plesk:


mysql -u admin -p`cat /etc/psa/.psa.shadow` [database_name] < dumpfilename.sql

Hotkey for Plesk specifically mounted to /olddrive/:

Before migrating, make sure the database you are migrating has already been created in Plesk with the correct username and password.

1. Log onto your server as root using ssh.

2. Edit the mysql config file to use the slave drive
nano /etc/my.cnf

3. Comment out the current path, add your slaved drive’s path and save the file


# datadir = /var/lib/mysql
datadir = /olddrive/var/lib/mysql


(save the file using then to exit.

4. Restart mysql to load the new settings


service mysqld restart

5. Create a dump file of the desired database


mysqldump -u admin -p`cat /etc/psa/.psa.shadow` [database_name] > /tmp/database_name.sql

(Repeat this step for all databases that need to be imported)

6. Repeat step 2-4 and reset the original setting


datadir = /var/lib/mysql
# datadir = /olddrive/var/lib/mysql

7. Import the database


mysql -u admin -p`cat /etc/psa/.psa.shadow` [database_name] < /tmp/database_name.sql

(Repeat this step for all .sql files created in step 5)

Optimize Mysql

Did you know that mysql comes with configuration files for better optimization? These files are located at:

/usr/share/doc/mysql-server-5.1.71/my-huge.cnf
/usr/share/doc/mysql-server-5.1.71/my-innodb-heavy-4G.cnf
/usr/share/doc/mysql-server-5.1.71/my-large.cnf
/usr/share/doc/mysql-server-5.1.71/my-medium.cnf
/usr/share/doc/mysql-server-5.1.71/my-small.cnf
/usr/share/man/man5/openssl.cnf.5ssl.gz

Or

/usr/share/mysql/my-huge.cnf
/usr/share/mysql/my-innodb-heavy-4G.cnf
/usr/share/mysql/my-large.cnf
/usr/share/mysql/my-medium.cnf
/usr/share/mysql/my-small.cnf

Stop mysql

# service mysqld stop
Stopping mysqld:  [  OK  ]

backup the original my.cnf file.

# mv /etc/my.cnf /etc/my.cnf.original

Move the new config file to /etc based on your needs:

# cp /usr/share/mysql/my-medium.cnf /etc/my.cnf

Restart MySQL:

# service mysqld start

Other Resources:
http://geekdecoder.com/using-mysqltuner/
http://www.hardwaresecrets.com/article/How-to-Optimize-a-MySQL-Server/1747
http://www.thegeekstuff.com/2011/03/sar-examples/
http://www.cyberciti.biz/tips/top-linux-monitoring-tools.html
http://www.codero.com/knowledge-base/questions/319/How+to+install+mytop+for+database+performance+monitoring%3A
http://httpd.apache.org/docs/current/misc/perf-tuning.html
http://dev.mysql.com/doc/refman/5.0/en/optimization.html

Mytop

Install the repository
CentOS 5.x

rpm -Uhv http://apt.sw.be/redhat/el5/en/i386/rpmforge/RPMS/rpmforge-release-0.3.6-1.el5.rf.i386.rpm

CentOS 6.x 64 bit

rpm -Uhv http://apt.sw.be/redhat/el6/en/i386/rpmforge/RPMS/rpmforge-release-0.5.3-1.el6.rf.x86_64.rpm

Install mytop

yum -y install mytop

Configuration

Odds are, you will get an error initially running this program unless you configure it first.

whereis mytop

Example: /usr/bin/mytop or /usr/local/bin/mytop

nano /usr/bin/mytop or nano /usr/local/bin/mytop

Locate the following lines:

my %config = (
delay => 5,
host => ‘localhost’,
db => ‘test’,
user => ‘root’, …

change the following line and save the file:

db => ‘mysql’,
Run mytop

Plesk

mytop -u admin -p`cat /etc/psa/.psa.shadow`

WHM / cPanel

mytop

Now – go here to learn how to use it!
http://linux.die.net/man/1/mytop

Install SSL Certificate in Plesk

CSRs can actually be generated within Plesk by following the steps I have listed below. I have also included the steps on how to install your SSL after getting it from the SSL provider. Please feel free to let us know if you are in need of anything else or have any further questions.

How to request a CSR in Plesk
1. Log into your Plesk control panel.
2. Click on “Domains” on the left hand side.
3. Find the domain in the list you wish to generate the CSR for, then click on “Manage Hosting” on the right hand side of the domain.
4. Click on “Websites & Domains” at the top of the page.
5. Click on your domain name that you are trying to generate the CSR for, it will be in larger bolder letters with “Hosting Settings” right next to it, however you will want to click on the domain name itself.
6. Click on “Secure Your Sites”.
7. Click the “Add SSL Certificate” button.
8. Once there, fill out the information for “Certificate Name” (typically this is the name of the domain and the year/month the SSL has been issued, so that it makes it easier to figure out when the SSL will need to be renewed and how long it is good for.) and under “Settings” (This would be your company’s information).
9. Once that is all filled out, click the “Request” button, this will generate the CSR for that specific domain.
10. After the CSR is generated it should appear underneath the Certificate text boxes, you will copy all of this and provide it to the SSL provider.

How to install a SSL Certificate in Plesk.
1. Log into your Plesk control panel.
2. Click on “Domains” on the left hand side.
3. Find the domain in the list you wish to install the SSL for, then click “Manage Hosting” on the right hand side of the domain.
4. Click on “Websites & Domains” at the top of the page.
5. Click on your domain name that you are trying to generate the CSR for, it will be in larger bolder letters with “Hosting Settings” right next to it, however you will want to click on the domain name itself.
6. Click on “Secure Your Sites”.
7. Find the SSL in the list that you added to generate the CSR so that you could get the SSL.
8. Once there, either upload the certificate documents that the SSL provider has given you, or if you have the text you can copy and paste them into the text boxes. After doing so you would either click “Send Text” or “Send File”.
9. Next you will need to ensure the new SSL is active for the domain, you will go back to “Websites & Domains” tab that you were on previously, and click the “Hosting Settings” that were next to the domain.
10. On this page towards the middle of it you should see a “Security” section, if the “SSL Support” box is not checked you will need to check here, and then from the certificate dropdown menu you would select the new SSL that you uploaded and then select “OK” at the bottom of the page.

Debugging sleeping connections with MySQL

Have you ever seen connection in the SHOW PROCESSLIST output which is in “Sleep” state for a long time and you have no idea why this would happen ?

I see if frequently with web applications and it is often indication of trouble. Not only it means you may run out of MySQL connections quicker than you expected but it also frequently indicates serious problems in the application. If you do not use persistent connections and you have connection in Sleep stage for 600 seconds what could it be ? It may mean some of your pages take that long to generate (or might be the code simply gets into the tight loop and page never gets generated) it also could mean some of external Web Services are slow or not available and you’re not dealing with timeouts properly. Or may be you have several connections to MySQL server and right now running query which takes that long ? In any case it is something frequently worth looking at.

First task is to find to which process the connection belongs. Using different user names for different application is a good practice however it will not tell you which of apache children is handling request in question. If you just want to fix it, ie by restarting apache it is enough but if you want to figure our why it is happening you need more info.

You my notice in the “Host” filed of SHOW PROCESSLIST output not only host but also port is specified, showing you something like “192.168.1.70:58555″ This port can be used to identify the process which owns connection in question:

[root@w1 ~]# netstat -ntp | grep :45384
tcp        0      0 192.168.1.70:45384          192.168.1.82:3306           ESTABLISHED 28540/php-cgi

As you can see in this case we can find php-cgi is holding connection in question (this is lighttpd based system with fastcgi)

Now you know the process and you can use your favorite tools to check what that process is doing.

[root@w1 ~]# netstat -ntp | grep 28540
tcp        0      0 192.168.1.70:58555          192.168.1.90:11211          ESTABLISHED 28540/php-cgi
tcp        0      0 192.168.1.70:52711          192.168.1.88:8080           ESTABLISHED 28540/php-cgi
tcp        0      0 192.168.1.70:45384          192.168.1.82:3306           ESTABLISHED 28540/php-cgi
tcp        0      0 192.168.1.70:45399          192.168.1.82:3306           ESTABLISHED 28540/php-cgi
tcp        0      0 192.168.1.70:45407          192.168.1.82:3306           ESTABLISHED 28540/php-cgi
tcp        0      0 192.168.1.70:45408          192.168.1.82:3306           ESTABLISHED 28540/php-cgi
tcp        0      0 192.168.1.70:35556          192.168.1.92:11211          ESTABLISHED 28540/php-cgi

Using same netstat command and filtering on the PID we can find which connections does this process have. Here you can see it has couple of memcached connections. Few MySQL connections (to the same host, which if usually bad idea) and connection to some external web server.

You can use strace -p to see what host is doing, it often gives a clue. In this case I for example found the process is stuck in pool() system call reading from network. Using netstat can give you an idea what it can be but if you do not like guessing you can use gdb -p . It will not print you exact line of code in PHP which is running but can give you some good ideas – for example in this case I could find stack trace originated from php stream functions not from libmysql or memcache.so, which means it is not MySQL or memcache connections leaving last candidate as the only choice. I also could see some of the variables in GDB “bt” command output which also hinted what could be the problem.

By the way does anyone know any debugger which can connect to PHP process or apache with mod_php and provide backtrace in PHP terms not the one for zend engine ? That would be pretty cool.

Yet another great tool which you can use is server-status if you’re running apache. This way you will see the URL which that process is processing and so get few more hints on what may be happening or even get repeatable example in some cases.

The tools I mentioned regarding figuring our what is happening with the process are not only helpful to debug sleeping connections with MySQL but many other cases when you see web application locking up or starting to runs in the tight loop consuming too much CPU time.

If you know any other tools which could be helpful in this regard would appreciate your comments. There might be some smarter tools out where for production tracing.

ClamAV on CentOS

ClamAV is a free anti-virus program available for Linux operating systems.This will explain how to Install ClamAV on CentOS 6 64.

Install the epel repository

First, determine the most current version of the repository that is available. Using a web browser, visit http://download.fedoraproject.org/pub/epel/6/x86_64/

Note you can substitute the CentOS version ( /6/ ) with your current version.

Scroll down the page until you find epel-release-v-r.noarch-rpm, substituting v for your CentOS version and r will be the current repository version. For this example, the current version listed is epel-release-6-8.noarch-rpm .

Log into your server as root and run the following command using the correct repository version you discovered in the previous step

CentOS 6.x

rpm -Uvh http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

CentOS 5.x

rpm -Uvh http://mirror.pnl.gov/epel/5/x86_64/epel-release-5-4.noarch.rpm

Enable Epel Repo – Set enabled=1:

nano /etc/yum.d/epel.repo
[epel]
name=Extra Packages for Enterprise Linux 6 - $basearch
#baseurl=http://download.fedoraproject.org/pub/epel/6/$basearch
mirrorlist=https://mirrors.fedoraproject.org/metalink?repo=epel-6&arch=$basearch
failovermethod=priority
enabled=0
gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-EPEL-6

Install clamav


# yum -y install clamav clamd

Set clamav to start on reboot

# chkconfig clamd on

Update the clamav virus database

# /usr/bin/freshclam

Error:

# /usr/bin/freshclam
ERROR: Please edit the example config file /etc/freshclam.conf
ERROR: Can't open/parse the config file /etc/freshclam.conf

Comment out the Line with “Example”


# nano /etc/freshclam.conf
##
## Example config file for freshclam
## Please read the freshclam.conf(5) manual before editing this file.
##
# Comment or remove the line below.
Example

Change to


# nano /etc/freshclam.conf
##
## Example config file for freshclam
## Please read the freshclam.conf(5) manual before editing this file.
##
# Comment or remove the line below.
# Example

Run freshclam again

# /usr/bin/freshclam

Start Clamav

# service clamd start

# service clamd start
Starting Clam AntiVirus Daemon: ERROR: Please edit the example config file /etc/clamd.conf
ERROR: Can't open/parse the config file /etc/clamd.conf
 [FAILED]

Edit the config file, comment out “Example”

##
## Example config file for the Clam AV daemon
## Please read the clamd.conf(5) manual before editing this file.
##
# Comment or remove the line below.
#Example

Set Clamav to run a daily scan

# nano /etc/cron.daily/clamscan
#!/bin/bash
# setup the scan location and scan log
CLAM_SCAN_DIR="/var/www/vhosts"
CLAM_LOG_FILE="/var/log/clamav/dailyscan.log"
# update the virus database
/usr/bin/freshclam
# run the scan
/usr/bin/clamscan -i -r $CLAM_SCAN_DIR >> $CLAM_LOG_FILE
MAILTO=user@domain.com

or

clamscan -i -r --log=/var/log/clamscan-date.txt /var/www/vhosts/*

Set the cron file as an executible

chmod 555 /etc/cron.daily/clamscan

Test your installation and cron job

/etc/cron.daily/clamscan