Find Exim Spam

How to locate the top scripts on your server that send out email. You can then search the Exim mail log for those scripts to determine if it looks like spam, and even check your Apache access logs in order to find how a spammer might be using your scripts to send out spam. Login to your server via SSH as the root user. Run the following command to pull the most used mailing script’s location from the Exim mail log:

grep cwd /var/log/exim_mainlog | grep -v /var/spool | awk -F"cwd=" '{print $2}' | awk '{print $1}' | sort | uniq -c | sort -n

Code breakdown:

grep cwd /var/log/exim_mainlog 	

Use the grep command to locate mentions of cwd from the Exim mail log. This stands for current working directory.

grep -v /var/spool

Use the grep with the -v flag which is an invert match, so we don’t show any lines that start with /var/spool as these are normal Exim deliveries not sent in from a script.

   
awk -F"cwd=" '{print $2}' | awk '{print $1}' 	

Use the awk command with the -Field seperator set to cwd=, then just print out the $2nd set of data, finally pipe that to the awk command again only printing out the $1st column so that we only get back the script path.

sort | uniq -c | sort -n 	

Sort the script paths by their name, uniquely count them, then sort them again numerically from lowest to highest.

You should see something like this:


    15 /home/username/public_html/about-us
    25 /home/username/public_html
    7866 /home/username/public_html/data

Here we can see that the /home/userna5/public_html/data directory by far has more deliveries coming in than any others.Now we can run the following command to see what scripts are located in that directory:

ls -lahtr /username/public_html/data

In thise case we got back:

    drwxr-xr-x 17 username username 4.0K Jan 20 10:25 ../
    -rw-r--r-- 1 username username 5.6K Jan 20 11:27 mailer.php
    drwxr-xr-x 2 username username 4.0K Jan 20 11:27 ./

So we can see there is a script called mailer.php in this directory. Knowing the mailer.php script was sending mail into Exim, we can now take a look at our Apache access log to see what IP addresses are accessing this script using the following command:

grep "mailer.php" /home/username/access-logs/example.com | awk '{print $1}' | sort -n | uniq -c | sort -n

You should get back something similar to this:


    2 123.123.123.126
    2 123.123.123.125
    2 123.123.123.124
    7860 123.123.123.123

So we can clearly see that the IP address 123.123.123.123 was responsible for using our mailer script in a malicious nature. If you did find a malicious IP address sending out a large volume of messages from a script on your server you’ll probably want to go ahead and block them at your server’s firewall so that they can’t try to connect again.

Remove exim emails:

# exim -bp | exiqgrep -i | xargs exim -Mrm

Rkhunter Emails

Getting rkhunter failed emails in your email? Here is how to configure the email to send to a correct address.

Edit /etc/sysconfig/rkhunter:


nano /etc/sysconfig/rkhunter

# System configuration file for Rootkit Hunter which
# stores RPM system specifics for cron run, etc.
#
#    MAILTO= <email address to send scan report>
# DIAG_SCAN= no  - perform  normal  report scan
#            yes - perform detailed report scan
#                  (includes application check)

MAILTO=root@localhost
DIAG_SCAN=no

Change the email to your email.

# System configuration file for Rootkit Hunter which
# stores RPM system specifics for cron run, etc.
#
#    MAILTO= <email address to send scan report>
# DIAG_SCAN= no  - perform  normal  report scan
#            yes - perform detailed report scan
#                  (includes application check)

MAILTO=admin@yourdomain.com
DIAG_SCAN=no

Save and you are all set!

RCA

Root Cause Analysis

Root cause analysis (RCA) is a method of problem solving that tries to identify the root causes of faults or problems.

RCA practice tries to solve problems by attempting to identify and correct the root causes of events, as opposed to simply addressing their symptoms. Focusing correction on root causes has the goal of preventing problem recurrence. RCFA (Root Cause Failure Analysis) recognizes that complete prevention of recurrence by one corrective action is not always possible.

http://en.wikipedia.org/wiki/Root_cause_analysis

Hotmail and Outlook Black List

If you cannot send emails to Outlook or Hotmail or MSN, then your server’s IP address maybe blacklisted. Here are some tips to get removed from the MSN blacklist.

Before jumping through the blacklist removal hoops, you may want to double-check that your emails are not simply going into the spam folder. This process will not help you with emails being dropped into the spam folder. This is for getting off of MSN’s blacklist. I am going to outline 3 steps.

Verify you are on the MSN blacklist.
Perform preliminary blacklist removal checks.
Submit MSN blacklist delisting request.

Delist Here – Sender Information for Outlook.com Delivery –

https://support.microsoft.com/en-us/getsupport?oaspworkflow=start_1.0.0.0&wfname=capsub&productkey=edfsmsbl3&locale=en-us&ccsid=635808707851479494&wa=wsignin1.0

MSN Blacklist Check

If MSN has blacklisted your IP, you will receive a delivery rejection notice from MSN or Hotmail. If you check your server’s logs or your email bounce you may see something like this:

SMTP error from remote mail server after end of data:
host mx1.hotmail.com [65.54.188.94]: 550 SC-001 (BAY0-MC2-F59) Unfortunately, messages from 216.55.xxx.xxx weren't sent. Please contact your Internet service provider since part of their network is on our block list. You can also refer your provider to <a href="http://mail.live.com/mail/troubleshooting.aspx#errors." target="_blank" rel="noopener">http://mail.live.com/mail/troubleshooting.aspx#errors.</a>

If you are seeing this or a similar email error, then your server’s IP has likely been blocked by MSN/Hotmail. There could be other response codes, but typically all MSN blacklist notifications will include a 500 series error. MSN’s postmaster service as a list of MSN’s blacklist codes.

MSN Blacklist Codes

I suggest you check this list to find the exact reason Hotmail or MSN is rejecting your emails.

There are some 400 series errors that deal with email volume rather than suspected spam. If you are sending high volumes of email to MSN, you may need to sign up for their bulk sender’s program.

If you are not seeing 500 errors, then you may not have an email blacklist problem but some other email delivery issue.
Preliminary Blacklist Delisting Tasks

Before requesting removal from MSN’s blacklist, you will want to take some steps to stop whatever caused the listing.

Make sure there is no unauthorized email going from your server.

  • Check the daily volume of email going to Hotmail, MSN or Outlook
  • Look for compromised user accounts.
  • Look for people forwarding email to Hotmail, MSN, or Outlook.com.
  • Do you have SPF and rDNS records set up?

If someone is forwarding email to Hotmail related addresses and then marketing it as spam, Hotmail will lower your server’s sender reputation. Window’s Live and related email services such as Hotmail and MSN.com emails work with Return Path to filter email. So email server reputation is more important for sending to these accounts than some of the other ISP’s covered in this series.

Hotmail/MSN Blacklist Removal Process

To start the process of getting removed from Hotmail’s blacklist, you will need to complete their sender information form.

Unfortunately since Microsoft maintains their own blacklist they have no obligation to accept email from anyone. Please have a look at some of their suggestions located at https://mail.live.com/mail/services.aspx

I would suggest signing up for both SNDS and Microsoft’s Junk Mail Reporting Program.

Submit to get Delisted!

Sender Information for Outlook.com Delivery – https://support.microsoft.com/en-us/getsupport?oaspworkflow=start_1.0.0.0&wfname=capsub&productkey=edfsmsbl3&ccsid=636529520240187401&wa=wsignin1.0

Provide all of the requested information. Unlike some other ISPs, MSN Support requires you to run some telnet tests from the command line on your server. If you do not know how to run these tests, you will need to get someone to help you.

In working with MSN, I have found it very important to provide accurate email headers. If you provide reliable information and are truly not spamming their systems, you will typically see removal in 2-3 business days. MSN is very picky about DNS. So be sure your DNS, PTR and SPF/SenderID records are in order before requesting removal.

If you have root access and need to send email now, try below:

Partial Solution:

Re-route the IP on port 25 if you have a linux box and you have another IP that is not blacklisted.

apply an iptables rule to route your outbound SMTP to a new IP

216.55.xxx.xxx


# iptables -t nat -A POSTROUTING -p tcp --dport 25 -j SNAT --to-source 216.55.xxx.xxx

Managing your Exim Server

Here are some useful things to know for managing an Exim 4 server. This assumes a prior working knowledge of SMTP, MTAs, and a UNIX shell prompt.

Message-IDs and spool files

The message-IDs that Exim uses to refer to messages in its queue are mixed-case alpha-numeric, and take the form of: XXXXXX-YYYYYY-ZZ. Most commands related to managing the queue and logging use these message-ids.

There are three — count ’em, THREE — files for each message in the spool directory. If you’re dealing with these files by hand, instead of using the appropriate exim commands as detailed below, make sure you get them all, and don’t leave Exim with remnants of messages in the queue. I used to mess directly with these files when I first started running Exim machines, but thanks to the utilities described below, I haven’t needed to do that in many months.

Files in /var/spool/exim/msglog contain logging information for each message and are named the same as the message-id.

Files in /var/spool/exim/input are named after the message-id, plus a suffix denoting whether it is the envelope header (-H) or message data (-D).

These directories may contain further hashed sub directories to deal with larger mail queues, so don’t expect everything to always appear directly in the top /var/spool/exim/input or /var/spool/exim/msglog directories; any searches or greps will need to be recursive. See if there is a proper way to do what you’re doing before working directly on the spool files.
Basic information

Print a count of the messages in the queue:

root@localhost# exim -bpc

Print a listing of the messages in the queue (time queued, size, message-id, sender, recipient):


root@localhost# exim -bp

Print a summary of messages in the queue (count, volume, oldest, newest, domain, and totals):


root@localhost# exim -bp | exiqsumm

Print what Exim is doing right now:


root@localhost# exiwhat

Test how exim will route a given address:


root@localhost# exim -bt alias@localdomain.com
user@thishost.com
router = localuser, transport = local_delivery
root@localhost# exim -bt user@thishost.com
user@thishost.com
router = localuser, transport = local_delivery
root@localhost# exim -bt user@remotehost.com
router = lookuphost, transport = remote_smtp
host mail.remotehost.com [1.2.3.4] MX=0

Run a pretend SMTP transaction from the command line, as if it were coming from the given IP address. This will display Exim’s checks, ACLs, and filters as they are applied. The message will NOT actually be delivered.


root@localhost# exim -bh 192.168.11.22

Display all of Exim’s configuration settings:


root@localhost# exim -bP

Searching the queue with exiqgrep

Exim includes a utility that is quite nice for grepping through the queue, called exiqgrep. Learn it. Know it. Live it. If you’re not using this, and if you’re not familiar with the various flags it uses, you’re probably doing things the hard way, like piping `exim -bp` into awk, grep, cut, or `wc -l`. Don’t make life harder than it already is.

First, various flags that control what messages are matched. These can be combined to come up with a very particular search.

Use -f to search the queue for messages from a specific sender:


root@localhost# exiqgrep -f [luser]@domain

Use -r to search the queue for messages for a specific recipient/domain:


root@localhost# exiqgrep -r [luser]@domain

Use -o to print messages older than the specified number of seconds. For example, messages older than 1 day:


root@localhost# exiqgrep -o 86400 [...]

Use -y to print messages that are younger than the specified number of seconds. For example, messages less than an hour old:


root@localhost# exiqgrep -y 3600 [...]

Use -s to match the size of a message with a regex. For example, 700-799 bytes:


root@localhost# exiqgrep -s '^7..$' [...]

Use -z to match only frozen messages, or -x to match only unfrozen messages.

There are also a few flags that control the display of the output.

Use -i to print just the message-id as a result of one of the above two searches:


root@localhost# exiqgrep -i [ -r | -f ] ...

Use -c to print a count of messages matching one of the above searches:


root@localhost# exiqgrep -c ...

Print just the message-id of the entire queue:


root@localhost# exiqgrep -i

Managing the queue

The main exim binary (/usr/sbin/exim) is used with various flags to make things happen to messages in the queue. Most of these require one or more message-IDs to be specified in the command line, which is where `exiqgrep -i` as described above really comes in handy.

Start a queue run:


root@localhost# exim -q -v

Start a queue run for just local deliveries:


root@localhost# exim -ql -v

Remove a message from the queue:


root@localhost# exim -Mrm [ ... ]

Freeze a message:


root@localhost# exim -Mf [ ... ]

Thaw a message:


root@localhost# exim -Mt [ ... ]

Deliver a message, whether it’s frozen or not, whether the retry time has been reached or not:


root@localhost# exim -M [ ... ]

Deliver a message, but only if the retry time has been reached:


root@localhost# exim -Mc [ ... ]

Force a message to fail and bounce as “cancelled by administrator”:


root@localhost# exim -Mg [ ... ]

Remove all frozen messages:


root@localhost# exiqgrep -z -i | xargs exim -Mrm

To remove all messages from the queue, enter:


# exim -bp | awk '/^ *[0-9]+[mhd]/{print "exim -Mrm " $3}' | bash

Or


# exim -bp | exiqgrep -i | xargs exim -Mrm

Remove all messages older than five days (86400 * 5 = 432000 seconds):


root@localhost# exiqgrep -o 432000 -i | xargs exim -Mrm

Freeze all queued mail from a given sender:


root@localhost# exiqgrep -i -f luser@example.tld | xargs exim -Mf

View a message’s headers:


root@localhost# exim -Mvh

View a message’s body:


root@localhost# exim -Mvb

View a message’s logs:


root@localhost# exim -Mvl

Add a recipient to a message:


root@localhost# exim -Mar <message-id> <address> [ <address> ... ]

Edit the sender of a message:


root@localhost# exim -Mes <address>

Access control

Exim allows you to apply access control lists at various points of the SMTP transaction by specifying an ACL to use and defining its conditions in exim.conf. You could start with the HELO string.


# Specify the ACL to use after HELO
acl_smtp_helo = check_helo

# Conditions for the check_helo ACL:
check_helo:

deny message = Gave HELO/EHLO as "friend"
log_message = HELO/EHLO friend
condition = ${if eq {$sender_helo_name}{friend} {yes}{no}}

deny message = Gave HELO/EHLO as our IP address
log_message = HELO/EHLO our IP address
condition = ${if eq {$sender_helo_name}{$interface_address} {yes}{no}}

accept

NOTE: Pursue HELO checking at your own peril. The HELO is fairly unimportant in the grand scheme of SMTP these days, so don’t put too much faith in whatever it contains. Some spam might seem to use a telltale HELO string, but you might be surprised at how many legitimate messages start off with a questionable HELO as well. Anyway, it’s just as easy for a spammer to send a proper HELO than it is to send HELO im.a.spammer, so consider yourself lucky if you’re able to stop much spam this way.

Next, you can perform a check on the sender address or remote host. This shows how to do that after the RCPT TO command; if you reject here, as opposed to rejecting after the MAIL FROM, you’ll have better data to log, such as who the message was intended for.

# Specify the ACL to use after RCPT TO
acl_smtp_rcpt = check_recipient

# Conditions for the check_recipient ACL
check_recipient:

# […]

drop hosts = /etc/exim_reject_hosts
drop senders = /etc/exim_reject_senders

# [ Probably a whole lot more… ]

This example uses two plain text files as blacklists. Add appropriate entries to these files – hostnames/IP addresses to /etc/exim_reject_hosts, addresses to /etc/exim_reject_senders, one entry per line.

It is also possible to perform content scanning using a regex against the body of a message, though obviously this can cause Exim to use more CPU than it otherwise would need to, especially on large messages.

# Specify the ACL to use after DATA
acl_smtp_data = check_message

# Conditions for the check_messages ACL
check_message:

deny message = “Sorry, Charlie: $regex_match_string”
regex = ^Subject:: .*Lower your self-esteem by becoming a sysadmin

accept

Fix SMTP-Auth for Pine

If pine can’t use SMTP authentication on an Exim host and just returns an “unable to authenticate” message without even asking for a password, add the following line to exim.conf:

begin authenticators

fixed_plain:
driver = plaintext
public_name = PLAIN
server_condition = “${perl{checkuserpass}{$1}{$2}{$3}}”
server_set_id = $2
> server_prompts = :

This was a problem on CPanel Exim builds awhile ago, but they seem to have added this line to their current stock configuration.
Log the subject line

This is one of the most useful configuration tweaks I’ve ever found for Exim. Add this to exim.conf, and you can log the subject lines of messages that pass through your server. This is great for troubleshooting, and for getting a very rough idea of what messages may be spam.

log_selector = +subject

Reducing or increasing what is logged.
Disable identd lookups

Frankly, I don’t think identd has been useful for a long time, if ever. Identd relies on the connecting host to confirm the identity (system UID) of the remote user who owns the process that is making the network connection. This may be of some use in the world of shell accounts and IRC users, but it really has no place on a high-volume SMTP server, where the UID is often simply “mail” or whatever the remote MTA runs as, which is useless to know. It’s overhead, and results in nothing but delays while the identd query is refused or times out. You can stop your Exim server from making these queries by setting the timeout to zero seconds in exim.conf:

rfc1413_query_timeout = 0s

Disable Attachment Blocking

To disable the executable-attachment blocking that many Cpanel servers do by default but don’t provide any controls for on a per-domain basis, add the following block to the beginning of the /etc/antivirus.exim file:

if $header_to: matches “example.com|example2.com”
then
finish
endif

It is probably possible to use a separate file to list these domains, but I haven’t had to do this enough times to warrant setting such a thing up.
Searching the logs with exigrep

The exigrep utility (not to be confused with exiqgrep) is used to search an exim log for a string or pattern. It will print all log entries with the same internal message-id as those that matched the pattern, which is very handy since any message will take up at least three lines in the log. exigrep will search the entire content of a log entry, not just particular fields.

One can search for messages sent from a particular IP address:

root@localhost# exigrep ‘<= .* [12.34.56.78] ‘ /path/to/exim_log Search for messages sent to a particular IP address: root@localhost# exigrep ‘=> .* [12.34.56.78]’ /path/to/exim_log

This example searches for outgoing messages, which have the “=>” symbol, sent to “user@domain.tld”. The pipe to grep for the “<=” symbol will match only the lines with information on the sender – the From address, the sender’s IP address, the message size, the message ID, and the subject line if you have enabled logging the subject. The purpose of doing such a search is that the desired information is not on the same log line as the string being searched for. root@localhost# exigrep ‘=> .*user@domain.tld’ /path/to/exim_log | fgrep ‘<=’

Generate and display Exim stats from a logfile:

root@localhost# eximstats /path/to/exim_mainlog

Same as above, with less verbose output:

root@localhost# eximstats -ne -nr -nt /path/to/exim_mainlog

Same as above, for one particular day:

root@localhost# fgrep YYYY-MM-DD /path/to/exim_mainlog | eximstats

Bonus!

To delete all queued messages containing a certain string in the body:

root@localhost# grep -lr ‘a certain string’ /var/spool/exim/input/ |
sed -e ‘s/^.*/([a-zA-Z0-9-]*)-[DH]$/1/g’ | xargs exim -Mrm

Note that the above only delves into /var/spool/exim in order to grep for queue files with the given string, and that’s just because exiqgrep doesn’t have a feature to grep the actual bodies of messages. If you are deleting these files directly, YOU ARE DOING IT WRONG! Use the appropriate exim command to properly deal with the queue.

If you have to feed many, many message-ids (such as the output of an `exiqgrep -i` command that returns a lot of matches) to an exim command, you may exhaust the limit of your shell’s command line arguments. In that case, pipe the listing of message-ids into xargs to run only a limited number of them at once. For example, to remove thousands of messages sent from joe@example.com:

root@localhost# exiqgrep -i -f ‘<joe@example.com>’ | xargs exim -Mrm

Speaking of “DOING IT WRONG” — Attention, CPanel forum readers

I get a number of hits to this page from a link in this post at the CPanel forums. The question is:

Due to spamming, spoofing from fields, etc., etc., etc., I am finding it necessary to spend more time to clear the exim queue from time to time. […] what command would I use to delete the queue

The answer is: Just turn exim off, because your customers are better off knowing that email simply isn’t running on your server, than having their queued messages deleted without notice.

Or, figure out what is happening. The examples given in that post pay no regard to the legitimacy of any message, they simply delete everything, making the presumption that if a message is in the queue, it’s junk. That is total fallacy. There are a number of reasons legitimate mail can end up in the queue. Maybe your backups or CPanel’s “upcp” process are running, and your load average is high — exim goes into a queue-only mode at a certain threshold, where it stops trying to deliver messages as they come in and just queues them until the load goes back down. Or, maybe it’s an outgoing message, and the DNS lookup failed, or the connection to the domain’s MX failed, or maybe the remote MX is busy or greylisting you with a 4xx deferral. These are all temporary failures, not permanent ones, and the whole point of having temporary failures in SMTP and a mail queue in your MTA is to be able to try again after awhile.

Exim already purges messages from the queue after the period of time specified in exim.conf. If you have this value set appropriately, there is absolutely no point in removing everything from your queue every day with a cron job. You will lose legitimate mail, and the sender and recipient will never know if or why it happened. Do not do this!

If you regularly have a large number of messages in your queue, find out why they are there. If they are outbound messages, see who is sending them, where they’re addressed to, and why they aren’t getting there. If they are inbound messages, find out why they aren’t getting delivered to your user’s account. If you need to delete some, use exiqgrep to pick out just the ones that should be deleted.
Reload the configuration

After making changes to exim.conf, you need to give the main exim pid a SIGHUP to re-exec it and have the configuration re-read. Sure, you could stop and start the service, but that’s overkill and causes a few seconds of unnecessary downtime. Just do this:

root@localhost# kill -HUP `cat /var/spool/exim/exim-daemon.pid`

You should then see something resembling the following in exim_mainlog:

pid 1079: SIGHUP received: re-exec daemon
exim 4.52 daemon started: pid=1079, -q1h, listening for SMTP on port 25 (IPv4)

Plesk Firewall

psa-firewall

To start the service through the command line:

/etc/init.d/psa-firewall start

To stop the service through the command line:

/etc/init.d/psa-firewall stop

To restart the service through the command line:

/etc/init.d/psa-firewall restart

Configuration files are accessible at:

  • /usr/local/psa/var/modules/firewall/firewall-active.sh
  • /usr/local/psa/var/modules/firewall/firewall-emergency.sh
  • /usr/local/psa/var/modules/firewall/firewall-new.sh

psa-firewall (IP forwarding)

To start the service through the command line:

/etc/init.d/psa-firewall-forward start

To stop the service through the command line:

/etc/init.d/psa-firewall-forward stop

To restart the service through the command line:

/etc/init.d/psa-firewall-forward restart

Configuration files are accessible at:

  • /usr/local/psa/var/modules/firewall/ip_forward.active
  • /usr/local/psa/var/modules/firewall/ip_forward.saved

How to Install Telnet Client on CentOS 6.3

Telnet is most likely to be used by system administrators, program developers and anyone who has a need to use specific applications or data located at a particular host computer. It’s a network protocol used on the Internet or local area networks to provide a bidirectional interactive text-oriented communication facility using a virtual terminal connection. Telnet does not installed automatically on CentOS 6.3. You can issue the following command to confirm that telnet is working :

telnet localhost 80

If telnet client does not installed, it should return something like :

[root@centos63 ~]# telnet localhost 80
-bash: telnet: command not found

Issue the following command to install telnet client :

[root@centos63 ~]# yum install telnet -y
Loaded plugins: fastestmirror, presto
Determining fastest mirrors
 * base: mirrors.hostemo.com
 * extras: mirrors.hostemo.com
 * updates: mirrors.hostemo.com
CentOS6.3-Repository                                                         | 4.0 kB     00:00 ...
base                                                                         | 3.7 kB     00:00
extras                                                                       | 3.0 kB     00:00
updates                                                                      | 3.5 kB     00:00
updates/primary_db                                                           | 2.2 MB     00:19
Setting up Install Process
Resolving Dependencies
--> Running transaction check
---> Package telnet.i686 1:0.17-47.el6 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

====================================================================================================
 Package           Arch            Version                    Repository                       Size
====================================================================================================
Installing:
 telnet            i686            1:0.17-47.el6              CentOS6.3-Repository             56 k

Transaction Summary
====================================================================================================
Install       1 Package(s)

Total download size: 56 k
Installed size: 102 k
Downloading Packages:
Setting up and reading Presto delta metadata
Processing delta metadata
Package(s) data still to download: 56 k
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : 1:telnet-0.17-47.el6.i686                                                        1/1
  Verifying  : 1:telnet-0.17-47.el6.i686                                                        1/1

Installed:
  telnet.i686 1:0.17-47.el6

Complete!

If telnet client installed, it should return something like :

[root@centos63 ~]# telnet localhost 80
Trying ::1...
Connected to localhost.
Escape character is '^]'.