6.3. Amavisd-new

Edit /etc/amavis/conf.d/15-content_filter_mode and ucomment antivirus and spam checking

use strict;

# You can modify this file to re-enable SPAM checking through spamassassin
# and to re-enable antivirus checking.

# Default antivirus checking mode
# Uncomment the two lines below to enable it back

@bypass_virus_checks_maps = (
   \%bypass_virus_checks, \@bypass_virus_checks_acl, \$bypass_virus_checks_re);

# Default SPAM checking mode
# Uncomment the two lines below to enable it back

@bypass_spam_checks_maps = (
   \%bypass_spam_checks, \@bypass_spam_checks_acl, \$bypass_spam_checks_re);

1;  # ensure a defined return

To integrate Amavis with Postfix we need to add a content_filter in /etc/postfix/main.cf.

content_filter = smtp-amavis:[]:10024

Open /etc/postfix/master.cf and add this at the end of the file:

smtp-amavis     unix    -       -       -       -       2       smtp
        -o smtp_data_done_timeout=1200
        -o smtp_send_xforward_command=yes
        -o disable_dns_lookups=yes
        -o max_use=20 inet    n       -       -       -       -       smtpd
        -o content_filter=
        -o local_recipient_maps=
        -o relay_recipient_maps=
        -o smtpd_restriction_classes=
        -o smtpd_delay_reject=no
        -o smtpd_client_restrictions=permit_mynetworks,reject
        -o smtpd_helo_restrictions=
        -o smtpd_sender_restrictions=
        -o smtpd_recipient_restrictions=permit_mynetworks,reject
        -o smtpd_data_restrictions=reject_unauth_pipelining
        -o smtpd_end_of_data_restrictions=
        -o mynetworks=
        -o smtpd_error_sleep_time=0
        -o smtpd_soft_error_limit=1001
        -o smtpd_hard_error_limit=1000
        -o smtpd_client_connection_count_limit=0
        -o smtpd_client_connection_rate_limit=0
        -o receive_override_options=no_header_body_checks,no_unknown_recipient_checks

and add this immediately below the “pickup” transport service:

         -o content_filter=
         -o receive_override_options=no_header_body_checks

This will prevent messages that are generated to report on spam from being classified as spam.

Amavisd-new listens on port 1024 where Postfix sends all of the mail coming into the server. After processing the email amavisd-new returns the mail to Postfix on port 1025 for final delivery.

Edit /etc/amavis/conf.d/50-user and paste this into the file:

use strict;

@local_domains_acl = ( ".$mydomain" );

$sa_spam_subject_tag = 'SPAM > ';
$sa_tag_level_deflt  = -999;  # add spam info headers if at, or above that level
$sa_tag2_level_deflt = 5; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 12; # triggers spam evasive actions

$final_virus_destiny      = D_DISCARD;  # (data not lost, see virus quarantine)
$final_banned_destiny     = D_REJECT;   # D_REJECT when front-end MTA
$final_spam_destiny       = D_DISCARD;
$final_bad_header_destiny = D_PASS;     # False-positive prone (for spam)

#------------ Do not modify anything below this line -------------
1;  # ensure a defined return

Of course, you do not just want to paste stuff into configuration files without knowing what are you actually doing. So let us go through the file, line by line:

@local_domains_acl = ( ".$mydomain" );

This is a list of domains for which this server considers itself as a final destination. If you do not add all your domains here, they will not be processed by amavis, so if you have more domains on your server, except for the default line, this should look something like this:

@local_domains_acl = ( ".$mydomain", "domain2.com", "domain3.org" );

Later, we will see how we can automate this to read the information from the database.

$sa_spam_subject_tag = 'SPAM > ';
$sa_tag_level_deflt  = -999;  # add spam info headers if at, or above that level
$sa_tag2_level_deflt = 5; # add 'spam detected' headers at that level
$sa_kill_level_deflt = 12; # triggers spam evasive actions

When SpamAssassin processes message, it gives them a score identifying what is the probability of message being a spam. $sa_tag_level_deflt tells amavis that ig the score is greater or equal to this level that amavis should append X-Spam headers to the message. The score of -999 means that we want to apply X-Spam header to all of our messages, so it is easier for us to see why is, or why is not, a message considered spam by examining the headers of the message.

$sa_tag2_level_deflt = 5; is the spam level the message needs to reach for amavis to apply the header X-Spam-Flag: YES so we (our mail client) knows that this message is considered to be spam. At this level the message subject is prefixed with the value of the variable $sa_spam_subject_tag. In this case the subject is prefixed with “SPAM > ”.

$sa_kill_level_deflt holds the value of the spam level that message needs to reach to do something with the message. What we are going to do with messages that are spam depends of the value of $final_spam_destiny variable. In this case we are discarding this messages.

This means that the message is not going to reach to recipients mailbox at all. But that does not mean that the message is lost. Spam messages that are discarded can still be fetched from the system. Default configuration is to store them in the /var/lib/amavis/virusmails folder. It is possible to store them in the database.

For releasing quarantined mail you need to add these lines:

$inet_socket_port = [10024,9998];
$interface_policy{'9998'} = 'AM.PDP-INET';
$policy_bank{'AM.PDP-INET'} = {
  protocol => 'AM.PDP',  # select Amavis policy delegation protocol
  inet_acl => [qw( [::1] )],  # restrict access to these IP addresses
# auth_required_release => 0,  # don't require secret_id for amavisd-release

If you want to store everything in the database you have to add these lines as well:

@lookup_sql_dsn =  ( ['DBI:mysql:database=DATABASE-NAME;host=;port=3306', 'USERNAME', 'PASSWORD']);
@storage_sql_dsn = @lookup_sql_dsn;

$virus_quarantine_method = 'sql:';
$spam_quarantine_method = 'sql:';
$banned_files_quarantine_method = 'sql:';
$bad_header_quarantine_method = 'sql:';

For this to work, you will have to create the database in MySQL and change the value of @lookup_sql_dsn variable to match your database, username & password. You will have to create the tables for this database by looking at the amavis documentation). When you set this all up, you can use Mailzu or Postvis Admin to retrieve the quarantined messages. For users of SquirrelMail webmail system, there as a plugin called AmavisNewSQL which enables access for users to quarantined messages that were addressed to them.


Quarantined messages (on the file system, or in the database) have to be deleted periodically!

If you do not want to quarantine any of the messages, change

$virus_quarantine_method = 'sql:';
$spam_quarantine_method = 'sql:';
$banned_files_quarantine_method = 'sql:';
$bad_header_quarantine_method = 'sql:';


$virus_quarantine_method = undef;
$spam_quarantine_method = undef;
$banned_files_quarantine_method = undef;
$bad_header_quarantine_method = undef;

SQL support for amavis, except for easier access to quarantined messages also provides features to change spam setting on a per user basis. Once you create the database, and configure @lookup_sql_dsn this part is very easy achieved by reading through the amavis documentation.

Messages that have a spam value between $sa_tag2_level_deflt and $sa_kill_level_deflt are delivered into the recipients mailbox and marked with “SPAM >” in the subject and have the X-Spam-Flag: YES header added to them. Although this may be enough for you, since I use only IMAP access to the mailboxes I wanted to move this messages to the Junk folder inside the users mailbox automatically. This has some benefits: Inbox will not be filled with spam messages, but in case of a false positive, user could check his Junk folder to check for messages marked as spam. This folder can also be used for learning SpamAssassing what is spam, and if users move the messages that are not recognized as spam to this folder instead of deleting them SpamAssassin will get smarter about what is considered to be spam, and what is not.

It's time to restart Postfix and Amavis and commit changes using etckeeper.

/etc/init.d/postfix restart
/etc/init.d/amavis restart
etckeeper commit "Configured amavisd-new"

6.3.1. Store configuration in a database

Documents regarding the use of amavisd-new with an SQL server are located in /usr/share/doc/amavisd-new. Use gunzip to uncompress the files ending with .gz.

When amavisd-new works with an SQL server it can use two sets of tables. The first part is read only and has the following tables: users, mailaddr, wblist and policy. The second part is read/write and is used when you store quarantined emails into the SQL. The tables in the second part are: maddr, msgs, msgrcpt, quarantine.

We are going to setup both sets of tables in the same database named mail_amavis. Enter mysql and run:

CREATE DATABASE mail_amavis;
CREATE USER 'amavis'@'localhost' IDENTIFIED BY 'new_password';
GRANT ALL PRIVILEGES ON `mail_amavis` . * TO 'amavis'@'localhost';

Create an amavis.sql file and paste this into it:

-- local users
  id         int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY,  -- unique id
  priority   integer      NOT NULL DEFAULT '7',  -- sort field, 0 is low prior.
  policy_id  integer unsigned NOT NULL DEFAULT '1',  -- JOINs with policy.id
  email      varbinary(255) NOT NULL UNIQUE,
  fullname   varchar(255) DEFAULT NULL,    -- not used by amavisd-new
  local      char(1)      -- Y/N  (optional field, see note further down)

-- any e-mail address (non- rfc2822-quoted), external or local,
-- used as senders in wblist
CREATE TABLE mailaddr (
  id         int unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY,
  priority   integer      NOT NULL DEFAULT '7',  -- 0 is low priority
  email      varbinary(255) NOT NULL UNIQUE

-- per-recipient whitelist and/or blacklist,
-- puts sender and recipient in relation wb  (white or blacklisted sender)
  rid        integer unsigned NOT NULL,  -- recipient: users.id
  sid        integer unsigned NOT NULL,  -- sender: mailaddr.id
  wb         varchar(10)  NOT NULL,  -- W or Y / B or N / space=neutral / score
  PRIMARY KEY (rid,sid)

                                    -- 'id' this is the _only_ required field
  policy_name      varchar(32),     -- not used by amavisd-new, a comment

  virus_lover          char(1) default NULL,     -- Y/N
  spam_lover           char(1) default NULL,     -- Y/N
  banned_files_lover   char(1) default NULL,     -- Y/N
  bad_header_lover     char(1) default NULL,     -- Y/N

  bypass_virus_checks  char(1) default NULL,     -- Y/N
  bypass_spam_checks   char(1) default NULL,     -- Y/N
  bypass_banned_checks char(1) default NULL,     -- Y/N
  bypass_header_checks char(1) default NULL,     -- Y/N

  spam_modifies_subj   char(1) default NULL,     -- Y/N

  virus_quarantine_to      varchar(64) default NULL,
  spam_quarantine_to       varchar(64) default NULL,
  banned_quarantine_to     varchar(64) default NULL,
  bad_header_quarantine_to varchar(64) default NULL,
  clean_quarantine_to      varchar(64) default NULL,
  other_quarantine_to      varchar(64) default NULL,

  spam_tag_level  float default NULL, -- higher score inserts spam info headers
  spam_tag2_level float default NULL, -- inserts 'declared spam' header fields
  spam_kill_level float default NULL, -- higher score triggers evasive actions
                                      -- e.g. reject/drop, quarantine, ...
                                     -- (subject to final_spam_destiny setting)
  spam_dsn_cutoff_level        float default NULL,
  spam_quarantine_cutoff_level float default NULL,

  addr_extension_virus      varchar(64) default NULL,
  addr_extension_spam       varchar(64) default NULL,
  addr_extension_banned     varchar(64) default NULL,
  addr_extension_bad_header varchar(64) default NULL,

  warnvirusrecip      char(1)     default NULL, -- Y/N
  warnbannedrecip     char(1)     default NULL, -- Y/N
  warnbadhrecip       char(1)     default NULL, -- Y/N
  newvirus_admin      varchar(64) default NULL,
  virus_admin         varchar(64) default NULL,
  banned_admin        varchar(64) default NULL,
  bad_header_admin    varchar(64) default NULL,
  spam_admin          varchar(64) default NULL,
  spam_subject_tag    varchar(64) default NULL,
  spam_subject_tag2   varchar(64) default NULL,
  message_size_limit  integer     default NULL, -- max size in bytes, 0 disable
  banned_rulenames    varchar(64) default NULL  -- comma-separated list of ...
        -- names mapped through %banned_rules to actual banned_filename tables

-- R/W part of the dataset (optional)
--   May reside in the same or in a separate database as lookups database;
--   REQUIRES SUPPORT FOR TRANSACTIONS; specified in @storage_sql_dsn
--  Wayne Smith adds: When using MySQL with InnoDB one might want to
--  increase buffer size for both pool and log, and might also want
--  to change flush settings for a little better performance. Example:
--    innodb_buffer_pool_size = 384M
--    innodb_log_buffer_size = 8M
--    innodb_flush_log_at_trx_commit = 0
--  The big performance increase is the first two, the third just helps with
--  lowering disk activity. Consider also adjusting the key_buffer_size.

-- provide unique id for each e-mail address, avoids storing copies
  partition_tag integer   DEFAULT 0,   -- see $sql_partition_tag
  id         bigint unsigned NOT NULL AUTO_INCREMENT PRIMARY KEY,
  email      varbinary(255) NOT NULL,  -- full mail address
  domain     varchar(255)   NOT NULL,  -- only domain part of the email address
                                       -- with subdomain fields in reverse
  CONSTRAINT part_email UNIQUE (partition_tag,email)

-- information pertaining to each processed message as a whole;
-- NOTE: records with NULL msgs.content should be ignored by utilities,
--   as such records correspond to messages just being processes, or were lost
-- NOTE: instead of a character field time_iso, one might prefer:
--   but the following MUST then be set in amavisd.conf: $timestamp_fmt_mysql=1
  partition_tag integer    DEFAULT 0,   -- see $sql_partition_tag
  mail_id    varbinary(12) NOT NULL PRIMARY KEY,  -- long-term unique mail id
  secret_id  varbinary(12)   DEFAULT '',  -- authorizes release of mail_id
  am_id      varchar(20)   NOT NULL,    -- id used in the log
  time_num   integer unsigned NOT NULL, -- rx_time: seconds since Unix epoch
  time_iso   char(16)      NOT NULL,    -- rx_time: ISO8601 UTC ascii time
  sid        bigint unsigned NOT NULL, -- sender: maddr.id
  policy     varchar(255)  DEFAULT '',  -- policy bank path (like macro %p)
  client_addr varchar(255) DEFAULT '',  -- SMTP client IP address (IPv4 or v6)
  size       integer unsigned NOT NULL, -- message size in bytes
  content    binary(1),                 -- content type: V/B/S/s/M/H/O/C:
    -- virus/banned/spam(kill)/spammy(tag2)/bad-mime/bad-header/oversized/clean
    -- is NULL on partially processed mail
    -- use binary instead of char for case sensitivity ('S' != 's')
  quar_type  binary(1),                 -- quarantined as: ' '/F/Z/B/Q/M/L
                                        --  none/file/zipfile/bsmtp/sql/
                                        --  /mailbox(smtp)/mailbox(lmtp)
  quar_loc   varbinary(255) DEFAULT '', -- quarantine location (e.g. file)
  dsn_sent   char(1),                   -- was DSN sent? Y/N/q (q=quenched)
  spam_level float,                     -- SA spam level (no boosts)
  message_id varchar(255)  DEFAULT '',  -- mail Message-ID header field
  from_addr  varchar(255)  DEFAULT '',  -- mail From header field,    UTF8
  subject    varchar(255)  DEFAULT '',  -- mail Subject header field, UTF8
  host       varchar(255)  NOT NULL,    -- hostname where amavisd is running
CREATE INDEX msgs_idx_sid      ON msgs (sid);
CREATE INDEX msgs_idx_mess_id  ON msgs (message_id); -- useful with pen pals
CREATE INDEX msgs_idx_time_num ON msgs (time_num);
-- alternatively when purging based on time_iso (instead of msgs_idx_time_num):
-- CREATE INDEX msgs_idx_time_iso ON msgs (time_iso);

-- per-recipient information related to each processed message;
-- NOTE: records in msgrcpt without corresponding msgs.mail_id record are
--  orphaned and should be ignored and eventually deleted by external utilities
CREATE TABLE msgrcpt (
  partition_tag integer    DEFAULT 0,    -- see $sql_partition_tag
  mail_id    varbinary(12) NOT NULL,     -- (must allow duplicates)
  rid        bigint unsigned NOT NULL,   -- recipient: maddr.id (dupl. allowed)
  ds         char(1)       NOT NULL,     -- delivery status: P/R/B/D/T
                                         -- pass/reject/bounce/discard/tempfail
  rs         char(1)       NOT NULL,     -- release status: initialized to ' '
  bl         char(1)       DEFAULT ' ',  -- sender blacklisted by this recip
  wl         char(1)       DEFAULT ' ',  -- sender whitelisted by this recip
  bspam_level float,                     -- spam level + per-recip boost
  smtp_resp  varchar(255)  DEFAULT '',   -- SMTP response given to MTA
CREATE INDEX msgrcpt_idx_mail_id  ON msgrcpt (mail_id);
CREATE INDEX msgrcpt_idx_rid      ON msgrcpt (rid);

-- mail quarantine in SQL, enabled by $*_quarantine_method='sql:'
-- NOTE: records in quarantine without corresponding msgs.mail_id record are
--  orphaned and should be ignored and eventually deleted by external utilities
CREATE TABLE quarantine (
  partition_tag integer    DEFAULT 0,    -- see $sql_partition_tag
  mail_id    varbinary(12) NOT NULL,     -- long-term unique mail id
  chunk_ind  integer unsigned NOT NULL,  -- chunk number, starting with 1
  mail_text  blob          NOT NULL,     -- store mail as chunks of octets
  PRIMARY KEY (mail_id,chunk_ind),

INSERT INTO policy (id, policy_name,
  virus_lover, spam_lover, banned_files_lover, bad_header_lover,
  bypass_virus_checks, bypass_spam_checks,
  bypass_banned_checks, bypass_header_checks, spam_modifies_subj,
  spam_tag_level, spam_tag2_level, spam_kill_level) VALUES
  (1, 'Non-paying',    'N','N','N','N', 'Y','Y','Y','N', 'Y', 3.0,   7, 10),
  (2, 'Uncensored',    'Y','Y','Y','Y', 'N','N','N','N', 'N', 3.0, 999, 999),
  (3, 'Wants all spam','N','Y','N','N', 'N','N','N','N', 'Y', 3.0, 999, 999),
  (4, 'Wants viruses', 'Y','N','Y','Y', 'N','N','N','N', 'Y', 3.0, 6.9, 6.9),
  (5, 'Normal',        'N','N','N','N', 'N','N','N','N', 'Y', -999, 6.9, 9.0),
  (6, 'Trigger happy', 'N','N','N','N', 'N','N','N','N', 'Y', 3.0,   5, 5),
  (7, 'Permissive',    'N','N','N','Y', 'N','N','N','N', 'Y', 3.0,  10, 20);

To run this SQL:

mysql -p mail_amavis < amavis-schema.sql

Now that we have the database set we need to reconfigure amavisd-new.

Open /etc/amavis/conf.d/50-user and add/set the the information that amavis needs to connect to the database (replace new_passwd with the password you chose for the amavis user):

@lookup_sql_dsn =  ( ['DBI:mysql:database=mail_amavis;host=;port=3306', 'amavis', 'new_passwd']);
@storage_sql_dsn = @lookup_sql_dsn;

If you decided to split read and read/write parts of the amavis database you will need to set @storage_sql_dsn with the correct options.

As we mentiond before, we were using @local_domains_acl = ( ".$mydomain", "domain2.com", "domain3.org" ); to tell amavis to run spam and virus checks for mail addressed to one of this domains. This is a problem because every time we add another domain we have to edit the configuration file to add the new domain to the list of local domains.

To solve this problem we have two options. One is to just check all of the incoming mail to the system for spam and viruses regardless of the recipients domaint. We can do this by rewriting the query that amavis uses to check if the email address is local to the system:

$sql_select_policy = 'SELECT "Y" AS local, 1 AS id;

But, this is not the recommended way. I didn't even test this, so your milage may vary. The recommended way is to just insert every domain you host in the users table.

For example:

mysql> USE mail_amavis;
Database changed
INSERT INTO users VALUES ( NULL, 5, 5, '@example.com',      NULL, 'Y');

To explain what this does, first lets take a look at the users table now:

mysql> select * from users;
| id | priority | policy_id | email        | fullname | local |
|  1 |        5 |         5 | @example.com | NULL     | Y     |

This raw in the table will match all email directed to any email address @example.com domain (that is because the local is set to Y). If we add another raw to the table that has email set to someuser@example.com and priority is higher than 5 this users policy will have precedance over the domains policy.

And finaly the policy_id. In this example it says that all the email directed to @example.com domain will use a policy with an identifier 5. If you take a look at the amavis-schema.sql file we used to create the database for amavis, at the end we also inserted some policies, and the policy with an ID 5 was inserted like this:

INSERT INTO policy (id, policy_name,
  virus_lover, spam_lover, banned_files_lover, bad_header_lover,
  bypass_virus_checks, bypass_spam_checks,
  bypass_banned_checks, bypass_header_checks, spam_modifies_subj,
  spam_tag_level, spam_tag2_level, spam_kill_level) VALUES
  (5, 'Normal',        'N','N','N','N', 'N','N','N','N', 'Y', -999, 6.9, 9);

We will skeep through most of the options (you have amavisd-new documentation if you want to go into detail) but we will look at the last three numbers inserted. spam_tag_level, spam_tag2_level and spam_kill_level are set to 3.0, 6.9 and 8.0 respectively. This means that emails that are flagged with over -999 (that should be all emails) by SpamAssassin will have X-Spam headers added to them. Emails tagged over 6.9 will have "SPAM" added to the subject and emails tagged over 9 will trigger spam evasive actions (depends on what you have $final_spam_destiny set to in the config file /etc/amavis/conf.d/50-user).

If you still want to set the same settings for all of the domains you can add a catchall user to the users table that will use the policy with ID 5.

INSERT INTO users VALUES (NULL, 0, 5, '@.',             NULL, 'Y');

6.3.2. Maintenance

You will have to cleanup the records in the MySQL from time to time, so we are going to create a PHP script that is going to be run daily with CRON.


* Amavisd-new cleanup script
* Based on the Amavisd-new documentation shipped with amavisd-new package in
* Debian 6.0 (Squeeze). Version of Amavisd-new 2.6.4.
* @author Goran Juric

$host = 'localhost';
$user = 'amavis';
$password = 'new_passwd';
$database = 'mail_amavis';
$keep_days = 14;

// Calculate the timestamp
$prune_older_then = time() - ($keep_days * 24 * 60 * 60);

// Connect to the database
$link = mysql_connect($host, $user, $password);
if (!$link) {
    die('Could not connect: ' . mysql_error());
$db = mysql_select_db($database, $link);
if (!$db) {
    die ("Can't use $database : " . mysql_error());

// Delete old msgs records based on timestamps only (for time_iso see next),
// and delete leftover msgs records from aborted mail checking operations
mysql_query("DELETE FROM msgs WHERE time_num < $prune_older_then ;", $link)
    or die(mysql_error());;
mysql_query("DELETE FROM msgs WHERE time_num < 60*60 AND content IS NULL;", $link)
    or die(mysql_error());;

// Delete unreferenced e-mail addresses
mysql_query("DELETE FROM maddr
AND NOT EXISTS (SELECT 1 FROM msgrcpt WHERE rid=id);", $link)
    or die(mysql_error());;

// When a FOREIGN KEY ... ON DELETE CASCADE is not used, tables msgrcpt
// and quarantine need to be purged explicitly, e.g.:
mysql_query("DELETE FROM quarantine
WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=quarantine.mail_id);", $link)
    or die(mysql_error());;
mysql_query("DELETE FROM msgrcpt
WHERE NOT EXISTS (SELECT 1 FROM msgs WHERE mail_id=msgrcpt.mail_id);", $link)
    or die(mysql_error());;

// Optimize tables
mysql_query("OPTIMIZE TABLE msgs, msgrcpt, quarantine, maddr;", $link)
    or die(mysql_error());;

// Close the database

Save this file to /etc/cron.daily/amavisd-new-cleanup (change the password at the top of the file) and make the script executable:

chmod +x /etc/cron.daily/amavisd-new-cleanup
chmod 700 /etc/cron.daily/amavisd-new-cleanup

6.3.3. Utilities

If you want a GUI for managing your Amavisd-new configuration take a look at Postvis Admin.