cleanup

Signed-off-by: Thomas Oettli <spacefreak@noop.ch>

README.md

@@ -6,10 +6,10 @@ A pymilter based sendmail/postfix pre-queue filter with the ability to ...
 * append and prepend disclaimers to e-mail text parts
 * quarantine e-mails (store e-mail, optionally notify receivers)
 
-It is useful in many cases due to its flexible configuration and the ability to handle any number of quarantines and modifications sequential and conditional. Storages and whitelists used by quarantines can be managed with the built-in CLI.  
+It is useful in many cases due to its flexible configuration and the ability to handle any number of quarantines and modifications sequential and conditional. Storages and lists used by quarantines can be managed with the built-in CLI.  
 
 Addionally, pyquarantine-milter provides a sanitized, harmless version of the text parts of e-mails as template variable, which can be embedded in e-mail notifications. This makes it easier for users to decide, if a match is a false-positive or not.  
-It is also possible to use any metavariable as template variable (e.g. storage ID, envelope-from address, ...). This may be used to give your users the ability to release e-mails or whitelist the from-address for example. A webservice then releases the e-mail from the quarantine.  
+It is also possible to use any metavariable as template variable (e.g. storage ID, envelope-from address, ...). This may be used to give your users the ability to release e-mails or add the from-address to an allowlist. A webservice then releases the e-mail from the quarantine.  
 
 The project is currently in beta status, but it is already used in a productive enterprise environment that processes about a million e-mails per month.
 
@@ -76,9 +76,40 @@ Global config options:
   * **debug**  
 * **pretend** (optional, default: false)  
   Pretend actions, for test purposes. This option may be overriden by any rule or action object.  
+* **storages**  
+  Object containing Storage objects.
+* **notifications**  
+  Object containing Notification objects.
+* **lists**  
+  Object containing List objects.
 * **rules**  
   List of rule objects.
 
+### Storage
+Config options for Storage objects:
+* **type**  
+  See section [Storage types](#Storage-types).
+* **original** (optional, default: false)  
+  If set to true, store the message as received by the MTA instead of storing the current state of the message, that may was modified already by other actions.
+* **metadata** (optional, default: false)  
+  Store metadata.
+* **metavar**  (optional)  
+  Prefix for the metavariable names. If not set, no metavariables will be provided.  
+  The storage provides the following metavariables:
+  * **ID** (the storage ID of the e-mail)  
+  * **DATAFILE** (path to the data file)  
+  * **METAFILE** (path to the meta file if **metadata** is set to **true**)  
+
+### Notification
+Config options for Notification objects:
+* **type**  
+  See section [Notification types](#Notification-types).
+
+### List
+Config options for List objects:
+* **type**  
+  See section [List types](#List-types).
+
 ### Rule
 Config options for rule objects:
 * **name**  
@@ -119,19 +150,14 @@ Config options for conditions objects:
   Matches e-mails for which all envelope-to addresses match the given regular expression.
 * **headers** (optional)  
   Matches e-mails for which all regular expressions in the given list are matching at least one e-mail header.
-* **whitelist** (optional)  
-  Matches e-mails for which the whitelist has no entry for the envelope-from and envelope-to address combination, see section [Whitelist](#Whitelist) for details.
+* **list** (optional)  
+  Matches e-mails for which the given list has an entry for the envelope-from and envelope-to address combination, see section [List](#List) for details.
 * **var** (optional)  
   Matches e-mails for which a previous action or condition has set the given metavariable.
 * **metavar** (optional)  
   Prefix for the name of metavariables which are possibly provided by the **envfrom**, **envto** or **headers** condition. Meta variables will be provided if the regular expressions contain named subgroups, see [python.re](https://docs.python.org/3/library/re.html) for details.
   If not set, no metavariables will be provided.
 
-### Whitelist
-Config options for whitelist objects:
-* **type**  
-  See section [Whitelist types](#Whitelist-types).
-
 ### Action types
 Available action types:
 ##### add_header
@@ -187,37 +213,27 @@ Options:
 ##### store
 Store e-mail.  
 Options:
-* **type**  
-  See section [Storage types](#Storage-types).
-* **original** (optional, default: false)  
-  If set to true, store the message as received by the MTA instead of storing the current state of the message, that may was modified already by other actions.
-* **metadata** (optional, default: false)  
-  Store metadata.
-* **metavar**  (optional)  
-  Prefix for the metavariable names. If not set, no metavariables will be provided.  
-  The storage provides the following metavariables:
-  * **ID** (the storage ID of the e-mail)  
-  * **DATAFILE** (path to the data file)  
-  * **METAFILE** (path to the meta file if **metadata** is set to **true**)  
+* **storage**  
+  Index of a Storage object in the global storages object.
 
 ##### notify
 Send notification.  
 Options:
-* **type**  
-  See section [Notification types](#Notification-types).
+* **notification**  
+  Index of a Notification object in the global notifications object.
 
 ##### quarantine
 Quarantine e-mail.  
 Options:
-* **store**  
-  Options for e-mail storage, see action [store](#store).  
-  If the option **metadata** is not specificall set for this storage, it will be set to true.
+* **storage**  
+  Index of a Storage object in the global storages object.
+  If the option **metadata** is not specifically set for this storage, it will be set to true.
 * **smtp_host**  
   SMTP host used to release e-mails from quarantine.
 * **smtp_port**  
   SMTP port used to release e-mails from quarantine.
-* **notify** (optional)  
-  Options for e-mail notifications, see action [notify](#notify).
+* **notification** (optional)  
+  Index of a Notification object in the global notifications object.
 * **milter_action** (optional)  
   Milter action to perform. If set, no further rules or actions will be processed.  
   Please think carefully what you set here or your MTA may do something you do not want it to do.  
@@ -230,8 +246,9 @@ Options:
     Tell the MTA to silently discard the e-mail.
 * **reject_reason** (optional, default: "Message rejected")  
   Reject message sent to MTA if milter_action is set to reject.
-* **whitelist** (optional)  
-  Options for a whitelist object, see section [Whitelist](#Whitelist).
+* **allowlist** (optional)  
+  Ignore e-mails for which the given list has an entry for the envelope-from and envelope-to address combination, see section [List](#List) for details.  
+  If an e-mail as multiple recipients, the decision is made per recipient.
 
 ### Storage types
 Available storage types:
@@ -289,10 +306,10 @@ Options:
 * **embed_imgs** (optional)  
   List of images to embed into the notification e-mail. The Content-ID of each image will be set to the filename, so you can reference it from the e-mail template.
 
-### Whitelist types
-Available whitelist types:
+### List types
+Available list types:
 ##### db
-Whitelist stored in database. The table is created automatically if it does not exist yet.  
+List stored in database. The table is created automatically if it does not exist yet.  
 Options:
 * **connection**  
   Database connection string, see [Peewee Playhouse Extension](https://docs.peewee-orm.com/en/latest/peewee/playhouse.html#db-url).
@@ -311,6 +328,38 @@ In this example it is assumed, that another milter (e.g. Amavisd or Rspamd) adds
 ```json
 {
     "socket": "unix:/tmp/pyquarantine.sock",
+    "storages": {
+        "virus": {
+            "type": "file",
+            "directory": "/mnt/data/quarantine/virus"
+        },
+        "spam": {
+            "type": "file",
+            "directory": "/mnt/data/quarantine/spam"
+        }
+    },
+    "notifications": {
+        "virus": {
+            "type": "email",
+            "smtp_host": "localhost",
+            "smtp_port": 2525,
+            "envelope_from": "notifications@example.com",
+            "from_header": "{FROM}",
+            "subject": "[VIRUS] {SUBJECT}",
+            "template": "/etc/pyquarantine/templates/notification.template",
+            "repl_img": "/etc/pyquarantine/templates/removed.png"
+        },
+        "spam": {
+            "type": "email",
+            "smtp_host": "localhost",
+            "smtp_port": 2525,
+            "envelope_from": "notifications@example.com",
+            "from_header": "{FROM}",
+            "subject": "[SPAM] {SUBJECT}",
+            "template": "/etc/pyquarantine/templates/notification.template",
+            "repl_img": "/etc/pyquarantine/templates/removed.png"
+        }
+    },
     "rules": [
         {
             "name": "inbound",
@@ -322,23 +371,11 @@ In this example it is assumed, that another milter (e.g. Amavisd or Rspamd) adds
                     "name": "virus",
                     "type": "quarantine",
                     "conditions": {
-                        "headers": ["^X-Virus: Yes"],
+                        "headers": ["^X-Virus: Yes"]
                     },
                     "options": {
-                        "store": {
-                            "type": "file",
-                            "directory": "/mnt/data/quarantine/virus",
-                        },
-                        "notify": {
-                            "type": "email",
-                            "smtp_host": "localhost",
-                            "smtp_port": 2525,
-                            "envelope_from": "notifications@example.com",
-                            "from_header": "{FROM}",
-                            "subject": "[VIRUS] {SUBJECT}",
-                            "template": "/etc/pyquarantine/templates/notification.template",
-                            "repl_img": "/etc/pyquarantine/templates/removed.png"
-                        },
+                        "storage": "virus",
+                        "notification": "virus",
                         "smtp_host": "localhost",
                         "smtp_port": 2525,
                         "milter_action": "REJECT",
@@ -351,20 +388,8 @@ In this example it is assumed, that another milter (e.g. Amavisd or Rspamd) adds
                         "headers": ["^X-Spam: Yes"]
                     },
                     "options": {
-                        "store": {
-                            "type": "file",
-                            "directory": "/mnt/data/quarantine/spam",
-                        },
-                        "notify": {
-                            "type": "email",
-                            "smtp_host": "localhost",
-                            "smtp_port": 2525,
-                            "envelope_from": "notifications@example.com",
-                            "from_header": "{FROM}",
-                            "subject": "[SPAM] {SUBJECT}",
-                            "template": "/etc/pyquarantine/templates/notification.template",
-                            "repl_img": "/etc/pyquarantine/templates/removed.png"
-                        },
+                        "storage": "spam",
+                        "notification": "spam",
                         "smtp_host": "localhost",
                         "smtp_port": 2525,
                         "milter_action": "DISCARD"
@@ -420,6 +445,13 @@ In this example it is assumed, that another milter (e.g. Amavisd or Rspamd) adds
 ```json
 {
     "socket": "unix:/tmp/pyquarantine.sock",
+    "storages": {
+        "orig": {
+            "type": "file",
+            "directory": "/mnt/data/incoming",
+            "original": true
+        }
+    },
     "rules": [
         {
             "name": "inbound",
@@ -431,9 +463,7 @@ In this example it is assumed, that another milter (e.g. Amavisd or Rspamd) adds
                     "name": "store_original",
                     "type": "store",
                     "options": {
-                        "type": "file",
-                        "directory": "/mnt/data/incoming",
-                        "original": true,
+                        "storage": "orig"
                     }
                 }
             ]

distribution/gentoo/mail-filter/pyquarantine-milter/pyquarantine-milter-9999.ebuild

@@ -2,7 +2,7 @@
 # Distributed under the terms of the GNU General Public License v2
 
 EAPI=7
-PYTHON_COMPAT=( python3_{9,10} )
+PYTHON_COMPAT=( python3_{10..11} )
 DISTUTILS_USE_SETUPTOOLS=rdepend
 
 SCM=""
@@ -37,7 +37,7 @@ RDEPEND="
 	lxml? ( dev-python/lxml[${PYTHON_USEDEP}] )
 	dev-python/netaddr[${PYTHON_USEDEP}]
 	dev-python/peewee[${PYTHON_USEDEP}]
-	dev-python/pymilter[${PYTHON_USEDEP}]"
+	>=dev-python/pymilter-1.5[${PYTHON_USEDEP}]"
 
 python_install_all() {
 	distutils-r1_python_install_all

pyquarantine/__init__.py

@@ -24,10 +24,10 @@ __all__ = [
     "rule",
     "run",
     "storage",
-    "whitelist",
+    "lists",
     "QuarantineMilter"]
 
-__version__ = "2.0.3"
+__version__ = "2.1.0"
 
 from pyquarantine import _runtime_patches
 
@@ -71,7 +71,7 @@ class QuarantineMilter(Milter.Base):
 
         logger = logging.getLogger(__name__)
         logger.setLevel(QuarantineMilter._loglevel)
-        for idx, rule_cfg in enumerate(cfg["rules"]):
+        for rule_cfg in cfg["rules"]:
             rule = Rule(rule_cfg, local_addrs, debug)
             logger.debug(rule)
             QuarantineMilter._rules.append(rule)
@@ -208,6 +208,7 @@ class QuarantineMilter(Milter.Base):
 
         return Milter.CONTINUE
 
+    @Milter.decode("replace")
     def envfrom(self, mailfrom, *str):
         try:
             self.mailfrom = "@".join(parse_addr(mailfrom)).lower()
@@ -219,6 +220,7 @@ class QuarantineMilter(Milter.Base):
 
         return Milter.CONTINUE
 
+    @Milter.decode("replace")
     def envrcpt(self, to, *str):
         try:
             self.rcpts.add("@".join(parse_addr(to)).lower())
@@ -243,6 +245,7 @@ class QuarantineMilter(Milter.Base):
 
         return Milter.CONTINUE
 
+    @Milter.decode("replace")
     def header(self, field, value):
         try:
             # remove CR and LF from address fields, otherwise pythons

pyquarantine/_install.py

@@ -21,14 +21,18 @@ import shutil
 import sys
 
 
-SYSTEMD_PATH = "/lib/systemd/system"
+SYSTEMD_PATHS = ["/lib/systemd/system", "/usr/lib/systemd/system"]
 OPENRC = "/sbin/openrc"
 
 
 def _systemd_files(pkg_dir, name):
+    for path in SYSTEMD_PATHS:
+        if os.path.isdir(path):
+            break
+
     return [
         (f"{pkg_dir}/misc/systemd/{name}-milter.service",
-            f"{SYSTEMD_PATH}/{name}-milter.service", True)]
+            f"{path}/{name}-milter.service", True)]
 
 
 def _openrc_files(pkg_dir, name):
@@ -117,7 +121,11 @@ def _check_root():
 
 
 def _check_systemd():
-    systemd = os.path.isdir(SYSTEMD_PATH)
+    for path in SYSTEMD_PATHS:
+        systemd = os.path.isdir(path)
+        if systemd:
+            break
+
     if systemd:
         logging.info("systemd detected")
 

pyquarantine/_runtime_patches.py

@@ -150,68 +150,6 @@ def get_obs_local_part(value):
 setattr(email._header_value_parser, "get_obs_local_part", get_obs_local_part)
 
 
-# https://bugs.python.org/issue30681
-#
-# fix: https://github.com/python/cpython/pull/22090
-
-import email.errors
-from email.errors import HeaderDefect
-
-
-class InvalidDateDefect(HeaderDefect):
-    """Header has unparseable or invalid date"""
-
-
-setattr(email.errors, "InvalidDateDefect", InvalidDateDefect)
-
-
-import email.utils
-from email.utils import _parsedate_tz
-import datetime
-
-
-def parsedate_to_datetime(data):
-    parsed_date_tz = _parsedate_tz(data)
-    if parsed_date_tz is None:
-        raise ValueError('Invalid date value or format "%s"' % str(data))
-    *dtuple, tz = parsed_date_tz
-    if tz is None:
-        return datetime.datetime(*dtuple[:6])
-    return datetime.datetime(*dtuple[:6],
-            tzinfo=datetime.timezone(datetime.timedelta(seconds=tz)))
-
-
-setattr(email.utils, "parsedate_to_datetime", parsedate_to_datetime)
-
-
-import email.headerregistry
-from email import utils, _header_value_parser as parser
-
-@classmethod
-def parse(cls, value, kwds):
-    if not value:
-        kwds['defects'].append(errors.HeaderMissingRequiredValue())
-        kwds['datetime'] = None
-        kwds['decoded'] = ''
-        kwds['parse_tree'] = parser.TokenList()
-        return
-    if isinstance(value, str):
-        kwds['decoded'] = value
-        try:
-            value = utils.parsedate_to_datetime(value)
-        except ValueError:
-            kwds['defects'].append(errors.InvalidDateDefect('Invalid date value or format'))
-            kwds['datetime'] = None
-            kwds['parse_tree'] = parser.TokenList()
-            return
-    kwds['datetime'] = value
-    kwds['decoded'] = utils.format_datetime(kwds['datetime'])
-    kwds['parse_tree'] = cls.value_parser(kwds['decoded'])
-
-
-setattr(email.headerregistry.DateHeader, "parse", parse)
-
-
 #######################################
 #  add charset alias for windows-874  #
 #######################################

pyquarantine/action.py

@@ -14,6 +14,7 @@
 
 __all__ = ["Action"]
 
+from copy import deepcopy
 from pyquarantine import modify, notify, storage
 from pyquarantine.conditions import Conditions
 
@@ -39,9 +40,8 @@ class Action:
             self.conditions["loglevel"] = cfg["loglevel"]
             self.conditions = Conditions(self.conditions, local_addrs, debug)
 
-        action_type = cfg["type"]
-        self.action = self.ACTION_TYPES[action_type](
-            cfg, local_addrs, debug)
+        self.action = self.ACTION_TYPES[cfg["type"]](
+            deepcopy(cfg), local_addrs, debug)
 
     def __str__(self):
         cfg = []

pyquarantine/base.py

@@ -71,7 +71,7 @@ class MilterMessage(MIMEPart):
         maintype, subtype = part.get_content_type().split("/")
         if maintype == "text":
             if subtype in preferencelist:
-                yield(preferencelist.index(subtype), parent)
+                yield (preferencelist.index(subtype), parent)
             return
         if maintype != "multipart" or not self.is_multipart():
             return
@@ -81,7 +81,7 @@ class MilterMessage(MIMEPart):
                     subpart, preferencelist, part)
             return
         if 'related' in preferencelist:
-            yield(preferencelist.index('related'), parent)
+            yield (preferencelist.index('related'), parent)
         candidate = None
         start = part.get_param('start')
         if start:
@@ -162,8 +162,9 @@ class MilterMessage(MIMEPart):
 def inject_body_part(part, content, subtype="plain"):
     parts = []
     text_body = None
+    text_content = None
     if subtype == "html":
-        text_body = part.get_body(preferencelist=("plain"))
+        text_body, text_content = part.get_body_content("plain")
 
     for p in part.iter_parts():
         if text_body and p == text_body:
@@ -173,8 +174,8 @@ def inject_body_part(part, content, subtype="plain"):
     boundary = part.get_boundary()
     p_subtype = part.get_content_subtype()
     part.clear_content()
-    if text_body:
-        part.set_content(content)
+    if text_content is not None:
+        part.set_content(text_content)
         part.add_alternative(content, subtype=subtype)
     else:
         part.set_content(content, subtype=subtype)

pyquarantine/cli.py

@@ -23,31 +23,49 @@ import time
 
 from pyquarantine.config import get_milter_config, ActionConfig
 from pyquarantine.storage import Quarantine
+from pyquarantine.lists import DatabaseList
 from pyquarantine import __version__ as version
 
 
-def _get_quarantine(quarantines, name, debug):
+def _get_quarantines(milter_cfg):
+    quarantines = []
+    for rule in milter_cfg["rules"]:
+        for action in rule["actions"]:
+            if action["type"] == "quarantine":
+                quarantines.append(action)
+    return quarantines
+
+
+def _get_quarantine(milter_cfg, name, debug):
     try:
-        quarantine = next((q for q in quarantines if q["name"] == name))
+        quarantine = next(
+            (q for q in _get_quarantines(milter_cfg) if q["name"] == name))
     except StopIteration:
         raise RuntimeError(f"invalid quarantine '{name}'")
-    return Quarantine(ActionConfig(quarantine), [], debug)
+
+    cfg = ActionConfig(quarantine, milter_cfg)
+    return Quarantine(cfg, [], debug)
 
 
-def _get_notification(quarantines, name, debug):
-    notification = _get_quarantine(quarantines, name, debug).notification
+def _get_notification(cfg, name, debug):
+    notification = _get_quarantine(cfg, name, debug).notification
     if not notification:
         raise RuntimeError(
                 "notification type is set to NONE")
     return notification
 
 
-def _get_whitelist(quarantines, name, debug):
-    whitelist = _get_quarantine(quarantines, name, debug).whitelist
-    if not whitelist:
-        raise RuntimeError(
-                "whitelist type is set to NONE")
-    return whitelist
+def _get_list(cfg, name, debug):
+    try:
+        list_cfg = cfg["lists"][name]
+    except KeyError:
+        raise RuntimeError(f"list '{name}' is not configured")
+
+    if list_cfg["type"] == "db":
+        list_cfg["loglevel"] = cfg["loglevel"]
+        return DatabaseList(list_cfg, debug)
+    else:
+        raise RuntimeError("invalid lists type")
 
 
 def print_table(columns, rows):
@@ -91,49 +109,66 @@ def print_table(columns, rows):
         print(row_format.format(*row))
 
 
-def list_quarantines(quarantines, args):
+def show(cfg, args):
+    quarantines = _get_quarantines(cfg)
     if args.batch:
         print("\n".join([q["name"] for q in quarantines]))
     else:
         qlist = []
         for q in quarantines:
-            cfg = q["options"]
-            storage_type = cfg["store"]["type"]
+            qcfg = q["options"]
 
-            if "notify" in cfg:
-                notification_type = cfg["notify"]["type"]
+            if "notification" in qcfg:
+                notification = cfg["notifications"][qcfg["notification"]]
+                notify_type = notification["type"]
             else:
-                notification_type = "NONE"
+                notify_type = "NONE"
 
-            if "whitelist" in cfg:
-                whitelist_type = cfg["whitelist"]["type"]
+            if "allowlist" in qcfg:
+                allowlist = qcfg["allowlist"]
             else:
-                whitelist_type = "NONE"
+                allowlist = "NONE"
 
-            if "milter_action" in cfg:
-                milter_action = cfg["milter_action"]
+            if "milter_action" in qcfg:
+                milter_action = qcfg["milter_action"]
             else:
                 milter_action = "NONE"
 
+            storage_type = cfg["storages"][qcfg["storage"]]["type"]
+
             qlist.append({
                 "name": q["name"],
                 "storage": storage_type,
-                "notification": notification_type,
-                "whitelist": whitelist_type,
+                "notification": notify_type,
+                "lists": allowlist,
                 "action": milter_action})
 
         print_table(
-            [("Name", "name"),
+            [("Quarantine", "name"),
              ("Storage", "storage"),
              ("Notification", "notification"),
-             ("Whitelist", "whitelist"),
+             ("Allowlist", "lists"),
              ("Action", "action")],
             qlist
         )
 
+        if "lists" in cfg:
+            lst_list = []
+            for name, options in cfg["lists"].items():
+                lst_list.append({
+                    "name": name,
+                    "type": options["type"]})
 
-def list_quarantine_emails(quarantines, args):
-    storage = _get_quarantine(quarantines, args.quarantine, args.debug).storage
+            print("\n")
+            print_table(
+                [("List", "name"),
+                 ("Type", "type")],
+                lst_list
+            )
+
+
+def list_quarantine_emails(cfg, args):
+    storage = _get_quarantine(cfg, args.quarantine, args.debug).storage
 
     # find emails and transform some metadata values to strings
     rows = []
@@ -180,16 +215,16 @@ def list_quarantine_emails(quarantines, args):
     )
 
 
-def list_whitelist(quarantines, args):
-    whitelist = _get_whitelist(quarantines, args.quarantine, args.debug)
+def list_list(cfg, args):
+    lst = _get_list(cfg, args.list, args.debug)
 
-    # find whitelist entries
-    entries = whitelist.find(
+    # find lists entries
+    entries = lst.find(
         mailfrom=args.mailfrom,
         recipients=args.recipients,
         older_than=args.older_than)
     if not entries:
-        print(f"whitelist of quarantine '{args.quarantine}' is empty")
+        print("list is empty")
         return
 
     # transform some values to strings
@@ -210,12 +245,12 @@ def list_whitelist(quarantines, args):
     )
 
 
-def add_whitelist_entry(quarantines, args):
+def add_list_entry(cfg, args):
     logger = logging.getLogger(__name__)
-    whitelist = _get_whitelist(quarantines, args.quarantine, args.debug)
+    lst = _get_list(cfg, args.list, args.debug)
 
     # check existing entries
-    entries = whitelist.check(args.mailfrom, args.recipient, logger)
+    entries = lst.check(args.mailfrom, args.recipient, logger)
     if entries:
         # check if the exact entry exists already
         for entry in entries.values():
@@ -245,47 +280,56 @@ def add_whitelist_entry(quarantines, args):
                 "from/to combination is already covered by the entries above, "
                 "use --force to override.")
 
-    # add entry to whitelist
-    whitelist.add(args.mailfrom, args.recipient, args.comment, args.permanent)
-    print("whitelist entry added successfully")
+    # add entry to lists
+    lst.add(args.mailfrom, args.recipient, args.comment, args.permanent)
+    print("list entry added successfully")
 
 
-def delete_whitelist_entry(quarantines, args):
-    whitelist = _get_whitelist(quarantines, args.quarantine, args.debug)
-    whitelist.delete(args.whitelist_id)
-    print("whitelist entry deleted successfully")
+def delete_list_entry(cfg, args):
+    lst = _get_list(cfg, args.list, args.debug)
+    lst.delete(args.list_id)
+    print("list entry deleted successfully")
 
 
-def notify(quarantines, args):
-    quarantine = _get_quarantine(quarantines, args.quarantine, args.debug)
+def notify(cfg, args):
+    quarantine = _get_quarantine(cfg, args.quarantine, args.debug)
     quarantine.notify(args.quarantine_id, args.recipient)
     print("notification sent successfully")
 
 
-def release(quarantines, args):
+def release(cfg, args):
     logger = logging.getLogger(__name__)
-    quarantine = _get_quarantine(quarantines, args.quarantine, args.debug)
+    quarantine = _get_quarantine(cfg, args.quarantine, args.debug)
     rcpts = quarantine.release(args.quarantine_id, args.recipient)
     rcpts = ", ".join(rcpts)
     logger.info(
-        f"{args.quarantine}: released message with id {args.quarantine_id} "
-        f"for {rcpts}")
+        f"{quarantine.cfg['name']}: released message with id "
+        f"{args.quarantine_id} for {rcpts}")
 
 
-def delete(quarantines, args):
-    storage = _get_quarantine(quarantines, args.quarantine, args.debug).storage
+def copy(cfg, args):
+    logger = logging.getLogger(__name__)
+    quarantine = _get_quarantine(cfg, args.quarantine, args.debug)
+    quarantine.copy(args.quarantine_id, args.recipient)
+    logger.info(
+        f"{args.quarantine}: sent a copy of message with id "
+        f"{args.quarantine_id} to {args.recipient}")
+
+
+def delete(cfg, args):
+    storage = _get_quarantine(cfg, args.quarantine, args.debug).storage
     storage.delete(args.quarantine_id, args.recipient)
     print("quarantined message deleted successfully")
 
 
-def get(quarantines, args):
-    storage = _get_quarantine(quarantines, args.quarantine, args.debug).storage
+def get(cfg, args):
+    storage = _get_quarantine(cfg, args.quarantine, args.debug).storage
     data = storage.get_mail_bytes(args.quarantine_id)
     sys.stdout.buffer.write(data)
 
 
-def metadata(quarantines, args):
-    storage = _get_quarantine(quarantines, args.quarantine, args.debug).storage
+def metadata(cfg, args):
+    storage = _get_quarantine(cfg, args.quarantine, args.debug).storage
     metadata = storage.get_metadata(args.quarantine_id)
     print(json.dumps(metadata))
 
@@ -330,15 +374,15 @@ def main():
     subparsers.required = True
 
     # list command
-    list_parser = subparsers.add_parser(
-        "list",
-        help="List available quarantines.",
+    show_parser = subparsers.add_parser(
+        "show",
+        help="Show quarantines.",
         formatter_class=formatter_class)
-    list_parser.add_argument(
+    show_parser.add_argument(
         "-b", "--batch",
         help="Print results using only quarantine names, each on a new line.",
         action="store_true")
-    list_parser.set_defaults(func=list_quarantines)
+    show_parser.set_defaults(func=show)
 
     # quarantine command group
     quar_parser = subparsers.add_parser(
@@ -432,6 +476,28 @@ def main():
         help="Release email for all recipients.",
         action="store_true")
     quar_release_parser.set_defaults(func=release)
+    # quarantine copy command
+    quar_copy_parser = quar_subparsers.add_parser(
+        "copy",
+        description="Send a copy of email to another recipient.",
+        help="Send a copy of email to another recipient.",
+        formatter_class=formatter_class)
+    quar_copy_parser.add_argument(
+        "quarantine_id",
+        metavar="ID",
+        help="Quarantine ID.")
+    quar_copy_parser.add_argument(
+        "-n",
+        "--disable-syslog",
+        dest="syslog",
+        help="Disable syslog messages.",
+        action="store_false")
+    quar_copy_parser.add_argument(
+        "-t", "--to",
+        dest="recipient",
+        required=True,
+        help="Release email for one recipient address.")
+    quar_copy_parser.set_defaults(func=copy)
     # quarantine delete command
     quar_delete_parser = quar_subparsers.add_parser(
         "delete",
@@ -481,86 +547,86 @@ def main():
         help="Quarantine ID.")
     quar_metadata_parser.set_defaults(func=metadata)
 
-    # whitelist command group
-    whitelist_parser = subparsers.add_parser(
-        "whitelist",
-        description="Manage whitelists.",
-        help="Manage whitelists.",
-        formatter_class=formatter_class)
-    whitelist_parser.add_argument(
-        "quarantine",
-        metavar="QUARANTINE",
-        help="Quarantine name.")
-    whitelist_subparsers = whitelist_parser.add_subparsers(
-        dest="command",
-        title="Whitelist commands")
-    whitelist_subparsers.required = True
-    # whitelist list command
-    whitelist_list_parser = whitelist_subparsers.add_parser(
+    # list command group
+    list_parser = subparsers.add_parser(
         "list",
-        description="List whitelist entries.",
-        help="List whitelist entries.",
+        description="Manage lists.",
+        help="Manage lists.",
         formatter_class=formatter_class)
-    whitelist_list_parser.add_argument(
+    list_parser.add_argument(
+        "list",
+        metavar="LIST",
+        help="List name.")
+    list_subparsers = list_parser.add_subparsers(
+        dest="command",
+        title="Lists commands.")
+    list_subparsers.required = True
+    # lists list command
+    list_list_parser = list_subparsers.add_parser(
+        "list",
+        description="List list entries.",
+        help="List list entries.",
+        formatter_class=formatter_class)
+    list_list_parser.add_argument(
         "-f", "--from",
         dest="mailfrom",
         help="Filter entries by from address.",
         default=None,
         nargs="+")
-    whitelist_list_parser.add_argument(
+    list_list_parser.add_argument(
         "-t", "--to",
         dest="recipients",
         help="Filter entries by recipient address.",
         default=None,
         nargs="+")
-    whitelist_list_parser.add_argument(
+    list_list_parser.add_argument(
         "-o", "--older-than",
         dest="older_than",
         help="Filter emails by last used date (days).",
         default=None,
         type=float)
-    whitelist_list_parser.set_defaults(func=list_whitelist)
-    # whitelist add command
-    whitelist_add_parser = whitelist_subparsers.add_parser(
+    list_list_parser.set_defaults(func=list_list)
+    # lists add command
+    list_add_parser = list_subparsers.add_parser(
         "add",
-        description="Add whitelist entry.",
-        help="Add whitelist entry.",
+        description="Add list entry.",
+        help="Add list entry.",
         formatter_class=formatter_class)
-    whitelist_add_parser.add_argument(
+    list_add_parser.add_argument(
         "-f", "--from",
         dest="mailfrom",
         help="From address.",
         required=True)
-    whitelist_add_parser.add_argument(
+    list_add_parser.add_argument(
         "-t", "--to",
         dest="recipient",
         help="Recipient address.",
         required=True)
-    whitelist_add_parser.add_argument(
+    list_add_parser.add_argument(
         "-c", "--comment",
         help="Comment.",
         default="added by CLI")
-    whitelist_add_parser.add_argument(
+    list_add_parser.add_argument(
         "-p", "--permanent",
         help="Add a permanent entry.",
         action="store_true")
-    whitelist_add_parser.add_argument(
+    list_add_parser.add_argument(
         "--force",
         help="Force adding an entry, "
              "even if already covered by another entry.",
         action="store_true")
-    whitelist_add_parser.set_defaults(func=add_whitelist_entry)
-    # whitelist delete command
-    whitelist_delete_parser = whitelist_subparsers.add_parser(
+    list_add_parser.set_defaults(func=add_list_entry)
+    # lists delete command
+    list_delete_parser = list_subparsers.add_parser(
         "delete",
-        description="Delete whitelist entry.",
-        help="Delete whitelist entry.",
+        description="Delete list entry.",
+        help="Delete list entry.",
         formatter_class=formatter_class)
-    whitelist_delete_parser.add_argument(
-        "whitelist_id",
+    list_delete_parser.add_argument(
+        "list_id",
         metavar="ID",
-        help="Whitelist ID.")
-    whitelist_delete_parser.set_defaults(func=delete_whitelist_entry)
+        help="List ID.")
+    list_delete_parser.set_defaults(func=delete_list_entry)
 
     args = parser.parse_args()
 
@@ -591,7 +657,8 @@ def main():
 
     try:
         logger.debug("read milter configuration")
-        cfg = get_milter_config(args.config, raw=True)
+        cfg = get_milter_config(args.config, rec=False)
+
         if "rules" not in cfg or not cfg["rules"]:
             raise RuntimeError("no rules configured")
 
@@ -599,16 +666,11 @@ def main():
             if "actions" not in rule or not rule["actions"]:
                 raise RuntimeError(
                     f"{rule['name']}: no actions configured")
+
     except (RuntimeError, AssertionError) as e:
         logger.error(f"config error: {e}")
         sys.exit(255)
 
-    quarantines = []
-    for rule in cfg["rules"]:
-        for action in rule["actions"]:
-            if action["type"] == "quarantine":
-                quarantines.append(action)
-
     if args.syslog:
         # setup syslog
         sysloghandler = logging.handlers.SysLogHandler(
@@ -625,7 +687,7 @@ def main():
 
     # call the commands function
     try:
-        args.func(quarantines, args)
+        args.func(cfg, args)
     except RuntimeError as e:
         logger.error(e)
         sys.exit(1)

pyquarantine/conditions.py

@@ -19,7 +19,7 @@ import re
 
 from netaddr import IPAddress, IPNetwork, AddrFormatError
 from pyquarantine import CustomLogger
-from pyquarantine.whitelist import DatabaseWhitelist
+from pyquarantine.lists import DatabaseList
 
 
 class Conditions:
@@ -33,7 +33,7 @@ class Conditions:
         self.logger.setLevel(cfg.get_loglevel(debug))
 
         for arg in ("local", "hosts", "envfrom", "envto", "headers", "metavar",
-                    "var"):
+                    "var", "list"):
             if arg not in cfg:
                 setattr(self, arg, None)
                 continue
@@ -59,30 +59,28 @@ class Conditions:
                             header, re.IGNORECASE + re.DOTALL + re.MULTILINE))
                 except re.error as e:
                     raise RuntimeError(e)
+            elif arg == "list":
+                if cfg["list"]["type"] == "db":
+                    cfg["list"]["name"] = cfg["name"]
+                    cfg["list"]["loglevel"] = cfg["loglevel"]
+                    self.list = DatabaseList(cfg["list"], debug)
+                else:
+                    raise RuntimeError("invalid list type")
             else:
                 setattr(self, arg, cfg[arg])
 
-        self.whitelist = cfg["whitelist"] if "whitelist" in cfg else None
-        if self.whitelist is not None:
-            self.whitelist["name"] = f"{cfg['name']}: whitelist"
-            self.whitelist["loglevel"] = cfg["loglevel"]
-            if self.whitelist["type"] == "db":
-                self.whitelist = DatabaseWhitelist(self.whitelist, debug)
-            else:
-                raise RuntimeError("invalid whitelist type")
-
     def __str__(self):
         cfg = []
         for arg in ("local", "hosts", "envfrom", "envto", "headers",
                     "var", "metavar"):
             if arg in self.cfg:
                 cfg.append(f"{arg}={self.cfg[arg]}")
-        if self.whitelist is not None:
-            cfg.append(f"whitelist={self.whitelist}")
+        if self.list is not None:
+            cfg.append(f"list={self.list}")
         return "Conditions(" + ", ".join(cfg) + ")"
 
-    def get_whitelist(self):
-        return self.whitelist
+    def get_list(self):
+        return self.list
 
     def match_host(self, host):
         logger = CustomLogger(
@@ -123,17 +121,6 @@ class Conditions:
 
         return True
 
-    def get_wl_rcpts(self, mailfrom, rcpts, logger):
-        if not self.whitelist:
-            return {}
-
-        wl_rcpts = []
-        for rcpt in rcpts:
-            if self.whitelist.check(mailfrom, rcpt, logger):
-                wl_rcpts.append(rcpt)
-
-        return wl_rcpts
-
     def update_msginfo_from_match(self, milter, match):
         if self.metavar is None:
             return
@@ -175,9 +162,9 @@ class Conditions:
                         f"envto does not match")
                     return False
 
-            logger.debug(
-                f"envto matches for "
-                f"envelope-to address {envto}")
+                logger.debug(
+                    f"envto matches for "
+                    f"envelope-to address {envto}")
             self.update_msginfo_from_match(milter, match)
 
         if self.headers is not None:
@@ -198,6 +185,27 @@ class Conditions:
 
         if self.var is not None:
             if self.var not in milter.msginfo["vars"]:
+                logger.debug(
+                    "ignore message, "
+                    "vars does not match")
                 return False
 
+            logger.debug(f"vars matches, variable {self.var} is available")
+
+        if self.list is not None:
+            envfrom = milter.msginfo["mailfrom"]
+            envto = milter.msginfo["rcpts"]
+            if not isinstance(envto, list):
+                envto = [envto]
+
+            for to in envto:
+                if not self.list.check(envfrom, to, logger):
+                    logger.debug(
+                        "ignore message, "
+                        "list does not match")
+                    return False
+
+                logger.debug(
+                    "list matches envelope-from and envelope-to address")
+
         return True

pyquarantine/config.py

@@ -20,9 +20,11 @@ __all__ = [
     "DelHeaderConfig",
     "AddDisclaimerConfig",
     "RewriteLinksConfig",
+    "StorageConfig",
     "StoreConfig",
+    "NotificationConfig",
     "NotifyConfig",
-    "WhitelistConfig",
+    "ListConfig",
     "QuarantineConfig",
     "ActionConfig",
     "RuleConfig",
@@ -43,7 +45,7 @@ class BaseConfig:
         "properties": {
             "loglevel": {"type": "string", "default": "info"}}}
 
-    def __init__(self, config):
+    def __init__(self, config, *args, **kwargs):
         required = self.JSON_SCHEMA["required"]
         properties = self.JSON_SCHEMA["properties"]
         for p in properties.keys():
@@ -89,19 +91,21 @@ class BaseConfig:
         return self._config
 
 
-class WhitelistConfig(BaseConfig):
+class ListConfig(BaseConfig):
     JSON_SCHEMA = {
         "type": "object",
         "required": ["type"],
         "additionalProperties": True,
         "properties": {
-            "type": {"enum": ["db"]}},
+            "type": {"enum": ["db"]},
+            "name": {"type": "string"}},
         "if": {"properties": {"type": {"const": "db"}}},
         "then": {
             "required": ["connection", "table"],
             "additionalProperties": False,
             "properties": {
                 "type": {"type": "string"},
+                "name": {"type": "string"},
                 "connection": {"type": "string"},
                 "table": {"type": "string"}}}}
 
@@ -121,14 +125,16 @@ class ConditionsConfig(BaseConfig):
             "headers": {"type": "array",
                         "items": {"type": "string"}},
             "var": {"type": "string"},
-            "whitelist": {"type": "object"}}}
+            "list": {"type": "string"}}}
 
-    def __init__(self, config, rec=True):
+    def __init__(self, config, lists, rec=True):
         super().__init__(config)
-        if not rec:
-            return
-        if "whitelist" in self:
-            self["whitelist"] = WhitelistConfig(self["whitelist"])
+        if "list" in self:
+            lst = self["list"]
+            try:
+                self["list"] = lists[lst]
+            except KeyError:
+                raise RuntimeError(f"list '{lst}' not found in config")
 
 
 class AddHeaderConfig(BaseConfig):
@@ -184,7 +190,7 @@ class RewriteLinksConfig(BaseConfig):
             "repl": {"type": "string"}}}
 
 
-class StoreConfig(BaseConfig):
+class StorageConfig(BaseConfig):
     JSON_SCHEMA = {
         "type": "object",
         "required": ["type"],
@@ -204,7 +210,24 @@ class StoreConfig(BaseConfig):
                 "original": {"type": "boolean", "default": False}}}}
 
 
-class NotifyConfig(BaseConfig):
+class StoreConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["storage"],
+        "additionalProperties": False,
+        "properties": {
+            "storage": {"type": "string"}}}
+
+    def __init__(self, config, milter_config):
+        super().__init__(config)
+        storage = self["storage"]
+        try:
+            self["storage"] = milter_config["storages"][storage]
+        except KeyError:
+            raise RuntimeError(f"storage '{storage}' not found")
+
+
+class NotificationConfig(BaseConfig):
     JSON_SCHEMA = {
         "type": "object",
         "required": ["type"],
@@ -232,33 +255,62 @@ class NotifyConfig(BaseConfig):
                     "default": []}}}}
 
 
+class NotifyConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["notification"],
+        "additionalProperties": False,
+        "properties": {
+            "notification": {"type": "string"}}}
+
+    def __init__(self, config, milter_config):
+        super().__init__(config)
+        notification = self["notification"]
+        try:
+            self["notification"] = milter_config["notifications"][notification]
+        except KeyError:
+            raise RuntimeError(f"notification '{notification}' not found")
+
+
 class QuarantineConfig(BaseConfig):
     JSON_SCHEMA = {
         "type": "object",
-        "required": ["store", "smtp_host", "smtp_port"],
+        "required": ["storage", "smtp_host", "smtp_port"],
         "additionalProperties": False,
         "properties": {
             "name": {"type": "string"},
-            "notify": {"type": "object"},
+            "notification": {"type": "string"},
             "milter_action": {"type": "string"},
             "reject_reason": {"type": "string"},
-            "whitelist": {"type": "object"},
-            "store": {"type": "object"},
+            "allowlist": {"type": "string"},
+            "storage": {"type": "string"},
             "smtp_host": {"type": "string"},
             "smtp_port": {"type": "number"}}}
 
-    def __init__(self, config, rec=True):
+    def __init__(self, config, milter_config, rec=True):
         super().__init__(config)
+        storage = self["storage"]
+        try:
+            self["storage"] = milter_config["storages"][storage]
+        except KeyError:
+            raise RuntimeError(f"storage '{storage}' not found")
+        if "metadata" not in self["storage"]:
+            self["storage"]["metadata"] = True
+        if "notification" in self:
+            name = self["notification"]
+            try:
+                self["notification"] = milter_config["notifications"][name]
+            except KeyError:
+                raise RuntimeError(f"notification '{name}' not found")
+        if "allowlist" in self:
+            allowlist = self["allowlist"]
+            try:
+                self["allowlist"] = milter_config["lists"][allowlist]
+            except KeyError:
+                raise RuntimeError(f"list '{allowlist}' not found")
+
         if not rec:
             return
-        if "metadata" not in self["store"]:
-            self["store"]["metadata"] = True
-        self["store"] = StoreConfig(self["store"])
-        if "notify" in self:
-            self["notify"] = NotifyConfig(self["notify"])
-        if "whitelist" in self:
-            self["whitelist"] = ConditionsConfig(
-                {"whitelist": self["whitelist"]}, rec)
 
 
 class ActionConfig(BaseConfig):
@@ -284,13 +336,16 @@ class ActionConfig(BaseConfig):
             "type": {"enum": list(ACTION_TYPES.keys())},
             "options": {"type": "object"}}}
 
-    def __init__(self, config, rec=True):
+    def __init__(self, config, milter_config, rec=True):
         super().__init__(config)
         if not rec:
             return
+        lists = milter_config["lists"]
         if "conditions" in self:
-            self["conditions"] = ConditionsConfig(self["conditions"])
-        self["action"] = self.ACTION_TYPES[self["type"]](self["options"])
+            self["conditions"] = ConditionsConfig(self["conditions"], lists)
+
+        self["action"] = self.ACTION_TYPES[self["type"]](
+            self["options"], milter_config)
 
 
 class RuleConfig(BaseConfig):
@@ -305,20 +360,21 @@ class RuleConfig(BaseConfig):
             "conditions": {"type": "object"},
             "actions": {"type": "array"}}}
 
-    def __init__(self, config, rec=True):
+    def __init__(self, config, milter_config, rec=True):
         super().__init__(config)
         if not rec:
             return
+        lists = milter_config["lists"]
         if "conditions" in self:
-            self["conditions"] = ConditionsConfig(self["conditions"])
+            self["conditions"] = ConditionsConfig(self["conditions"], lists)
 
         actions = []
-        for idx, action in enumerate(self["actions"]):
+        for action in self["actions"]:
             if "loglevel" not in action:
                 action["loglevel"] = config["loglevel"]
             if "pretend" not in action:
                 action["pretend"] = config["pretend"]
-            actions.append(ActionConfig(action, rec))
+            actions.append(ActionConfig(action, milter_config, rec))
         self["actions"] = actions
 
 
@@ -340,23 +396,50 @@ class QuarantineMilterConfig(BaseConfig):
                                 "192.168.0.0/16"]},
             "loglevel": {"type": "string", "default": "info"},
             "pretend": {"type": "boolean", "default": False},
+            "lists": {
+                "type": "object",
+                "patternProperties": {"^(.+)$": {"type": "object"}},
+                "additionalProperties": False,
+                "default": {}},
+            "storages": {
+                "type": "object",
+                "patternProperties": {"^(.+)$": {"type": "object"}},
+                "additionalProperties": False,
+                "default": {}},
+            "notifications": {
+                "type": "object",
+                "patternProperties": {"^(.+)$": {"type": "object"}},
+                "additionalProperties": False,
+                "default": {}},
             "rules": {"type": "array"}}}
 
     def __init__(self, config, rec=True):
         super().__init__(config)
+        for name, cfg in self["lists"].items():
+            if "name" not in cfg:
+                cfg["name"] = name
+            self["lists"][name] = ListConfig(cfg)
+
+        for name, cfg in self["storages"].items():
+            self["storages"][name] = StorageConfig(cfg)
+
+        for name, cfg in self["notifications"].items():
+            self["notifications"][name] = NotificationConfig(cfg)
+
         if not rec:
             return
+
         rules = []
-        for idx, rule in enumerate(self["rules"]):
+        for rule in self["rules"]:
             if "loglevel" not in rule:
                 rule["loglevel"] = config["loglevel"]
             if "pretend" not in rule:
                 rule["pretend"] = config["pretend"]
-            rules.append(RuleConfig(rule, rec))
+            rules.append(RuleConfig(rule, self, rec))
         self["rules"] = rules
 
 
-def get_milter_config(cfgfile, raw=False):
+def get_milter_config(cfgfile, rec=True):
     try:
         with open(cfgfile, "r") as fh:
             # remove lines with leading # (comments), they
@@ -371,6 +454,4 @@ def get_milter_config(cfgfile, raw=False):
         cfg_text = [f"{n+1}: {l}" for n, l in enumerate(cfg.splitlines())]
         msg = "\n".join(cfg_text)
         raise RuntimeError(f"{e}\n{msg}")
-    if raw:
-        return cfg
-    return QuarantineMilterConfig(cfg)
+    return QuarantineMilterConfig(cfg, rec)

pyquarantine/lists.py

@@ -13,8 +13,8 @@
 #
 
 __all__ = [
-    "DatabaseWhitelist",
-    "WhitelistBase"]
+    "DatabaseList",
+    "ListBase"]
 
 import logging
 import peewee
@@ -24,8 +24,8 @@ from datetime import datetime
 from playhouse.db_url import connect
 
 
-class WhitelistBase:
-    "Whitelist base class"
+class ListBase:
+    "List base class"
     def __init__(self, cfg, debug):
         self.cfg = cfg
         self.logger = logging.getLogger(cfg["name"])
@@ -47,15 +47,15 @@ class WhitelistBase:
         return self.batv_regex.sub(r"\g<LEFT_PART>@", addr, count=1)
 
     def check(self, mailfrom, recipient):
-        "Check if mailfrom/recipient combination is whitelisted."
+        "Check if mailfrom/recipient combination is listed."
         return
 
     def find(self, mailfrom=None, recipients=None, older_than=None):
-        "Find whitelist entries."
+        "Find list entries."
         return
 
     def add(self, mailfrom, recipient, comment, permanent):
-        "Add entry to whitelist."
+        "Add entry to list."
         # check if mailfrom and recipient are valid
         if not self.valid_entry_regex.match(mailfrom):
             raise RuntimeError("invalid from address")
@@ -63,12 +63,12 @@ class WhitelistBase:
             raise RuntimeError("invalid recipient")
         return
 
-    def delete(self, whitelist_id):
-        "Delete entry from whitelist."
+    def delete(self, list_id):
+        "Delete entry from list."
         return
 
 
-class WhitelistModel(peewee.Model):
+class DatabaseListModel(peewee.Model):
     mailfrom = peewee.CharField()
     recipient = peewee.CharField()
     created = peewee.DateTimeField(default=datetime.now)
@@ -84,9 +84,9 @@ class Meta:
     )
 
 
-class DatabaseWhitelist(WhitelistBase):
-    "Whitelist class to store whitelist in a database"
-    whitelist_type = "db"
+class DatabaseList(ListBase):
+    "List class to store lists in a database"
+    list_type = "db"
     _db_connections = {}
     _db_tables = {}
 
@@ -96,8 +96,8 @@ class DatabaseWhitelist(WhitelistBase):
         tablename = cfg["table"]
         connection_string = cfg["connection"]
 
-        if connection_string in DatabaseWhitelist._db_connections.keys():
-            db = DatabaseWhitelist._db_connections[connection_string]
+        if connection_string in DatabaseList._db_connections.keys():
+            db = DatabaseList._db_connections[connection_string]
         else:
             try:
                 # connect to database
@@ -112,22 +112,22 @@ class DatabaseWhitelist(WhitelistBase):
                 raise RuntimeError(
                     f"unable to connect to database: {e}")
 
-            DatabaseWhitelist._db_connections[connection_string] = db
+            DatabaseList._db_connections[connection_string] = db
 
         # generate model meta class
         self.meta = Meta
         self.meta.database = db
         self.meta.table_name = tablename
         self.model = type(
-            f"WhitelistModel_{self.cfg['name']}",
-            (WhitelistModel,),
+            f"DatabaseListModel_{self.cfg['name']}",
+            (DatabaseListModel,),
             {"Meta": self.meta})
 
-        if connection_string not in DatabaseWhitelist._db_tables.keys():
-            DatabaseWhitelist._db_tables[connection_string] = []
+        if connection_string not in DatabaseList._db_tables.keys():
+            DatabaseList._db_tables[connection_string] = []
 
-        if tablename not in DatabaseWhitelist._db_tables[connection_string]:
-            DatabaseWhitelist._db_tables[connection_string].append(tablename)
+        if tablename not in DatabaseList._db_tables[connection_string]:
+            DatabaseList._db_tables[connection_string].append(tablename)
             try:
                 db.create_tables([self.model])
             except Exception as e:
@@ -139,7 +139,7 @@ class DatabaseWhitelist(WhitelistBase):
         for arg in ("connection", "table"):
             if arg in self.cfg:
                 cfg.append(f"{arg}={self.cfg[arg]}")
-        return "DatabaseWhitelist(" + ", ".join(cfg) + ")"
+        return "DatabaseList(" + ", ".join(cfg) + ")"
 
     def _entry_to_dict(self, entry):
         result = {}
@@ -164,14 +164,14 @@ class DatabaseWhitelist(WhitelistBase):
         return value
 
     def check(self, mailfrom, recipient, logger):
-        # check if mailfrom/recipient combination is whitelisted
+        # check if mailfrom/recipient combination is listed
         super().check(mailfrom, recipient)
         mailfrom = self.remove_batv(mailfrom)
         recipient = self.remove_batv(recipient)
 
         # generate list of possible mailfroms
         logger.debug(
-            f"query database for whitelist entries from <{mailfrom}> "
+            f"query database for list entries from <{mailfrom}> "
             f"to <{recipient}>")
         mailfroms = [""]
         if "@" in mailfrom and not mailfrom.startswith("@"):
@@ -196,7 +196,7 @@ class DatabaseWhitelist(WhitelistBase):
             raise RuntimeError(f"unable to query database: {e}")
 
         if not entries:
-            # no whitelist entry found
+            # no list entry found
             return {}
 
         if len(entries) > 1:
@@ -213,7 +213,7 @@ class DatabaseWhitelist(WhitelistBase):
         return result
 
     def find(self, mailfrom=None, recipients=None, older_than=None):
-        "Find whitelist entries."
+        "Find list entries."
         super().find(mailfrom, recipients, older_than)
 
         if isinstance(mailfrom, str):
@@ -244,7 +244,7 @@ class DatabaseWhitelist(WhitelistBase):
         return entries
 
     def add(self, mailfrom, recipient, comment, permanent):
-        "Add entry to whitelist."
+        "Add entry to list."
         super().add(
             mailfrom,
             recipient,
@@ -263,16 +263,16 @@ class DatabaseWhitelist(WhitelistBase):
         except Exception as e:
             raise RuntimeError(f"unable to add entry to database: {e}")
 
-    def delete(self, whitelist_id):
-        "Delete entry from whitelist."
-        super().delete(whitelist_id)
+    def delete(self, list_id):
+        "Delete entry from list."
+        super().delete(list_id)
 
         try:
-            query = self.model.delete().where(self.model.id == whitelist_id)
+            query = self.model.delete().where(self.model.id == list_id)
             deleted = query.execute()
         except Exception as e:
             raise RuntimeError(
                 f"unable to delete entry from database: {e}")
 
         if deleted == 0:
-            raise RuntimeError("invalid whitelist id")
+            raise RuntimeError("invalid list id")

pyquarantine/notify.py

@@ -203,14 +203,14 @@ class EMailNotification(BaseNotification):
                 f"and its content")
             element.extract()
 
-        # remove not whitelisted elements, but keep their content
+        # remove not allowed elements, but keep their content
         for element in soup.find_all(True):
             if element.name not in EMailNotification._good_tags:
                 logger.debug(
                     f"removing tag '{element.name}', keep its content")
                 element.replaceWithChildren()
 
-        # remove not whitelisted attributes
+        # remove not allowed attributes
         for element in soup.find_all(True):
             for attribute in list(element.attrs.keys()):
                 if attribute not in EMailNotification._good_attributes:
@@ -223,7 +223,7 @@ class EMailNotification(BaseNotification):
                         logger.debug(
                             f"removing attribute '{attribute}' "
                             f"from tag '{element.name}'")
-                        del(element.attrs[attribute])
+                        del element.attrs[attribute]
         return soup
 
     def notify(self, msg, qid, mailfrom, recipients, logger,
@@ -317,6 +317,7 @@ class EMailNotification(BaseNotification):
     def execute(self, milter, logger):
         super().execute(milter, logger)
 
+        logger.info(f"send notification(s) to {milter.msginfo['rcpts']}")
         self.notify(msg=milter.msg, qid=milter.qid,
                     mailfrom=milter.msginfo["mailfrom"],
                     recipients=milter.msginfo["rcpts"],
@@ -333,16 +334,16 @@ class Notify:
         self.logger = logging.getLogger(cfg["name"])
         self.logger.setLevel(cfg.get_loglevel(debug))
 
-        nodification_type = cfg["options"]["type"]
-        del cfg["options"]["type"]
-        cfg["options"]["pretend"] = cfg["pretend"]
-        self._notification = self.NOTIFICATION_TYPES[nodification_type](
-            **cfg["options"])
+        nodification_type = cfg["options"]["notification"]["type"]
+        del cfg["options"]["notification"]["type"]
+
+        ncfg = cfg["options"]["notification"]
+        self._notification = self.NOTIFICATION_TYPES[nodification_type](**ncfg)
         self._headersonly = self._notification._headersonly
 
     def __str__(self):
         cfg = []
-        for key, value in self.cfg["options"].items():
+        for key, value in self.cfg["options"]["notification"].items():
             cfg.append(f"{key}={value}")
         class_name = type(self._notification).__name__
         return f"{class_name}(" + ", ".join(cfg) + ")"

pyquarantine/rule.py

@@ -46,8 +46,9 @@ class Rule:
         actions = []
         for action in self.actions:
             actions.append(str(action))
-        cfg.append("actions=[" + ", ".join(actions) + "]")
-        return "Rule(" + ", ".join(cfg) + ")"
+        cfg.append("actions=[\n        " +
+                   ",\n        ".join(actions) + "\n    ]")
+        return "Rule(\n    " + ",\n    ".join(cfg) + "\n)"
 
     def execute(self, milter):
         """Execute all actions of this rule."""

pyquarantine/run.py

@@ -135,7 +135,7 @@ def main():
     sysloghandler = logging.handlers.SysLogHandler(
         address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_MAIL)
     sysloghandler.setFormatter(
-        logging.Formatter("pyquarantine: %(message)s"))
+        logging.Formatter(f"{name}[%(process)d]: %(message)s"))
     root_logger.addHandler(sysloghandler)
 
     logger.info("milter starting")

pyquarantine/storage.py

@@ -31,8 +31,7 @@ from time import gmtime
 
 from pyquarantine import mailer
 from pyquarantine.base import CustomLogger, MilterMessage
-from pyquarantine.conditions import Conditions
-from pyquarantine.config import ActionConfig
+from pyquarantine.lists import DatabaseList
 from pyquarantine.notify import Notify
 
 
@@ -334,7 +333,7 @@ class FileMailStorage(BaseMailStorage):
 
         metafile, _ = self._get_file_paths(storage_id)
 
-        if type(recipients) == str:
+        if isinstance(recipients, str):
             recipients = [recipients]
 
         for recipient in recipients:
@@ -375,16 +374,16 @@ class Store:
         self.logger = logging.getLogger(cfg["name"])
         self.logger.setLevel(cfg.get_loglevel(debug))
 
-        storage_type = cfg["options"]["type"]
-        del cfg["options"]["type"]
-        cfg["options"]["pretend"] = cfg["pretend"]
-        self._storage = self.STORAGE_TYPES[storage_type](
-            **cfg["options"])
+        storage_type = cfg["options"]["storage"]["type"]
+        del cfg["options"]["storage"]["type"]
+
+        scfg = cfg["options"]["storage"]
+        self._storage = self.STORAGE_TYPES[storage_type](**scfg)
         self._headersonly = self._storage._headersonly
 
     def __str__(self):
         cfg = []
-        for key, value in self.cfg["options"].items():
+        for key, value in self.cfg["options"]["storage"].items():
             cfg.append(f"{key}={value}")
         class_name = type(self._storage).__name__
         return f"{class_name}(" + ", ".join(cfg) + ")"
@@ -407,36 +406,24 @@ class Quarantine:
         self.logger = logging.getLogger(cfg["name"])
         self.logger.setLevel(cfg.get_loglevel(debug))
 
-        storage_cfg = ActionConfig({
-            "name": cfg["name"],
-            "loglevel": cfg["loglevel"],
-            "pretend": cfg["pretend"],
-            "type": "store",
-            "options": cfg["options"]["store"].get_config()})
-        self._storage = Store(storage_cfg, local_addrs, debug)
+        self._storage = Store(cfg, local_addrs, debug)
 
         self.smtp_host = cfg["options"]["smtp_host"]
         self.smtp_port = cfg["options"]["smtp_port"]
 
         self._notification = None
-        if "notify" in cfg["options"]:
-            notify_cfg = ActionConfig({
-                "name": cfg["name"],
-                "loglevel": cfg["loglevel"],
-                "pretend": cfg["pretend"],
-                "type": "notify",
-                "options": cfg["options"]["notify"].get_config()})
-            self._notification = Notify(notify_cfg, local_addrs, debug)
+        if "notification" in cfg["options"]:
+            self._notification = Notify(cfg, local_addrs, debug)
 
-        self._whitelist = None
-        if "whitelist" in cfg["options"]:
-            whitelist_cfg = cfg["options"]["whitelist"]
-            whitelist_cfg["name"] = cfg["name"]
-            whitelist_cfg["loglevel"] = cfg["loglevel"]
-            self._whitelist = Conditions(
-                whitelist_cfg,
-                local_addrs=[],
-                debug=debug)
+        self._allowlist = None
+        if "allowlist" in cfg["options"]:
+            allowlist = cfg["options"]["allowlist"]
+            if allowlist["type"] == "db":
+                allowlist["name"] = f"{cfg['name']}: allowlist"
+                allowlist["loglevel"] = cfg["loglevel"]
+                self._allowlist = DatabaseList(allowlist, debug)
+            else:
+                raise RuntimeError("invalid allowlist type")
 
         self._milter_action = None
         if "milter_action" in cfg["options"]:
@@ -456,8 +443,8 @@ class Quarantine:
         cfg.append(f"store={str(self._storage)}")
         if self._notification is not None:
             cfg.append(f"notify={str(self._notification)}")
-        if self._whitelist is not None:
-            cfg.append(f"whitelist={str(self._whitelist)}")
+        if self._allowlist is not None:
+            cfg.append(f"allowlist={str(self._allowlist)}")
         for key in ["milter_action", "reject_reason"]:
             if key not in self.cfg["options"]:
                 continue
@@ -481,10 +468,8 @@ class Quarantine:
         return self._notification.get_notification()
 
     @property
-    def whitelist(self):
-        if self._whitelist is None:
-            return None
-        return self._whitelist.get_whitelist()
+    def allowlist(self):
+        return self._allowlist
 
     @property
     def milter_action(self):
@@ -512,7 +497,7 @@ class Quarantine:
 
     def release(self, storage_id, recipients=None):
         metadata, msg = self.storage.get_mail(storage_id)
-        if recipients and type(recipients) == str:
+        if recipients and isinstance(recipients, str):
             recipients = [recipients]
         else:
             recipients = metadata["recipients"]
@@ -535,31 +520,51 @@ class Quarantine:
 
         return recipients
 
+    def copy(self, storage_id, recipient):
+        metadata, msg = self.storage.get_mail(storage_id)
+        try:
+            mailer.smtp_send(
+                self.smtp_host,
+                self.smtp_port,
+                metadata["mailfrom"],
+                recipient,
+                msg.as_string())
+
+        except Exception as e:
+            raise RuntimeError(
+                f"error while sending message to '{recipient}': {e}")
+
     def execute(self, milter):
         logger = CustomLogger(
             self.logger, {"name": self.cfg["name"], "qid": milter.qid})
+
         rcpts = milter.msginfo["rcpts"]
-        wl_rcpts = []
-        if self._whitelist:
-            wl_rcpts = self._whitelist.get_wl_rcpts(
-                milter.msginfo["mailfrom"], rcpts, logger)
-            if wl_rcpts:
-                logger.info(f"whitelisted recipients: {wl_rcpts}")
-            rcpts = [rcpt for rcpt in rcpts if rcpt not in wl_rcpts]
+        allowed_rcpts = []
+        if self._allowlist:
+            allowed_rcpts = []
+            for rcpt in rcpts:
+                if self._allowlist.check(
+                        milter.msginfo["mailfrom"], rcpt, logger):
+                    allowed_rcpts.append(rcpt)
+            if allowed_rcpts:
+                logger.info(f"allowed recipients: {allowed_rcpts}")
+            rcpts = [rcpt for rcpt in rcpts if rcpt not in allowed_rcpts]
             if not rcpts:
-                # all recipients whitelisted
+                # all recipients allowed
                 return
             milter.msginfo["rcpts"] = rcpts.copy()
 
         if self._milter_action in ["REJECT", "DISCARD"]:
             logger.info(f"quarantine message for {rcpts}")
+        else:
+            logger.info(f"save message for {rcpts}")
 
         self._storage.execute(milter)
 
         if self._notification is not None:
             self._notification.execute(milter)
 
-        milter.msginfo["rcpts"].extend(wl_rcpts)
+        milter.msginfo["rcpts"].extend(allowed_rcpts)
 
         if self._milter_action is not None:
             milter.delrcpt(rcpts)

setup.py

@@ -18,7 +18,7 @@ setup(name = "pyquarantine",
         #   3 - Alpha
         #   4 - Beta
         #   5 - Production/Stable
-        "Development Status :: 4 - Beta",
+        "Development Status :: 5 - Production/Stable",
         "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
         "Operating System :: OS Independent",
         "Programming Language :: Python",
@@ -48,6 +48,6 @@ setup(name = "pyquarantine",
             ]
         )
     ],
-    install_requires = ["pymilter", "jsonschema", "netaddr", "beautifulsoup4[lxml]", "peewee"],
-    python_requires = ">=3.8"
+    install_requires = ["pymilter >= 1.5", "jsonschema", "netaddr", "beautifulsoup4[lxml]", "peewee"],
+    python_requires = ">=3.10"
 )