switch from cgi.escape to html.escape

Signed-off-by: Thomas Oettli <spacefreak@noop.ch>

MANIFEST.in

@@ -1,3 +1,3 @@
 include LICENSE README.md
-recursive-include docs *
+recursive-include pyquarantine/docs *
-recursive-include misc *
+recursive-include pyquarantine/misc *

README.md

@@ -8,7 +8,7 @@ Each quarantine can be configured with a quarantine type, notification type, whi
 
 Addionally, pyquarantine-milter provides a sanitized, harmless version of the text parts of e-mails, which can be embedded in e-mail notifications. This makes it easier for users to decide, if a match is a false-positive or not. If a matching quarantine provides a quarantine ID of the original e-mail, it is also available as a template variable. This is useful if you want to add links to a webservice to notification e-mails, to give your users the ability to release e-mails or whitelist the from-address for example. The webservice then releases the e-mail from the quarantine.
 
-The project is currently in alpha status, but will soon be used in a productive enterprise environment and possibly existing bugs will be fixed.
+The project is currently in beta status, but it is already used in a productive enterprise environment which processes about a million e-mails per month.
 
 ## Requirements
 * pymilter <https://pythonhosted.org/pymilter/>
@@ -36,8 +36,8 @@ The following configuration options are mandatory in the global section:
 The following configuration options are mandatory in each quarantine section:
 * **regex**  
   Case insensitive regular expression to filter e-mail headers.
-* **quarantine_type**  
+* **storage_type**  
-  One of the quarantine-types described below.
+  One of the storage types described below.
 * **action**  
   One of the actions described below.
 * **notification_type**  
@@ -45,18 +45,18 @@ The following configuration options are mandatory in each quarantine section:
 * **whitelist_type**  
   One of the whitelist types described below.
 * **smtp_host**  
-  SMTP host to inject original e-mails. This is needed if not all recipients of an e-mail are whitelisted
+  SMTP host used to release original e-mails from the quarantine.
 * **smtp_port**  
   SMTP port
 
 The following configuration options are optional in each quarantine section:
-* **ignore_hosts**  
+* **host_whitelist**  
   Comma-separated list of host and network addresses to be ignored by this quarantine.
 * **reject_reason**  
   Reason to return to the client if action is set to reject.
 
 
-### Quarantine types
+### Storage types
 * **NONE**  
   Original e-mails scrapped, sent to nirvana, black-holed or however you want to call it.
 
@@ -64,7 +64,7 @@ The following configuration options are optional in each quarantine section:
   Original e-mails are stored on the filesystem with a unique filename. The filename is available as a
   template variable used in notifiaction templates.  
   The following configuration options are mandatory for this quarantine type:
-  * **quarantine_directory**  
+  * **storage_directory**  
     The directory in which quarantined e-mails are stored.
 
 
@@ -77,17 +77,23 @@ The following configuration options are optional in each quarantine section:
 
   The following template variables are available:
   * **{EMAIL_ENVELOPE_FROM}**  
-    E-mail from-address received by the milter.
+    E-mail sender address received by the milter.
+  * **{EMAIL_ENVELOPE_FROM_URL}**  
+    Like EMAIL_ENVELOPE_FROM, but URL encoded
   * **{EMAIL_FROM}**  
-    Value of the from header of the original e-mail.
+    Value of the FROM header of the original e-mail.
-  * **{EMAIL_TO}**  
+  * **{EMAIL_ENVELOPE_TO}**  
     E-mail recipient address of this notification.
+  * **{EMAIL_ENVELOPE_TO_URL}**  
+    Like EMAIL_ENVELOPE_TO, but URL encoded
+  * **{EMAIL_TO}**  
+    Value of the TO header of the original e-mail.
   * **{EMAIL_SUBJECT}**  
     Configured e-mail subject.
   * **{EMAIL_QUARANTINE_ID}**  
     Quarantine-ID of the original e-mail if available, empty otherwise.
   * **{EMAIL_HTML_TEXT}**  
-    Sanitized version of the e-mail text part of the original e-mail. Only harmless HTML tags and attributes are included. Images are replaced with the image set by notification_email_replacement_img option.
+    Sanitized version of the e-mail text part of the original e-mail. Only harmless HTML tags and attributes are included. Images are optionally stripped or replaced with the image set by notification_email_replacement_img option.
 
   Some template variables are only available if the regex of the matching quarantine contains subgroups or named subgroups (python syntax). This is useful to include information (e.g. virus names, spam points, ...) of the matching header within the notification.  
   The following dynamic template variables are available:
@@ -97,20 +103,31 @@ The following configuration options are optional in each quarantine section:
     Content of a named subgroup, 'subgroup_name' will be replaced by its name.
 
   The following configuration options are mandatory for this notification type:
+  * **notification_email_smtp_host**  
+    SMTP host used to send notification e-mails.
+  * **notification_email_smtp_port**  
+    SMTP port.
   * **notification_email_envelope_from**  
     Notification e-mail envelope from-address.
   * **notification_email_from**  
-    Value of the notification e-mail from header. Every e-mail template variable described above is usable.
+    Value of the notification e-mail from header. Optionally, you may use the EMAIL_FROM template variable described above.
   * **notification_email_subject**  
-    Notification e-mail subject. All e-mail template variable described above is usable.
+    Notification e-mail subject. Optionally, you may use the EMAIL_SUBJECT template variable described above.
   * **notification_email_template**  
     Path to the notification e-mail template. It is hold in memory during runtime.
-  * **notification_email_replacement_img**  
-    Path to the image to replace images in e-mails. It is hold in memory during runtime. Leave it empty to disable.
   * **notification_email_embedded_imgs**  
     Comma-separated list of images to embed into the notification e-mail. The Content-ID of each image will be set to the filename, so you can reference it from the e-mail template. All images are hold in memory during runtime.  
     Leave empty to disable.
 
+  The following configuration options are optional for this notification type:
+  * **notification_email_strip_images**  
+    Enable to strip images from e-mails. This option superseeds notification_email_replacement_img.
+  * **notification_email_replacement_img**  
+    Path to an image to replace images in e-mails. It is hold in memory during runtime.
+  * **notification_email_parser_lib**  
+    HTML parser library used to parse text part of emails.
+
+
 ### Actions
 Every quarantine responds with a milter-action if an e-mail header matches the configured regular expression. Please think carefully what you set here or your MTA will do something you do not want.  
 The following actions are available:

pyquarantine/__init__.py

@@ -12,78 +12,361 @@
 # along with PyQuarantineMilter.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-__all__ = ["QuarantineMilter", "generate_milter_config", "reload_config", "mailer", "notifications", "run", "quarantines", "whitelists"]
-name = "pyquarantine"
-
 import Milter
 import configparser
 import logging
 import os
 import re
-import sys
+import encodings
 
 from Milter.utils import parse_addr
 from collections import defaultdict
+from email.charset import Charset
+from email.header import Header, decode_header
 from io import BytesIO
 from itertools import groupby
-from netaddr import IPAddress, IPNetwork
+from netaddr import IPAddress, IPNetwork, AddrFormatError
-from pyquarantine import quarantines
+from pyquarantine import mailer
 from pyquarantine import notifications
+from pyquarantine import storages
 from pyquarantine import whitelists
 
+__all__ = [
+    "make_header",
+    "Quarantine",
+    "QuarantineMilter",
+    "setup_milter",
+    "reload_config",
+    "cli",
+    "mailer",
+    "notifications",
+    "storages",
+    "run",
+    "whitelists"]
+
+__version__ = "1.0.8"
+
+
+################################################
+#  add charset alias for windows-874 encoding  #
+################################################
+
+aliases = encodings.aliases.aliases
+
+for alias in ["windows-874", "windows_874"]:
+    if alias not in aliases:
+        aliases[alias] = "cp874"
+
+setattr(encodings.aliases, "aliases", aliases)
+
+################################################
+
+
+def make_header(decoded_seq, maxlinelen=None, header_name=None,
+                continuation_ws=' ', errors='strict'):
+    """Create a Header from a sequence of pairs as returned by decode_header()
+
+    decode_header() takes a header value string and returns a sequence of
+    pairs of the format (decoded_string, charset) where charset is the string
+    name of the character set.
+
+    This function takes one of those sequence of pairs and returns a Header
+    instance.  Optional maxlinelen, header_name, and continuation_ws are as in
+    the Header constructor.
+    """
+    h = Header(maxlinelen=maxlinelen, header_name=header_name,
+               continuation_ws=continuation_ws)
+    for s, charset in decoded_seq:
+        # None means us-ascii but we can simply pass it on to h.append()
+        if charset is not None and not isinstance(charset, Charset):
+            charset = Charset(charset)
+        h.append(s, charset, errors=errors)
+    return h
+
+
+class Quarantine(object):
+    """Quarantine class suitable for QuarantineMilter
+
+    The class holds all the objects and functions needed
+    for QuarantineMilter quarantine.
+
+    """
+
+    # list of possible actions
+    _actions = {
+        "ACCEPT": Milter.ACCEPT,
+        "REJECT": Milter.REJECT,
+        "DISCARD": Milter.DISCARD}
+
+    def __init__(self, name, index=0, regex=None, storage=None, whitelist=None,
+                 host_whitelist=[], notification=None, action="ACCEPT",
+                 reject_reason=None):
+        self.logger = logging.getLogger(__name__)
+        self.name = name
+        self.index = index
+        self.regex = regex
+        if regex:
+            self.re = re.compile(
+                regex, re.MULTILINE + re.DOTALL + re.IGNORECASE)
+        self.storage = storage
+        self.whitelist = whitelist
+        self.host_whitelist = host_whitelist
+        self.notification = notification
+        action = action.upper()
+        assert action in self._actions
+        self.action = action
+        self.milter_action = self._actions[action]
+        self.reject_reason = reject_reason
+
+    def setup_from_cfg(self, global_cfg, cfg, test=False):
+        defaults = {
+            "action": "accept",
+            "reject_reason": "Message rejected",
+            "storage_type": "none",
+            "notification_type": "none",
+            "whitelist_type": "none",
+            "host_whitelist": ""
+        }
+        # check config
+        for opt in ["regex", "smtp_host", "smtp_port"] + list(defaults.keys()):
+            if opt in cfg:
+                continue
+            if opt in global_cfg:
+                cfg[opt] = global_cfg[opt]
+            elif opt in defaults:
+                cfg[opt] = defaults[opt]
+            else:
+                raise RuntimeError(
+                    f"mandatory option '{opt}' not present in "
+                    f"config section '{self.name}' or 'global'")
+
+        # pre-compile regex
+        self.logger.debug(
+            f"{self.name}: compiling regex '{cfg['regex']}'")
+        self.re = re.compile(
+            cfg["regex"], re.MULTILINE + re.DOTALL + re.IGNORECASE)
+
+        self.smtp_host = cfg["smtp_host"]
+        self.smtp_port = cfg["smtp_port"]
+
+        # create storage instance
+        storage_type = cfg["storage_type"].lower()
+        if storage_type in storages.TYPES:
+            self.logger.debug(
+                f"{self.name}: initializing storage "
+                f"type '{storage_type.upper()}'")
+            self.storage = storages.TYPES[storage_type](
+                self.name, global_cfg, cfg, test)
+        elif storage_type == "none":
+            self.logger.debug(f"{self.name}: storage is NONE")
+            self.storage = None
+        else:
+            raise RuntimeError(
+                f"{self.name}: unknown storage type '{storage_type}'")
+
+        # create whitelist instance
+        whitelist_type = cfg["whitelist_type"].lower()
+        if whitelist_type in whitelists.TYPES:
+            self.logger.debug(
+                f"{self.name}: initializing whitelist "
+                f"type '{whitelist_type.upper()}'")
+            self.whitelist = whitelists.TYPES[whitelist_type](
+               self.name, global_cfg, cfg, test)
+        elif whitelist_type == "none":
+            self.logger.debug(f"{self.name}: whitelist is NONE")
+            self.whitelist = None
+        else:
+            raise RuntimeError(
+                f"{self.name}: unknown whitelist type '{whitelist_type}'")
+
+        # create notification instance
+        notification_type = cfg["notification_type"].lower()
+        if notification_type in notifications.TYPES:
+            self.logger.debug(
+                f"{self.name}: initializing notification "
+                f"type '{notification_type.upper()}'")
+            self.notification = notifications.TYPES[notification_type](
+                self.name, global_cfg, cfg, test)
+        elif notification_type == "none":
+            self.logger.debug(f"{self.name}: notification is NONE")
+            self.notification = None
+        else:
+            raise RuntimeError(
+                f"{self.name}: unknown notification "
+                f"type '{notification_type}'")
+
+        # determining milter action for this quarantine
+        action = cfg["action"].upper()
+        if action in self._actions:
+            self.logger.debug(f"{self.name}: action is {action}")
+            self.action = action
+            self.milter_action = self._actions[action]
+        else:
+            raise RuntimeError(
+                f"{self.name}: unknown action '{action}'")
+
+        self.reject_reason = cfg["reject_reason"]
+
+        # create host/network whitelist
+        self.host_whitelist = []
+        host_whitelist = set([p.strip()
+                             for p in cfg["host_whitelist"].split(",") if p])
+        for host in host_whitelist:
+            if not host:
+                continue
+            # parse network notation
+            try:
+                net = IPNetwork(host)
+            except AddrFormatError as e:
+                raise RuntimeError(
+                    f"{self.name}: error parsing host_whitelist: {e}")
+            else:
+                self.host_whitelist.append(net)
+        if self.host_whitelist:
+            whitelist = ", ".join([str(ip) for ip in host_whitelist])
+            self.logger.debug(
+                f"{self.name}: host whitelist: {whitelist}")
+
+    def notify(self, storage_id, recipient=None, synchronous=True):
+        "Notify recipient about email in storage."
+        if not self.storage:
+            raise RuntimeError(
+                "storage type is set to None, unable to send notification")
+
+        if not self.notification:
+            raise RuntimeError(
+                "notification type is set to None, "
+                "unable to send notification")
+
+        fp, metadata = self.storage.get_mail(storage_id)
+
+        if recipient is not None:
+            if recipient not in metadata["recipients"]:
+                raise RuntimeError(f"invalid recipient '{recipient}'")
+            recipients = [recipient]
+        else:
+            recipients = metadata["recipients"]
+
+        self.notification.notify(
+            metadata["queue_id"], storage_id, metadata["mailfrom"],
+            recipients, metadata["headers"], fp,
+            metadata["subgroups"], metadata["named_subgroups"],
+            synchronous)
+        fp.close()
+
+    def release(self, storage_id, recipients=None):
+        "Release email from storage."
+        if not self.storage:
+            raise RuntimeError(
+                "storage type is set to None, unable to release email")
+
+        fp, metadata = self.storage.get_mail(storage_id)
+        try:
+            mail = fp.read()
+            fp.close()
+        except IOError as e:
+            raise RuntimeError(f"unable to read data file: {e}")
+
+        if recipients and type(recipients) == str:
+            recipients = [recipients]
+        else:
+            recipients = metadata["recipients"]
+
+        for recipient in recipients:
+            if recipient not in metadata["recipients"]:
+                raise RuntimeError(f"invalid recipient '{recipient}'")
+
+            try:
+                mailer.smtp_send(
+                    self.smtp_host,
+                    self.smtp_port,
+                    metadata["mailfrom"],
+                    recipient,
+                    mail)
+            except Exception as e:
+                raise RuntimeError(
+                    f"error while sending email to '{recipient}': {e}")
+            self.storage.delete(storage_id, recipient)
+
+    def get_storage(self):
+        return self.storage
+
+    def get_notification(self):
+        return self.notification
+
+    def get_whitelist(self):
+        return self.whitelist
+
+    def host_in_whitelist(self, hostaddr):
+        ip = IPAddress(hostaddr[0])
+        for entry in self.host_whitelist:
+            if ip in entry:
+                return True
+        return False
+
+    def match(self, header):
+        return self.re.search(header)
+
 
 class QuarantineMilter(Milter.Base):
     """QuarantineMilter based on Milter.Base to implement milter communication
 
-    The class variable config needs to be filled with the result of the generate_milter_config function.
+    The class variable quarantines needs to be filled by
+    runng the setup_milter function.
 
     """
-    config = None
+    quarantines = []
-    global_config = None
+    preferred_action = "first"
 
     # list of default config files
-    _config_files = ["/etc/pyquarantine/pyquarantine.conf", os.path.expanduser('~/pyquarantine.conf'), "pyquarantine.conf"]
+    _cfg_files = [
-    # list of possible actions
+        "/etc/pyquarantine/pyquarantine.conf",
-    _actions = {"ACCEPT": Milter.ACCEPT, "REJECT": Milter.REJECT, "DISCARD": Milter.DISCARD}
+        os.path.expanduser('~/pyquarantine.conf'),
+        "pyquarantine.conf"]
 
     def __init__(self):
         self.logger = logging.getLogger(__name__)
-        # save config, it must not change during runtime
+        # save runtime config, it must not change during runtime
-        self.global_config = QuarantineMilter.global_config
+        self.quarantines = QuarantineMilter.quarantines
-        self.config = QuarantineMilter.config
 
     def _get_preferred_quarantine(self):
-        matching_quarantines = [q for q in self.recipients_quarantines.values() if q]
+        matching_quarantines = [
-        if self.global_config["preferred_quarantine_action"] == "first":
+            q for q in self.rcpts_quarantines.values() if q]
-            quarantine = sorted(matching_quarantines, key=lambda x: x["index"])[0]
+        if self.preferred_action == "first":
+            quarantine = sorted(
+                matching_quarantines,
+                key=lambda q: q.index)[0]
         else:
-            quarantine = sorted(matching_quarantines, key=lambda x: x["index"], reverse=True)[0]
+            quarantine = sorted(
+                matching_quarantines,
+                key=lambda q: q.index,
+                reverse=True)[0]
         return quarantine
 
     @staticmethod
-    def get_configfiles():
+    def get_cfg_files():
-        return QuarantineMilter._config_files
+        return QuarantineMilter._cfg_files
 
     @staticmethod
-    def get_actions():
+    def set_cfg_files(cfg_files):
-        return QuarantineMilter._actions
+        QuarantineMilter._cfg_files = cfg_files
 
-    @staticmethod
+    def connect(self, hostname, family, hostaddr):
-    def set_configfiles(config_files):
+        self.hostaddr = hostaddr
-        QuarantineMilter._config_files = config_files
+        self.logger.debug(
-
+            f"accepted milter connection from {hostaddr[0]} "
-    def connect(self, IPname, family, hostaddr):
+            f"port {hostaddr[1]}")
-        self.logger.debug("accepted milter connection from {} port {}".format(*hostaddr))
+        for quarantine in self.quarantines.copy():
-        ip = IPAddress(hostaddr[0])
+            if quarantine.host_in_whitelist(hostaddr):
-        for quarantine in self.config.copy():
+                self.logger.debug(
-            for ignore in quarantine["ignore_hosts_list"]:
+                    f"host {hostaddr[0]} is in whitelist of "
-                if ip in ignore:
+                    f"quarantine {quarantine.name}")
-                    self.logger.debug("host {} is ignored by quarantine {}".format(hostaddr[0], quarantine["name"]))
+                self.quarantines.remove(quarantine)
-                    self.config.remove(quarantine)
+                if not self.quarantines:
-                    break
+                    self.logger.debug(
-            if not self.config:
+                        f"host {hostaddr[0]} is in whitelist of all "
-                self.logger.debug("host {} is ignored by all quarantines, skip further processing", hostaddr[0])
+                        f"quarantines, skip further processing")
-                return Milter.ACCEPT
+                    return Milter.ACCEPT
         return Milter.CONTINUE
 
     @Milter.noreply
@@ -99,111 +382,158 @@ class QuarantineMilter(Milter.Base):
 
     @Milter.noreply
     def data(self):
-        self.queueid = self.getsymval('i')
+        self.qid = self.getsymval('i')
-        self.logger.debug("{}: received queue-id from MTA".format(self.queueid))
+        self.logger.debug(
+            f"{self.qid}: received queue-id from MTA")
         self.recipients = list(self.recipients)
+        self.logger.debug(
+            f"{self.qid}: initializing memory buffer to save email data")
+        # initialize memory buffer to save email data
+        self.fp = BytesIO()
         self.headers = []
         return Milter.CONTINUE
 
-    @Milter.noreply
     def header(self, name, value):
-        self.headers.append((name, value))
+        try:
-        return Milter.CONTINUE
+            # remove surrogates from value
+            value = value.encode(
+                errors="surrogateescape").decode(errors="replace")
+            self.logger.debug(f"{self.qid}: received header: {name}: {value}")
+            # write email header to memory buffer
+            self.fp.write(f"{name}: {value}\r\n".encode(
+                encoding="ascii", errors="replace"))
+            header = make_header(decode_header(value), errors="replace")
+            value = str(header).replace("\x00", "")
+            self.logger.debug(
+                f"{self.qid}: decoded header: {name}: {value}")
+            self.headers.append((name, value))
+            return Milter.CONTINUE
+        except Exception as e:
+            self.logger.exception(
+                f"an exception occured in header function: {e}")
+            return Milter.TEMPFAIL
 
     def eoh(self):
         try:
-            self.whitelist_cache = whitelists.WhitelistCache()
+            self.fp.write("\r\n".encode(encoding="ascii"))
+            self.wl_cache = whitelists.WhitelistCache()
 
             # initialize dicts to set quaranines per recipient and keep matches
-            self.recipients_quarantines = {}
+            self.rcpts_quarantines = {}
-            self.quarantines_matches = {}
+            self.matches = {}
 
             # iterate email headers
-            recipients_to_check = self.recipients.copy()
+            rcpts_to_check = self.recipients.copy()
             for name, value in self.headers:
-                header = "{}: {}".format(name, value)
+                header = f"{name}: {value}"
-                self.logger.debug("{}: checking header against configured quarantines: {}".format(self.queueid, header))
+                self.logger.debug(
+                    f"{self.qid}: checking header against configured "
+                    f"quarantines: {header}")
                 # iterate quarantines
-                for quarantine in self.config:
+                for quarantine in self.quarantines:
-                    if len(self.recipients_quarantines) == len(self.recipients):
+                    if len(self.rcpts_quarantines) == len(
+                            self.recipients):
                         # every recipient matched a quarantine already
-                        if quarantine["index"] >= max([q["index"] for q in self.recipients_quarantines.values()]):
+                        if quarantine.index >= max(
-                             # all recipients matched a quarantine with at least the same precedence already, skip checks against quarantines with lower precedence
+                                [q.index for q in
-                             self.logger.debug("{}: {}: skip further checks of this header".format(self.queueid, quarantine["name"]))
+                                    self.rcpts_quarantines.values()]):
-                             break
+                            # all recipients matched a quarantine with at least
+                            # the same precedence already, skip checks against
+                            # quarantines with lower precedence
+                            self.logger.debug(
+                                f"{self.qid}: {quarantine.name}: skip further "
+                                f"checks of this header")
+                            break
 
                     # check email header against quarantine regex
-                    self.logger.debug("{}: {}: checking header against regex '{}'".format(self.queueid, quarantine["name"], quarantine["regex"]))
+                    self.logger.debug(
-                    match = quarantine["regex_compiled"].search(header)
+                        f"{self.qid}: {quarantine.name}: checking header "
+                        f"against regex '{str(quarantine.regex)}'")
+                    match = quarantine.match(header)
                     if match:
-                        self.logger.debug("{}: {}: header matched regex".format(self.queueid, quarantine["name"]))
+                        self.logger.debug(
+                            f"{self.qid}: {quarantine.name}: "
+                            f"header matched regex")
                         # check for whitelisted recipients
-                        whitelist = quarantine["whitelist_obj"]
+                        whitelist = quarantine.get_whitelist()
-                        if whitelist != None:
+                        if whitelist:
                             try:
-                                whitelisted_recipients = self.whitelist_cache.get_whitelisted_recipients(whitelist, self.mailfrom, recipients_to_check)
+                                wl_recipients = self.wl_cache.get_recipients(
+                                        whitelist,
+                                        self.mailfrom,
+                                        rcpts_to_check)
                             except RuntimeError as e:
-                                self.logger.error("{}: {}: unable to query whitelist: {}".format(self.queueid, quarantine["name"], e))
+                                self.logger.error(
+                                    f"{self.qid}: {quarantine.name}: unable "
+                                    f"to query whitelist: {e}")
                                 return Milter.TEMPFAIL
                         else:
-                            whitelisted_recipients = {}
+                            wl_recipients = {}
 
                         # iterate recipients
-                        for recipient in recipients_to_check.copy():
+                        for rcpt in rcpts_to_check.copy():
-                            if recipient in whitelisted_recipients:
+                            if rcpt in wl_recipients:
                                 # recipient is whitelisted in this quarantine
-                                self.logger.debug("{}: {}: recipient '{}' is whitelisted".format(self.queueid, quarantine["name"], recipient))
+                                self.logger.debug(
+                                    f"{self.qid}: {quarantine.name}: "
+                                    f"recipient '{rcpt}' is whitelisted")
                                 continue
 
-                            if recipient not in self.recipients_quarantines.keys() or self.recipients_quarantines[recipient]["index"] > quarantine["index"]:
+                            if rcpt not in self.rcpts_quarantines or \
-                                self.logger.debug("{}: {}: set quarantine for recipient '{}'".format(self.queueid, quarantine["name"], recipient))
+                                    self.rcpts_quarantines[rcpt].index > \
-                                # save match for later use as template variables
+                                    quarantine.index:
-                                self.quarantines_matches[quarantine["name"]] = match
+                                self.logger.debug(
-                                self.recipients_quarantines[recipient] = quarantine
+                                    f"{self.qid}: {quarantine.name}: set "
-                                if quarantine["index"] == 0:
+                                    f"quarantine for recipient '{rcpt}'")
-                                    # we do not need to check recipients which matched the quarantine with the highest precedence already
+                                # save match for later use as template
-                                    recipients_to_check.remove(recipient)
+                                # variables
+                                self.matches[quarantine.name] = match
+                                self.rcpts_quarantines[rcpt] = quarantine
+                                if quarantine.index == 0:
+                                    # we do not need to check recipients which
+                                    # matched the quarantine with the highest
+                                    # precedence already
+                                    rcpts_to_check.remove(rcpt)
                             else:
-                                self.logger.debug("{}: {}: a quarantine with the same or higher precedence matched already for recipient '{}'".format(self.queueid, quarantine["name"], recipient))
+                                self.logger.debug(
+                                    f"{self.qid}: {quarantine.name}: a "
+                                    f"quarantine with same or higher "
+                                    f"precedence matched already for "
+                                    f"recipient '{rcpt}'")
 
-                if not recipients_to_check:
+                if not rcpts_to_check:
-                    self.logger.debug("{}: all recipients matched the first quarantine, skipping all remaining header checks".format(self.queueid))
+                    self.logger.debug(
+                        f"{self.qid}: all recipients matched the first "
+                        f"quarantine, skipping all remaining header checks")
                     break
 
             # check if no quarantine has matched for all recipients
-            if not self.recipients_quarantines:
+            if not self.rcpts_quarantines:
                 # accept email
-                self.logger.info("{}: passed clean for all recipients".format(self.queueid))
+                self.logger.info(
+                    f"{self.qid}: passed clean for all recipients")
                 return Milter.ACCEPT
 
-            # check if the email body is needed
+            # check if the mail body is needed
-            keep_body = False
+            for recipient, quarantine in self.rcpts_quarantines.items():
-            for recipient, quarantine in self.recipients_quarantines.items():
+                if quarantine.get_storage() or quarantine.get_notification():
-                if quarantine["quarantine_obj"] or quarantine["notification_obj"]:
+                    # mail body is needed, continue processing
-                    keep_body = True
+                    return Milter.CONTINUE
-                    break
 
-            if keep_body:
+            # quarantine and notification are disabled on all matching
-                self.logger.debug("{}: initializing memory buffer to save email data".format(self.queueid))
+            # quarantines, just return configured action
-                # initialize memory buffer to save email data
+            quarantine = self._get_preferred_quarantine()
-                self.fp = BytesIO()
+            self.logger.info(
-                # write email headers to memory buffer
+                f"{self.qid}: {self.preferred_action} "
-                for name, value in self.headers:
+                f"matching quarantine is '{quarantine.name}', performing "
-                    self.fp.write("{}: {}\n".format(name, value).encode())
+                f"milter action {quarantine.action}")
-                self.fp.write("\n".encode())
+            if quarantine.action == "REJECT":
-            else:
+                self.setreply("554", "5.7.0", quarantine.reject_reason)
-                # quarantine and notification are disabled on all matching quarantines, return configured action
+            return quarantine.milter_action
-                quarantine = self._get_preferred_quarantine()
-                self.logger.info("{}: {} matching quarantine is '{}', performing milter action {}".format(self.queueid, self.global_config["preferred_quarantine_action"],
-                        quarantine["name"], quarantine["action"].upper()))
-                if quarantine["action"] == "reject":
-                    self.setreply("554", "5.7.0", quarantine["reject_reason"])
-                return quarantine["milter_action"]
-
-            return Milter.CONTINUE
 
         except Exception as e:
-            self.logger.exception("an exception occured in eoh function: {}".format(e))
+            self.logger.exception(
+                f"an exception occured in eoh function: {e}")
             return Milter.TEMPFAIL
 
     def body(self, chunk):
@@ -211,7 +541,8 @@ class QuarantineMilter(Milter.Base):
             # save received body chunk
             self.fp.write(chunk)
         except Exception as e:
-            self.logger.exception("an exception occured in body function: {}".format(e))
+            self.logger.exception(
+                f"an exception occured in body function: {e}")
             return Milter.TEMPFAIL
         return Milter.CONTINUE
 
@@ -220,39 +551,56 @@ class QuarantineMilter(Milter.Base):
             # processing recipients grouped by quarantines
             quarantines = []
             for quarantine, recipients in groupby(
-                    sorted(self.recipients_quarantines, key=lambda x: self.recipients_quarantines[x]["index"])
+                    sorted(self.rcpts_quarantines,
-                    , lambda x: self.recipients_quarantines[x]):
+                           key=lambda x: self.rcpts_quarantines[x].index),
+                    lambda x: self.rcpts_quarantines[x]):
                 quarantines.append((quarantine, list(recipients)))
 
             # iterate quarantines sorted by index
-            for quarantine, recipients in sorted(quarantines, key=lambda x: x[0]["index"]):
+            for quarantine, recipients in sorted(
-                quarantine_id = ""
+                    quarantines, key=lambda x: x[0].index):
                 headers = defaultdict(str)
                 for name, value in self.headers:
                     headers[name.lower()] = value
-                subgroups = self.quarantines_matches[quarantine["name"]].groups(default="")
+                subgroups = self.matches[quarantine.name].groups(
-                named_subgroups = self.quarantines_matches[quarantine["name"]].groupdict(default="")
+                    default="")
+                named_subgroups = self.matches[quarantine.name].groupdict(
+                    default="")
 
-                # check if a quarantine is configured
+                rcpts = ", ".join(recipients)
-                if quarantine["quarantine_obj"] != None:
+
+                # check if a storage is configured
+                storage_id = ""
+                storage = quarantine.get_storage()
+                if storage:
                     # add email to quarantine
-                    self.logger.info("{}: adding to quarantine '{}' for: {}".format(self.queueid, quarantine["name"], ", ".join(recipients)))
+                    self.logger.info(
+                        f"{self.qid}: adding to quarantine "
+                        f"'{quarantine.name}' for: {rcpts}")
                     try:
-                        quarantine_id = quarantine["quarantine_obj"].add(self.queueid, self.mailfrom, recipients, headers, self.fp,
+                        storage_id = storage.add(
-                                subgroups, named_subgroups)
+                            self.qid, self.mailfrom, recipients, headers,
+                            self.fp, subgroups, named_subgroups)
                     except RuntimeError as e:
-                        self.logger.error("{}: unable to add to quarantine '{}': {}".format(self.queueid, quarantine["name"], e))
+                        self.logger.error(
+                            f"{self.qid}: unable to add to quarantine "
+                            f"'{quarantine.name}': {e}")
                         return Milter.TEMPFAIL
 
                 # check if a notification is configured
-                if quarantine["notification_obj"] != None:
+                notification = quarantine.get_notification()
+                if notification:
                     # notify
-                    self.logger.info("{}: sending notification for quarantine '{}' to: {}".format(self.queueid, quarantine["name"], ", ".join(recipients)))
+                    self.logger.info(
+                        f"{self.qid}: sending notification to: {rcpts}")
                     try:
-                        quarantine["notification_obj"].notify(self.queueid, quarantine_id, self.mailfrom, recipients, headers, self.fp,
+                        notification.notify(
-                                subgroups, named_subgroups)
+                            self.qid, storage_id,
+                            self.mailfrom, recipients, headers, self.fp,
+                            subgroups, named_subgroups)
                     except RuntimeError as e:
-                        self.logger.error("{}: unable to send notification for quarantine '{}': {}".format(self.queueid, quarantine["name"], e))
+                        self.logger.error(
+                            f"{self.qid}: unable to send notification: {e}")
                         return Milter.TEMPFAIL
 
                 # remove processed recipient
@@ -264,168 +612,96 @@ class QuarantineMilter(Milter.Base):
 
             # email passed clean for at least one recipient, accepting email
             if self.recipients:
-                self.logger.info("{}: passed clean for: {}".format(self.queueid, ", ".join(self.recipients)))
+                rcpts = ", ".join(recipients)
+                self.logger.info(
+                    f"{self.qid}: passed clean for: {rcpts}")
                 return Milter.ACCEPT
 
-            ## return configured action
+            # return configured action
             quarantine = self._get_preferred_quarantine()
-            self.logger.info("{}: {} matching quarantine is '{}', performing milter action {}".format(self.queueid, self.global_config["preferred_quarantine_action"],
+            self.logger.info(
-                    quarantine["name"], quarantine["action"].upper()))
+                f"{self.qid}: {self.preferred_action} matching "
-            if quarantine["action"] == "reject":
+                f"quarantine is '{quarantine.name}', performing milter "
-                self.setreply("554", "5.7.0", quarantine["reject_reason"])
+                f"action {quarantine.action}")
-            return quarantine["milter_action"]
+            if quarantine.action == "REJECT":
+                self.setreply("554", "5.7.0", quarantine.reject_reason)
+            return quarantine.milter_action
 
         except Exception as e:
-            self.logger.exception("an exception occured in eom function: {}".format(e))
+            self.logger.exception(
+                f"an exception occured in eom function: {e}")
             return Milter.TEMPFAIL
 
+    def close(self):
+        self.logger.debug(
+            f"disconnect from {self.hostaddr[0]} port {self.hostaddr[1]}")
+        return Milter.CONTINUE
 
-def generate_milter_config(configtest=False, config_files=[]):
+
+def setup_milter(test=False, cfg_files=[]):
     "Generate the configuration for QuarantineMilter class."
     logger = logging.getLogger(__name__)
 
     # read config file
     parser = configparser.ConfigParser()
-    if not config_files:
+    if not cfg_files:
-        config_files = parser.read(QuarantineMilter.get_configfiles())
+        cfg_files = parser.read(QuarantineMilter.get_cfg_files())
     else:
-        config_files = parser.read(config_files)
+        cfg_files = parser.read(cfg_files)
-    if not config_files:
+    if not cfg_files:
         raise RuntimeError("config file not found")
 
-    QuarantineMilter.set_configfiles(config_files)
+    QuarantineMilter.set_cfg_files(cfg_files)
-    os.chdir(os.path.dirname(config_files[0]))
+    os.chdir(os.path.dirname(cfg_files[0]))
 
     # check if mandatory config options in global section are present
     if "global" not in parser.sections():
-        raise RuntimeError("mandatory section 'global' not present in config file")
+        raise RuntimeError(
+            "mandatory section 'global' not present in config file")
     for option in ["quarantines", "preferred_quarantine_action"]:
         if not parser.has_option("global", option):
-            raise RuntimeError("mandatory option '{}' not present in config section 'global'".format(option))
+            raise RuntimeError(
+                f"mandatory option '{option}' not present in config "
+                f"section 'global'")
 
     # read global config section
-    global_config = dict(parser.items("global"))
+    global_cfg = dict(parser.items("global"))
-    global_config["preferred_quarantine_action"] = global_config["preferred_quarantine_action"].lower()
+    preferred_action = global_cfg["preferred_quarantine_action"].lower()
-    if global_config["preferred_quarantine_action"] not in ["first", "last"]:
+    if preferred_action not in ["first", "last"]:
-        raise RuntimeError("option preferred_quarantine_action has illegal value")
+        raise RuntimeError(
+            "option preferred_quarantine_action has illegal value")
 
     # read active quarantine names
-    quarantine_names = [ q.strip() for q in global_config["quarantines"].split(",") ]
+    quarantines = [
-    if len(quarantine_names) != len(set(quarantine_names)):
+        q.strip() for q in global_cfg["quarantines"].split(",")]
-        raise RuntimeError("at least one quarantine is specified multiple times in quarantines option")
+    if len(quarantines) != len(set(quarantines)):
-    if "global" in quarantine_names:
+        raise RuntimeError(
-        quarantine_names.remove("global")
+            "at least one quarantine is specified multiple times "
-        logger.warning("removed illegal quarantine name 'global' from list of active quarantines")
+            "in quarantines option")
-    if not quarantine_names:
+    if "global" in quarantines:
+        quarantines.remove("global")
+        logger.warning(
+            "removed illegal quarantine name 'global' from list of "
+            "active quarantines")
+    if not quarantines:
         raise RuntimeError("no quarantines configured")
 
-    milter_config = []
+    milter_quarantines = []
-
     logger.debug("preparing milter configuration ...")
     # iterate quarantine names
-    for index, quarantine_name in enumerate(quarantine_names):
+    for index, name in enumerate(quarantines):
-
         # check if config section for current quarantine exists
-        if quarantine_name not in parser.sections():
+        if name not in parser.sections():
-            raise RuntimeError("config section '{}' does not exist".format(quarantine_name))
+            raise RuntimeError(
-        config = dict(parser.items(quarantine_name))
+                f"config section '{name}' does not exist")
 
-        # check if mandatory config options are present in config
+        cfg = dict(parser.items(name))
-        for option in ["regex", "quarantine_type", "notification_type", "action", "whitelist_type", "smtp_host", "smtp_port"]:
+        quarantine = Quarantine(name, index)
-            if option not in config.keys() and \
+        quarantine.setup_from_cfg(global_cfg, cfg, test)
-                    option in global_config.keys():
+        milter_quarantines.append(quarantine)
-                config[option] = global_config[option]
-            if option not in config.keys():
-                raise RuntimeError("mandatory option '{}' not present in config section '{}' or 'global'".format(option, quarantine_name))
 
-        # check if optional config options are present in config
+    QuarantineMilter.preferred_action = preferred_action
-        defaults = {
+    QuarantineMilter.quarantines = milter_quarantines
-            "reject_reason": "Message rejected",
-            "ignore_hosts": ""
-        }
-        for option in defaults.keys():
-            if option not in config.keys() and \
-                    option in global_config.keys():
-                config[option] = global_config[option]
-            if option not in config.keys():
-                config[option] = defaults[option]
-
-        # set quarantine name
-        config["name"] = quarantine_name
-
-        # set the index
-        config["index"] = index
-
-        # pre-compile regex
-        logger.debug("{}: compiling regex '{}'".format(quarantine_name, config["regex"]))
-        config["regex_compiled"] = re.compile(config["regex"], re.MULTILINE + re.DOTALL + re.IGNORECASE)
-
-        # create quarantine instance
-        quarantine_type = config["quarantine_type"].lower()
-        if quarantine_type in quarantines.TYPES.keys():
-            logger.debug("{}: initializing quarantine type '{}'".format(quarantine_name, quarantine_type.upper()))
-            quarantine = quarantines.TYPES[quarantine_type](global_config, config, configtest)
-        elif quarantine_type == "none":
-            logger.debug("{}: quarantine is NONE".format(quarantine_name))
-            quarantine = None
-        else:
-            raise RuntimeError("{}: unknown quarantine type '{}'".format(quarantine_name, quarantine_type))
-
-        config["quarantine_obj"] = quarantine
-
-        # create whitelist instance
-        whitelist_type = config["whitelist_type"].lower()
-        if whitelist_type in whitelists.TYPES.keys():
-            logger.debug("{}: initializing whitelist type '{}'".format(quarantine_name, whitelist_type.upper()))
-            whitelist = whitelists.TYPES[whitelist_type](global_config, config, configtest)
-        elif whitelist_type == "none":
-            logger.debug("{}: whitelist is NONE".format(quarantine_name))
-            whitelist = None
-        else:
-            raise RuntimeError("{}: unknown whitelist type '{}'".format(quarantine_name, whitelist_type))
-
-        config["whitelist_obj"] = whitelist
-
-        # create notification instance
-        notification_type = config["notification_type"].lower()
-        if notification_type in notifications.TYPES.keys():
-            logger.debug("{}: initializing notification type '{}'".format(quarantine_name, notification_type.upper()))
-            notification = notifications.TYPES[notification_type](global_config, config, configtest)
-        elif notification_type == "none":
-            logger.debug("{}: notification is NONE".format(quarantine_name))
-            notification = None
-        else:
-            raise RuntimeError("{}: unknown notification type '{}'".format(quarantine_name, notification_type))
-
-        config["notification_obj"] = notification
-
-        # determining milter action for this quarantine
-        action = config["action"].upper()
-        if action in QuarantineMilter.get_actions().keys():
-            logger.debug("{}: action is {}".format(quarantine_name, action))
-            config["milter_action"] = QuarantineMilter.get_actions()[action]
-        else:
-            raise RuntimeError("{}: unknown action '{}'".format(quarantine_name, action))
-
-        # create host/network whitelist
-        config["ignore_hosts_list"] = []
-        ignored = set([ p.strip() for p in config["ignore_hosts"].split(",") if p])
-        for ignore in ignored:
-            if not ignore:
-                continue
-            # parse network notation
-            try:
-                net = IPNetwork(ignore)
-            except AddrFormatError as e:
-                raise RuntimeError("error parsing ignore_hosts: {}".format(e))
-            else:
-                config["ignore_hosts_list"].append(net)
-        if config["ignore_hosts_list"]:
-            logger.debug("{}: ignore hosts: {}".format(quarantine_name, ", ".join(ignored)))
-
-        milter_config.append(config)
-
-    return global_config, milter_config
 
 
 def reload_config():
@@ -433,11 +709,9 @@ def reload_config():
     logger = logging.getLogger(__name__)
 
     try:
-        global_config, config = generate_milter_config()
+        setup_milter()
     except RuntimeError as e:
         logger.info(e)
         logger.info("daemon is still running with previous configuration")
     else:
-        logger.info("reloading configuration")
+        logger.info("reloaded configuration")
-        QuarantineMilter.global_config = global_config
-        QuarantineMilter.config = config

pyquarantine/cli.py

@@ -20,23 +20,43 @@ import logging.handlers
 import sys
 import time
 
-import pyquarantine
+from pyquarantine import QuarantineMilter, setup_milter
+from pyquarantine import __version__ as version
 
 
-def _get_quarantine_obj(config, quarantine):
+def _get_quarantine(quarantines, name):
     try:
-        quarantine_obj = next((q["quarantine_obj"] for q in config if q["name"] == quarantine))
+        quarantine = next((q for q in quarantines if q.name == name))
     except StopIteration:
-        raise RuntimeError("invalid quarantine '{}'".format(quarantine))
+        raise RuntimeError("invalid quarantine 'name'")
-    return quarantine_obj
+    return quarantine
 
 
-def _get_whitelist_obj(config, quarantine):
+def _get_storage(quarantines, name):
-    try:
+    quarantine = _get_quarantine(quarantines, name)
-        whitelist_obj = next((q["whitelist_obj"] for q in config if q["name"] == quarantine))
+    storage = quarantine.get_storage()
-    except StopIteration:
+    if not storage:
-        raise RuntimeError("invalid quarantine '{}'".format(quarantine))
+        raise RuntimeError(
-    return whitelist_obj
+                "storage type is set to NONE")
+    return storage
+
+
+def _get_notification(quarantines, name):
+    quarantine = _get_quarantine(quarantines, name)
+    notification = quarantine.get_notification()
+    if not notification:
+        raise RuntimeError(
+                "notification type is set to NONE")
+    return notification
+
+
+def _get_whitelist(quarantines, name):
+    quarantine = _get_quarantine(quarantines, name)
+    whitelist = quarantine.get_whitelist()
+    if not whitelist:
+        raise RuntimeError(
+                "whitelist type is set to NONE")
+    return whitelist
 
 
 def print_table(columns, rows):
@@ -51,11 +71,12 @@ def print_table(columns, rows):
         # get the length of the header string
         lengths = [len(header)]
         # get the length of the longest value
-        lengths.append(len(str(max(rows, key=lambda x: len(str(x[key])))[key])))
+        lengths.append(
-        # use the the longer one
+            len(str(max(rows, key=lambda x: len(str(x[key])))[key])))
+        # use the longer one
         length = max(lengths)
         column_lengths.append(length)
-        column_formats.append("{{:<{}}}".format(length))
+        column_formats.append(f"{{:<{length}}}")
 
     # define row format
     row_format = " | ".join(column_formats)
@@ -70,7 +91,7 @@ def print_table(columns, rows):
     print(row_format.format(*[column[0] for column in columns]))
     print(separator)
 
-    keys = [ entry[1] for entry in columns ]
+    keys = [entry[1] for entry in columns]
     # print rows
     for entry in rows:
         row = []
@@ -79,39 +100,70 @@ def print_table(columns, rows):
         print(row_format.format(*row))
 
 
-def list_quarantines(config, args):
+def list_quarantines(quarantines, args):
     if args.batch:
-        print("\n".join([ quarantine["name"] for quarantine in config ]))
+        print("\n".join([q.name for q in quarantines]))
     else:
+        qlist = []
+        for q in quarantines:
+            storage = q.get_storage()
+            if storage:
+                storage_type = q.get_storage().storage_type
+            else:
+                storage_type = "NONE"
+
+            notification = q.get_notification()
+            if notification:
+                notification_type = q.get_notification().notification_type
+            else:
+                notification_type = "NONE"
+
+            whitelist = q.get_whitelist()
+            if whitelist:
+                whitelist_type = q.get_whitelist().whitelist_type
+            else:
+                whitelist_type = "NONE"
+
+            qlist.append({
+                "name": q.name,
+                "storage": storage_type,
+                "notification": notification_type,
+                "whitelist": whitelist_type,
+                "action": q.action})
         print_table(
-            [("Name", "name"), ("Quarantine", "quarantine_type"), ("Notification", "notification_type"), ("Action", "action")],
+            [("Name", "name"),
-            config
+             ("Storage", "storage"),
+             ("Notification", "notification"),
+             ("Whitelist", "whitelist"),
+             ("Action", "action")],
+            qlist
         )
 
 
-def list_quarantine_emails(config, args):
+def list_quarantine_emails(quarantines, args):
     logger = logging.getLogger(__name__)
-
+    storage = _get_storage(quarantines, args.quarantine)
-    # get quarantine object
-    quarantine = _get_quarantine_obj(config, args.quarantine)
-    if quarantine == None:
-        raise RuntimeError("quarantine type is set to None, unable to list emails")
-
     # find emails and transform some metadata values to strings
     rows = []
-    emails = quarantine.find(mailfrom=args.mailfrom, recipients=args.recipients, older_than=args.older_than)
+    emails = storage.find(
-    for quarantine_id, metadata in emails.items():
+        args.mailfrom, args.recipients, args.older_than)
-        row = emails[quarantine_id]
+    for storage_id, metadata in emails.items():
-        row["quarantine_id"] = quarantine_id
+        row = emails[storage_id]
-        row["date"] = time.strftime('%Y-%m-%d %H:%M:%S', time.localtime(metadata["date"]))
+        row["storage_id"] = storage_id
+        row["date"] = time.strftime(
+            '%Y-%m-%d %H:%M:%S',
+            time.localtime(
+                metadata["date"]))
         row["mailfrom"] = metadata["mailfrom"]
         row["recipient"] = metadata["recipients"].pop(0)
-        row["subject"] = emails[quarantine_id]["headers"]["subject"][:60]
+        if "subject" not in emails[storage_id]["headers"].keys():
+            emails[storage_id]["headers"]["subject"] = ""
+        row["subject"] = emails[storage_id]["headers"]["subject"][:60].strip()
         rows.append(row)
 
         if metadata["recipients"]:
             row = {
-                "quarantine_id": "",
+                "storage_id": "",
                 "date": "",
                 "mailfrom": "",
                 "recipient": metadata["recipients"].pop(0),
@@ -124,119 +176,113 @@ def list_quarantine_emails(config, args):
         print("\n".join(emails.keys()))
         return
 
-    if not emails: logger.info("quarantine '{}' is empty".format(args.quarantine))
+    if not emails:
+        logger.info(f"quarantine '{args.quarantine}' is empty")
     print_table(
-        [("Quarantine-ID", "quarantine_id"), ("Date", "date"), ("From", "mailfrom"), ("Recipient(s)", "recipient"), ("Subject", "subject")],
+        [("Quarantine-ID", "storage_id"), ("Date", "date"),
+         ("From", "mailfrom"), ("Recipient(s)", "recipient"),
+         ("Subject", "subject")],
         rows
     )
 
 
-def list_whitelist(config, args):
+def list_whitelist(quarantines, args):
     logger = logging.getLogger(__name__)
-
+    whitelist = _get_whitelist(quarantines, args.quarantine)
-    # get whitelist object
-    whitelist = _get_whitelist_obj(config, args.quarantine)
-    if whitelist == None:
-        raise RuntimeError("whitelist type is set to None, unable to list entries")
 
     # find whitelist entries
-    entries = whitelist.find(mailfrom=args.mailfrom, recipients=args.recipients, older_than=args.older_than)
+    entries = whitelist.find(
+        mailfrom=args.mailfrom,
+        recipients=args.recipients,
+        older_than=args.older_than)
     if not entries:
-        logger.info("whitelist of quarantine '{}' is empty".format(args.quarantine))
+        logger.info(
+            f"whitelist of quarantine '{args.quarantine}' is empty")
         return
 
     # transform some values to strings
     for entry_id, entry in entries.items():
         entries[entry_id]["permanent_str"] = str(entry["permanent"])
-        entries[entry_id]["created_str"] = entry["created"].strftime('%Y-%m-%d %H:%M:%S')
+        entries[entry_id]["created_str"] = entry["created"].strftime(
-        entries[entry_id]["last_used_str"] = entry["last_used"].strftime('%Y-%m-%d %H:%M:%S')
+            '%Y-%m-%d %H:%M:%S')
+        entries[entry_id]["last_used_str"] = entry["last_used"].strftime(
+            '%Y-%m-%d %H:%M:%S')
 
     print_table(
         [
-            ("ID", "id"), ("From", "mailfrom"), ("To", "recipient"), ("Created", "created_str"),
+            ("ID", "id"), ("From", "mailfrom"), ("To", "recipient"),
-            ("Last used", "last_used_str"), ("Comment", "comment"), ("Permanent", "permanent_str")
+            ("Created", "created_str"), ("Last used", "last_used_str"),
+            ("Comment", "comment"), ("Permanent", "permanent_str")
         ],
         entries.values()
     )
 
 
-def add_whitelist_entry(config, args):
+def add_whitelist_entry(quarantines, args):
     logger = logging.getLogger(__name__)
-
+    whitelist = _get_whitelist(quarantines, args.quarantine)
-    # get whitelist object
-    whitelist = _get_whitelist_obj(config, args.quarantine)
-    if whitelist == None:
-        raise RuntimeError("whitelist type is set to None, unable to add entries")
 
     # check existing entries
     entries = whitelist.check(args.mailfrom, args.recipient)
     if entries:
         # check if the exact entry exists already
         for entry in entries.values():
-            if entry["mailfrom"] == args.mailfrom and entry["recipient"] == args.recipient:
+            if entry["mailfrom"] == args.mailfrom and \
-                raise RuntimeError("an entry with this from/to combination already exists")
+                    entry["recipient"] == args.recipient:
+                raise RuntimeError(
+                    "an entry with this from/to combination already exists")
 
         if not args.force:
             # the entry is already covered by others
             for entry_id, entry in entries.items():
                 entries[entry_id]["permanent_str"] = str(entry["permanent"])
-                entries[entry_id]["created_str"] = entry["created"].strftime('%Y-%m-%d %H:%M:%S')
+                entries[entry_id]["created_str"] = entry["created"].strftime(
-                entries[entry_id]["last_used_str"] = entry["last_used"].strftime('%Y-%m-%d %H:%M:%S')
+                    '%Y-%m-%d %H:%M:%S')
+                entries[entry_id]["last_used_str"] = entry["last_used"].strftime(
+                    '%Y-%m-%d %H:%M:%S')
             print_table(
                 [
-                    ("ID", "id"), ("From", "mailfrom"), ("To", "recipient"), ("Created", "created_str"),
+                    ("ID", "id"), ("From", "mailfrom"), ("To", "recipient"),
-                    ("Last used", "last_used_str"), ("Comment", "comment"), ("Permanent", "permanent_str")
+                    ("Created", "created_str"), ("Last used", "last_used_str"),
+                    ("Comment", "comment"), ("Permanent", "permanent_str")
                 ],
                 entries.values()
             )
             print("")
-            raise RuntimeError("from/to combination is already covered by the entries above, use --force to override.")
+            raise RuntimeError(
+                "from/to combination is already covered by the entries above, "
+                "use --force to override.")
 
     # add entry to whitelist
     whitelist.add(args.mailfrom, args.recipient, args.comment, args.permanent)
     logger.info("whitelist entry added successfully")
 
 
-def delete_whitelist_entry(config, args):
+def delete_whitelist_entry(quarantines, args):
     logger = logging.getLogger(__name__)
-
+    whitelist = _get_whitelist(quarantines, args.quarantine)
-    whitelist = _get_whitelist_obj(config, args.quarantine)
-    if whitelist == None:
-        raise RuntimeError("whitelist type is set to None, unable to delete entries")
-
     whitelist.delete(args.whitelist_id)
     logger.info("whitelist entry deleted successfully")
 
 
-def notify_email(config, args):
+def notify(quarantines, args):
     logger = logging.getLogger(__name__)
-
+    quarantine = _get_quarantine(quarantines, args.quarantine)
-    quarantine = _get_quarantine_obj(config, args.quarantine)
-    if quarantine == None:
-        raise RuntimeError("quarantine type is set to None, unable to send notification")
     quarantine.notify(args.quarantine_id, args.recipient)
-    logger.info("sent notification successfully")
+    logger.info("notification sent successfully")
 
 
-def release_email(config, args):
+def release(quarantines, args):
     logger = logging.getLogger(__name__)
-
+    quarantine = _get_quarantine(quarantines, args.quarantine)
-    quarantine = _get_quarantine_obj(config, args.quarantine)
-    if quarantine == None:
-        raise RuntimeError("quarantine type is set to None, unable to release email")
-
     quarantine.release(args.quarantine_id, args.recipient)
     logger.info("quarantined email released successfully")
 
 
-def delete_email(config, args):
+def delete(quarantines, args):
     logger = logging.getLogger(__name__)
-
+    storage = _get_storage(quarantines, args.quarantine)
-    quarantine = _get_quarantine_obj(config, args.quarantine)
+    storage.delete(args.quarantine_id, args.recipient)
-    if quarantine == None:
-        raise RuntimeError("quarantine type is set to None, unable to delete email")
-
-    quarantine.delete(args.quarantine_id, args.recipient)
     logger.info("quarantined email deleted successfully")
 
 
@@ -253,78 +299,238 @@ class StdOutFilter(logging.Filter):
 def main():
     "PyQuarantine command-line interface."
     # parse command line
-    formatter_class = lambda prog: argparse.HelpFormatter(prog, max_help_position=50, width=140)
+    def formatter_class(prog): return argparse.HelpFormatter(
-    parser = argparse.ArgumentParser(description="PyQuarantine CLI", formatter_class=formatter_class)
+        prog, max_help_position=50, width=140)
-    parser.add_argument("-c", "--config", help="Config files to read.", nargs="+", metavar="CFG",
+    parser = argparse.ArgumentParser(
-            default=pyquarantine.QuarantineMilter.get_configfiles())
+        description="PyQuarantine CLI",
-    parser.add_argument("-d", "--debug", help="Log debugging messages.", action="store_true")
+        formatter_class=formatter_class)
+    parser.add_argument(
+        "-c", "--config",
+        help="Config files to read.",
+        nargs="+", metavar="CFG",
+        default=QuarantineMilter.get_cfg_files())
+    parser.add_argument(
+        "-d", "--debug",
+        help="Log debugging messages.",
+        action="store_true")
+    parser.add_argument(
+        "-v", "--version",
+        help="Print version.",
+        action="version",
+        version=f"%(prog)s ({version})")
     parser.set_defaults(syslog=False)
-    subparsers = parser.add_subparsers(dest="command", title="Commands")
+    subparsers = parser.add_subparsers(
+        dest="command",
+        title="Commands")
     subparsers.required = True
 
     # list command
-    list_parser = subparsers.add_parser("list", help="List available quarantines.", formatter_class=formatter_class)
+    list_parser = subparsers.add_parser(
-    list_parser.add_argument("-b", "--batch", help="Print results using only quarantine names, each on a new line.", action="store_true")
+        "list",
+        help="List available quarantines.",
+        formatter_class=formatter_class)
+    list_parser.add_argument(
+        "-b", "--batch",
+        help="Print results using only quarantine names, each on a new line.",
+        action="store_true")
     list_parser.set_defaults(func=list_quarantines)
 
     # quarantine command group
-    quarantine_parser = subparsers.add_parser("quarantine", description="Manage quarantines.", help="Manage quarantines.", formatter_class=formatter_class)
+    quarantine_parser = subparsers.add_parser(
-    quarantine_parser.add_argument("quarantine", metavar="QUARANTINE", help="Quarantine name.")
+        "quarantine",
-    quarantine_subparsers = quarantine_parser.add_subparsers(dest="command", title="Quarantine commands")
+        description="Manage quarantines.",
+        help="Manage quarantines.",
+        formatter_class=formatter_class)
+    quarantine_parser.add_argument(
+        "quarantine",
+        metavar="QUARANTINE",
+        help="Quarantine name.")
+    quarantine_subparsers = quarantine_parser.add_subparsers(
+        dest="command",
+        title="Quarantine commands")
     quarantine_subparsers.required = True
     # quarantine list command
-    quarantine_list_parser = quarantine_subparsers.add_parser("list", description="List emails in quarantines.", help="List emails in quarantine.", formatter_class=formatter_class)
+    quarantine_list_parser = quarantine_subparsers.add_parser(
-    quarantine_list_parser.add_argument("-f", "--from", dest="mailfrom", help="Filter emails by from address.", default=None, nargs="+")
+        "list",
-    quarantine_list_parser.add_argument("-t", "--to", dest="recipients", help="Filter emails by recipient address.", default=None, nargs="+")
+        description="List emails in quarantines.",
-    quarantine_list_parser.add_argument("-o", "--older-than", dest="older_than", help="Filter emails by age (days).", default=None, type=float)
+        help="List emails in quarantine.",
-    quarantine_list_parser.add_argument("-b", "--batch", help="Print results using only email quarantine IDs, each on a new line.", action="store_true")
+        formatter_class=formatter_class)
+    quarantine_list_parser.add_argument(
+        "-f", "--from",
+        dest="mailfrom",
+        help="Filter emails by from address.",
+        default=None,
+        nargs="+")
+    quarantine_list_parser.add_argument(
+        "-t", "--to",
+        dest="recipients",
+        help="Filter emails by recipient address.",
+        default=None,
+        nargs="+")
+    quarantine_list_parser.add_argument(
+        "-o", "--older-than",
+        dest="older_than",
+        help="Filter emails by age (days).",
+        default=None,
+        type=float)
+    quarantine_list_parser.add_argument(
+        "-b", "--batch",
+        help="Print results using only email quarantine IDs, each on a new line.",
+        action="store_true")
     quarantine_list_parser.set_defaults(func=list_quarantine_emails)
     # quarantine notify command
-    quarantine_notify_parser = quarantine_subparsers.add_parser("notify", description="Notify recipient about email in quarantine.", help="Notify recipient about email in quarantine.", formatter_class=formatter_class)
+    quarantine_notify_parser = quarantine_subparsers.add_parser(
-    quarantine_notify_parser.add_argument("quarantine_id", metavar="ID", help="Quarantine ID.")
+        "notify",
-    quarantine_notify_parser_group = quarantine_notify_parser.add_mutually_exclusive_group(required=True)
+        description="Notify recipient about email in quarantine.",
-    quarantine_notify_parser_group.add_argument("-t", "--to", dest="recipient", help="Release email for one recipient address.")
+        help="Notify recipient about email in quarantine.",
-    quarantine_notify_parser_group.add_argument("-a", "--all", help="Release email for all recipients.", action="store_true")
+        formatter_class=formatter_class)
-    quarantine_notify_parser.set_defaults(func=notify_email)
+    quarantine_notify_parser.add_argument(
+        "quarantine_id",
+        metavar="ID",
+        help="Quarantine ID.")
+    quarantine_notify_parser_group = quarantine_notify_parser.add_mutually_exclusive_group(
+        required=True)
+    quarantine_notify_parser_group.add_argument(
+        "-t", "--to",
+        dest="recipient",
+        help="Release email for one recipient address.")
+    quarantine_notify_parser_group.add_argument(
+        "-a", "--all",
+        help="Release email for all recipients.",
+        action="store_true")
+    quarantine_notify_parser.set_defaults(func=notify)
     # quarantine release command
-    quarantine_release_parser = quarantine_subparsers.add_parser("release", description="Release email from quarantine.", help="Release email from quarantine.", formatter_class=formatter_class)
+    quarantine_release_parser = quarantine_subparsers.add_parser(
-    quarantine_release_parser.add_argument("quarantine_id", metavar="ID", help="Quarantine ID.")
+        "release",
-    quarantine_release_parser.add_argument("-n", "--disable-syslog", dest="syslog", help="Disable syslog messages.", action="store_false")
+        description="Release email from quarantine.",
-    quarantine_release_parser_group = quarantine_release_parser.add_mutually_exclusive_group(required=True)
+        help="Release email from quarantine.",
-    quarantine_release_parser_group.add_argument("-t", "--to", dest="recipient", help="Release email for one recipient address.")
+        formatter_class=formatter_class)
-    quarantine_release_parser_group.add_argument("-a", "--all", help="Release email for all recipients.", action="store_true")
+    quarantine_release_parser.add_argument(
-    quarantine_release_parser.set_defaults(func=release_email)
+        "quarantine_id",
+        metavar="ID",
+        help="Quarantine ID.")
+    quarantine_release_parser.add_argument(
+        "-n",
+        "--disable-syslog",
+        dest="syslog",
+        help="Disable syslog messages.",
+        action="store_false")
+    quarantine_release_parser_group = quarantine_release_parser.add_mutually_exclusive_group(
+        required=True)
+    quarantine_release_parser_group.add_argument(
+        "-t", "--to",
+        dest="recipient",
+        help="Release email for one recipient address.")
+    quarantine_release_parser_group.add_argument(
+        "-a", "--all",
+        help="Release email for all recipients.",
+        action="store_true")
+    quarantine_release_parser.set_defaults(func=release)
     # quarantine delete command
-    quarantine_delete_parser = quarantine_subparsers.add_parser("delete", description="Delete email from quarantine.", help="Delete email from quarantine.", formatter_class=formatter_class)
+    quarantine_delete_parser = quarantine_subparsers.add_parser(
-    quarantine_delete_parser.add_argument("quarantine_id", metavar="ID", help="Quarantine ID.")
+        "delete",
-    quarantine_delete_parser.add_argument("-n", "--disable-syslog", dest="syslog", help="Disable syslog messages.", action="store_false")
+        description="Delete email from quarantine.",
-    quarantine_delete_parser_group = quarantine_delete_parser.add_mutually_exclusive_group(required=True)
+        help="Delete email from quarantine.",
-    quarantine_delete_parser_group.add_argument("-t", "--to", dest="recipient", help="Delete email for one recipient address.")
+        formatter_class=formatter_class)
-    quarantine_delete_parser_group.add_argument("-a", "--all", help="Delete email for all recipients.", action="store_true")
+    quarantine_delete_parser.add_argument(
-    quarantine_delete_parser.set_defaults(func=delete_email)
+        "quarantine_id",
+        metavar="ID",
+        help="Quarantine ID.")
+    quarantine_delete_parser.add_argument(
+        "-n", "--disable-syslog",
+        dest="syslog",
+        help="Disable syslog messages.",
+        action="store_false")
+    quarantine_delete_parser_group = quarantine_delete_parser.add_mutually_exclusive_group(
+        required=True)
+    quarantine_delete_parser_group.add_argument(
+        "-t", "--to",
+        dest="recipient",
+        help="Delete email for one recipient address.")
+    quarantine_delete_parser_group.add_argument(
+        "-a", "--all",
+        help="Delete email for all recipients.",
+        action="store_true")
+    quarantine_delete_parser.set_defaults(func=delete)
 
     # whitelist command group
-    whitelist_parser = subparsers.add_parser("whitelist", description="Manage whitelists.", help="Manage whitelists.", formatter_class=formatter_class)
+    whitelist_parser = subparsers.add_parser(
-    whitelist_parser.add_argument("quarantine", metavar="QUARANTINE", help="Quarantine name.")
+        "whitelist",
-    whitelist_subparsers = whitelist_parser.add_subparsers(dest="command", title="Whitelist commands")
+        description="Manage whitelists.",
+        help="Manage whitelists.",
+        formatter_class=formatter_class)
+    whitelist_parser.add_argument(
+        "quarantine",
+        metavar="QUARANTINE",
+        help="Quarantine name.")
+    whitelist_subparsers = whitelist_parser.add_subparsers(
+        dest="command",
+        title="Whitelist commands")
     whitelist_subparsers.required = True
     # whitelist list command
-    whitelist_list_parser = whitelist_subparsers.add_parser("list", description="List whitelist entries.", help="List whitelist entries.", formatter_class=formatter_class)
+    whitelist_list_parser = whitelist_subparsers.add_parser(
-    whitelist_list_parser.add_argument("-f", "--from", dest="mailfrom", help="Filter entries by from address.", default=None, nargs="+")
+        "list",
-    whitelist_list_parser.add_argument("-t", "--to", dest="recipients", help="Filter entries by recipient address.", default=None, nargs="+")
+        description="List whitelist entries.",
-    whitelist_list_parser.add_argument("-o", "--older-than", dest="older_than", help="Filter emails by last used date (days).", default=None, type=float)
+        help="List whitelist entries.",
+        formatter_class=formatter_class)
+    whitelist_list_parser.add_argument(
+        "-f", "--from",
+        dest="mailfrom",
+        help="Filter entries by from address.",
+        default=None,
+        nargs="+")
+    whitelist_list_parser.add_argument(
+        "-t", "--to",
+        dest="recipients",
+        help="Filter entries by recipient address.",
+        default=None,
+        nargs="+")
+    whitelist_list_parser.add_argument(
+        "-o", "--older-than",
+        dest="older_than",
+        help="Filter emails by last used date (days).",
+        default=None,
+        type=float)
     whitelist_list_parser.set_defaults(func=list_whitelist)
     # whitelist add command
-    whitelist_add_parser = whitelist_subparsers.add_parser("add", description="Add whitelist entry.", help="Add whitelist entry.", formatter_class=formatter_class)
+    whitelist_add_parser = whitelist_subparsers.add_parser(
-    whitelist_add_parser.add_argument("-f", "--from", dest="mailfrom", help="From address.", required=True)
+        "add",
-    whitelist_add_parser.add_argument("-t", "--to", dest="recipient", help="Recipient address.", required=True)
+        description="Add whitelist entry.",
-    whitelist_add_parser.add_argument("-c", "--comment", help="Comment.", default="added by CLI")
+        help="Add whitelist entry.",
-    whitelist_add_parser.add_argument("-p", "--permanent", help="Add a permanent entry.", action="store_true")
+        formatter_class=formatter_class)
-    whitelist_add_parser.add_argument("--force", help="Force adding an entry, even if already covered by another entry.", action="store_true")
+    whitelist_add_parser.add_argument(
+        "-f", "--from",
+        dest="mailfrom",
+        help="From address.",
+        required=True)
+    whitelist_add_parser.add_argument(
+        "-t", "--to",
+        dest="recipient",
+        help="Recipient address.",
+        required=True)
+    whitelist_add_parser.add_argument(
+        "-c", "--comment",
+        help="Comment.",
+        default="added by CLI")
+    whitelist_add_parser.add_argument(
+        "-p", "--permanent",
+        help="Add a permanent entry.",
+        action="store_true")
+    whitelist_add_parser.add_argument(
+        "--force",
+        help="Force adding an entry, even if already covered by another entry.",
+        action="store_true")
     whitelist_add_parser.set_defaults(func=add_whitelist_entry)
     # whitelist delete command
-    whitelist_delete_parser = whitelist_subparsers.add_parser("delete", description="Delete whitelist entry.", help="Delete whitelist entry.", formatter_class=formatter_class)
+    whitelist_delete_parser = whitelist_subparsers.add_parser(
-    whitelist_delete_parser.add_argument("whitelist_id", metavar="ID", help="Whitelist ID.")
+        "delete",
+        description="Delete whitelist entry.",
+        help="Delete whitelist entry.",
+        formatter_class=formatter_class)
+    whitelist_delete_parser.add_argument(
+        "whitelist_id",
+        metavar="ID",
+        help="Whitelist ID.")
     whitelist_delete_parser.set_defaults(func=delete_whitelist_entry)
 
     args = parser.parse_args()
@@ -336,7 +542,8 @@ def main():
 
     # setup console log
     if args.debug:
-        formatter = logging.Formatter("%(levelname)s: [%(name)s] - %(message)s")
+        formatter = logging.Formatter(
+            "%(levelname)s: [%(name)s] - %(message)s")
     else:
         formatter = logging.Formatter("%(levelname)s: %(message)s")
     # stdout
@@ -355,17 +562,21 @@ def main():
 
     # try to generate milter configs
     try:
-        global_config, config = pyquarantine.generate_milter_config(config_files=args.config, configtest=True)
+        setup_milter(
+            cfg_files=args.config, test=True)
     except RuntimeError as e:
         logger.error(e)
         sys.exit(255)
 
     if args.syslog:
         # setup syslog
-        sysloghandler = logging.handlers.SysLogHandler(address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_MAIL)
+        sysloghandler = logging.handlers.SysLogHandler(
+            address="/dev/log",
+            facility=logging.handlers.SysLogHandler.LOG_MAIL)
         sysloghandler.setLevel(loglevel)
         if args.debug:
-            formatter = logging.Formatter("pyquarantine: [%(name)s] [%(levelname)s] %(message)s")
+            formatter = logging.Formatter(
+                "pyquarantine: [%(name)s] [%(levelname)s] %(message)s")
         else:
             formatter = logging.Formatter("pyquarantine: %(message)s")
         sysloghandler.setFormatter(formatter)
@@ -373,7 +584,7 @@ def main():
 
     # call the commands function
     try:
-        args.func(config, args)
+        args.func(QuarantineMilter.quarantines, args)
     except RuntimeError as e:
         logger.error(e)
         sys.exit(1)

pyquarantine/docs/pyquarantine.conf.example

@@ -30,12 +30,12 @@ preferred_quarantine_action = last
 
 
 [spam]
-# Option: ignore_hosts
+# Option: host_whitelist
 # Notes:  Set a list of host and network addresses to be ignored by this quarantine.
 #         All the common host/network notations are supported, including IPv6.
 # Value:  [ HOST ]
 #
-ignore_hosts = 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
+host_whitelist = 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
 
 # Option: regex
 # Notes:  Set the case insensitive regular expression to match against email headers.
@@ -57,18 +57,18 @@ smtp_host = 127.0.0.1
 #
 smtp_port = 25
 
-# Option: quarantine_type
+# Option: storage_type
-# Notes:  Set the quarantine type.
+# Notes:  Set the storage type.
 # Values: [ file | none ]
 #
-quarantine_type = file
+storage_type = file
 
-# Option: quarantine_directory
+# Option: storage_directory
 # Notes:  Set the directory to store quarantined emails.
 #         This option is needed by quarantine type 'file'.
 # Values: [ DIRECTORY ]
 #
-quarantine_directory = /var/lib/pyquarantine/spam
+storage_directory = /var/lib/pyquarantine/spam
 
 # Option: action
 # Notes:  Set the milter action to perform if email is processed by this quarantine.
@@ -88,6 +88,18 @@ reject_reason = Message rejected
 #
 notification_type = email
 
+# Option: notification_email_smtp_host
+# Notes:  Set the SMTP host. It will be used to send notification e-mails.
+# Values: [ HOSTNAME | IP_ADDRESS ]
+#
+notification_email_smtp_host = 127.0.0.1
+
+# Option: notification_email_smtp_port
+# Notes:  Set the SMTP port.
+# Values: [ PORT ]
+#
+notification_email_smtp_port = 25
+
 # Option: notification_email_envelope_from
 # Notes:  Set the envelope-from address used when sending notification emails.
 #         This option is needed by notification type 'email'.
@@ -117,10 +129,15 @@ notification_email_subject = Spam Quarantine Notification
 #
 notification_email_template = templates/notification.template
 
+# Option: notification_email_strip_images
+# Notes:  Optionally enable this option to strip img tags from emails.
+# Values: [ TRUE | ON | YES | FALSE | OFF | NO ]
+#
+notification_email_strip_images = False
+
 # Option: notification_email_replacement_img
-# Notes:  Set the path to the replacement image for img tags within emails.
+# Notes:  Optionally set the path to a replacement image for img tags within emails.
 #         A relative path to this config file can be used.
-#         This option is needed by notification type 'email'.
 # Values: [ IMAGE_PATH ]
 #
 notification_email_replacement_img = templates/removed.png
@@ -133,6 +150,13 @@ notification_email_replacement_img = templates/removed.png
 #
 notification_email_embedded_imgs = templates/logo.png
 
+# Option: notification_email_parser_lib
+# Notes:  Optionally set the parser library used to parse
+#         the text part of emails.
+# Values: [ lxml | html.parser ]
+#
+notification_email_parser_lib = lxml
+
 # Option: whitelist_type
 # Notes:  Set the whitelist type.
 # Values: [ db | none ]

pyquarantine/docs/templates/notification.template

@@ -10,6 +10,10 @@
     <td><b>From:</b></td>
     <td>{EMAIL_FROM}</td>
    </tr>
+   <tr>
+    <td><b>Envelope-To:</b></td>
+    <td>{EMAIL_ENVELOPE_TO}</td>
+   </tr>
    <tr>
     <td><b>To:</b></td>
     <td>{EMAIL_TO}</td>

pyquarantine/mailer.py

@@ -14,7 +14,6 @@
 
 import logging
 import smtplib
-import sys
 
 from multiprocessing import Process, Queue
 
@@ -26,6 +25,10 @@ process = None
 
 def smtp_send(smtp_host, smtp_port, mailfrom, recipient, mail):
     s = smtplib.SMTP(host=smtp_host, port=smtp_port)
+    s.ehlo()
+    if s.has_extn("STARTTLS"):
+        s.starttls()
+        s.ehlo()
     s.sendmail(mailfrom, [recipient], mail)
     s.quit()
 
@@ -38,31 +41,36 @@ def mailprocess():
     try:
         while True:
             m = queue.get()
-            if not m: break
+            if not m:
+                break
 
-            smtp_host, smtp_port, queueid, mailfrom, recipient, mail, emailtype = m
+            smtp_host, smtp_port, qid, mailfrom, recipient, mail, emailtype = m
             try:
                 smtp_send(smtp_host, smtp_port, mailfrom, recipient, mail)
             except Exception as e:
-                logger.error("{}: error while sending {} to '{}': {}".format(queueid, emailtype, recipient, e))
+                logger.error(
+                    f"{qid}: error while sending {emailtype} "
+                    f"to '{recipient}': {e}")
             else:
-                logger.info("{}: successfully sent {} to: {}".format(queueid, emailtype, recipient))
+                logger.info(
+                    f"{qid}: successfully sent {emailtype} to: {recipient}")
     except KeyboardInterrupt:
         pass
     logger.debug("mailer process terminated")
 
 
-def sendmail(smtp_host, smtp_port, queueid, mailfrom, recipients, mail, emailtype="email"):
+def sendmail(smtp_host, smtp_port, qid, mailfrom, recipients, mail,
+             emailtype="email"):
     "Send an email."
     global logger
     global process
     global queue
 
-    if type(recipients) == str:
+    if isinstance(recipients, str):
         recipients = [recipients]
 
     # start mailprocess if it is not started yet
-    if process == None:
+    if process is None:
         process = Process(target=mailprocess)
         process.daemon = True
         logger.debug("starting mailer process")
@@ -70,6 +78,9 @@ def sendmail(smtp_host, smtp_port, queueid, mailfrom, recipients, mail, emailtyp
 
     for recipient in recipients:
         try:
-            queue.put((smtp_host, smtp_port, queueid, mailfrom, recipient, mail, emailtype), timeout=30)
+            queue.put(
-        except Queue.Full as e:
+                (smtp_host, smtp_port, qid, mailfrom, recipient, mail,
+                 emailtype),
+                timeout=30)
+        except Queue.Full:
             raise RuntimeError("email queue is full")

pyquarantine/notifications.py

@@ -17,107 +17,132 @@ import logging
 import re
 
 from bs4 import BeautifulSoup
-from cgi import escape
 from collections import defaultdict
+from email.policy import default as default_policy
 from email.mime.multipart import MIMEMultipart
 from email.mime.text import MIMEText
 from email.mime.image import MIMEImage
+from html import escape
 from os.path import basename
+from urllib.parse import quote
 
 from pyquarantine import mailer
 
+
 class BaseNotification(object):
     "Notification base class"
-    def __init__(self, global_config, config, configtest=False):
+    notification_type = "base"
-        self.quarantine_name = config["name"]
+
-        self.global_config = global_config
+    def __init__(self, name, global_cfg, cfg, test=False):
-        self.config = config
+        self.name = name
         self.logger = logging.getLogger(__name__)
 
-    def notify(self, queueid, quarantine_id, mailfrom, recipients, headers, fp, subgroups=None, named_subgroups=None, synchronous=False):
+    def notify(self, qid, storage_id, mailfrom, recipients, headers,
+               fp, subgroups=None, named_subgroups=None, synchronous=False):
         fp.seek(0)
         pass
 
 
 class EMailNotification(BaseNotification):
     "Notification class to send notifications via mail."
-    _html_text = "text/html"
+    notification_type = "email"
-    _plain_text = "text/plain"
     _bad_tags = [
-            "applet",
+        "applet",
-            "embed",
+        "embed",
-            "frame",
+        "frame",
-            "frameset",
+        "frameset",
-            "head",
+        "head",
-            "iframe",
+        "iframe",
-            "script"
+        "script"
     ]
     _good_tags = [
-            "a",
+        "a",
-            "b",
+        "b",
-            "br",
+        "br",
-            "center",
+        "center",
-            "div",
+        "div",
-            "font",
+        "font",
-            "h1",
+        "h1",
-            "h2",
+        "h2",
-            "h3",
+        "h3",
-            "h4",
+        "h4",
-            "h5",
+        "h5",
-            "h6",
+        "h6",
-            "i",
+        "i",
-            "img",
+        "img",
-            "li",
+        "li",
-            "span",
+        "p",
-            "table",
+        "pre",
-            "td",
+        "span",
-            "th",
+        "table",
-            "tr",
+        "td",
-            "tt",
+        "th",
-            "u",
+        "tr",
-            "ul"
+        "tt",
+        "u",
+        "ul"
     ]
-    good_attributes = [
+    _good_attributes = [
-            "align",
+        "align",
-            "alt",
+        "alt",
-            "bgcolor",
+        "bgcolor",
-            "border",
+        "border",
-            "cellpadding",
+        "cellpadding",
-            "cellspacing",
+        "cellspacing",
-            "color",
+        "class",
-            "colspan",
+        "color",
-            "dir",
+        "colspan",
-            "face",
+        "dir",
-            "headers",
+        "face",
-            "height",
+        "headers",
-            "id",
+        "height",
-            "name",
+        "id",
-            "rowspan",
+        "name",
-            "size",
+        "rowspan",
-            "src",
+        "size",
-            "style",
+        "src",
-            "title",
+        "style",
-            "type",
+        "title",
-            "valign",
+        "type",
-            "value",
+        "valign",
-            "width"
+        "value",
+        "width"
     ]
 
-    def __init__(self, global_config, config, configtest=False):
+    def __init__(self, name, global_cfg, cfg, test=False):
-        super(EMailNotification, self).__init__(global_config, config, configtest)
+        super(EMailNotification, self).__init__(
+            name, global_cfg, cfg, test)
 
-        # check if mandatory options are present in config
+        defaults = {
-        for option in ["smtp_host", "smtp_port", "notification_email_envelope_from", "notification_email_from", "notification_email_subject", "notification_email_template", "notification_email_replacement_img", "notification_email_embedded_imgs"]:
+            "notification_email_replacement_img": "",
-            if option not in self.config.keys() and option in self.global_config.keys():
+            "notification_email_strip_images": "false",
-                self.config[option] = self.global_config[option]
+            "notification_email_parser_lib": "lxml"
-            if option not in self.config.keys():
+        }
-                raise RuntimeError("mandatory option '{}' not present in config section '{}' or 'global'".format(option, self.quarantine_name))
+        # check config
+        for opt in [
+            "notification_email_smtp_host",
+            "notification_email_smtp_port",
+            "notification_email_envelope_from",
+            "notification_email_from",
+            "notification_email_subject",
+            "notification_email_template",
+                "notification_email_embedded_imgs"] + list(defaults.keys()):
+            if opt in cfg:
+                continue
+            if opt in global_cfg:
+                cfg[opt] = global_cfg[opt]
+            elif opt in defaults:
+                cfg[opt] = defaults[opt]
+            else:
+                raise RuntimeError(
+                    f"mandatory option '{opt}' not present in config "
+                    f"section '{self.name}' or 'global'")
 
-        self.smtp_host = self.config["smtp_host"]
+        self.smtp_host = cfg["notification_email_smtp_host"]
-        self.smtp_port = self.config["smtp_port"]
+        self.smtp_port = cfg["notification_email_smtp_port"]
-        self.mailfrom = self.config["notification_email_envelope_from"]
+        self.mailfrom = cfg["notification_email_envelope_from"]
-        self.from_header = self.config["notification_email_from"]
+        self.from_header = cfg["notification_email_from"]
-        self.subject = self.config["notification_email_subject"]
+        self.subject = cfg["notification_email_subject"]
 
         testvars = defaultdict(str, test="TEST")
 
@@ -125,193 +150,264 @@ class EMailNotification(BaseNotification):
         try:
             self.from_header.format_map(testvars)
         except ValueError as e:
-            raise RuntimeError("error parsing notification_email_from: {}".format(e))
+            raise RuntimeError(
+                f"error parsing notification_email_from: {e}")
 
         # test-parse subject
         try:
             self.subject.format_map(testvars)
         except ValueError as e:
-            raise RuntimeError("error parsing notification_email_subject: {}".format(e))
+            raise RuntimeError(
+                f"error parsing notification_email_subject: {e}")
 
         # read and parse email notification template
         try:
-            self.template = open(self.config["notification_email_template"], "r").read()
+            self.template = open(
+                cfg["notification_email_template"], "r").read()
             self.template.format_map(testvars)
         except IOError as e:
-            raise RuntimeError("error reading template: {}".format(e))
+            raise RuntimeError(f"error reading template: {e}")
         except ValueError as e:
-            raise RuntimeError("error parsing template: {}".format(e))
+            raise RuntimeError(f"error parsing template: {e}")
+
+        strip_images = cfg["notification_email_strip_images"].strip().upper()
+        if strip_images in ["TRUE", "ON", "YES"]:
+            self.strip_images = True
+        elif strip_images in ["FALSE", "OFF", "NO"]:
+            self.strip_images = False
+        else:
+            raise RuntimeError(
+                "error parsing notification_email_strip_images: unknown value")
+
+        self.parser_lib = cfg["notification_email_parser_lib"].strip()
+        if self.parser_lib not in ["lxml", "html.parser"]:
+            raise RuntimeError(
+                "error parsing notification_email_parser_lib: unknown value")
 
         # read email replacement image if specified
-        replacement_img_path = self.config["notification_email_replacement_img"].strip()
+        replacement_img = cfg["notification_email_replacement_img"].strip()
-        if replacement_img_path:
+        if not self.strip_images and replacement_img:
             try:
-                self.replacement_img = MIMEImage(open(replacement_img_path, "rb").read())
+                self.replacement_img = MIMEImage(
+                    open(replacement_img, "rb").read())
             except IOError as e:
-                raise RuntimeError("error reading replacement image: {}".format(e))
+                raise RuntimeError(
+                    f"error reading replacement image: {e}")
             else:
-                self.replacement_img.add_header("Content-ID", "<removed_for_security_reasons>")
+                self.replacement_img.add_header(
+                    "Content-ID", "<removed_for_security_reasons>")
         else:
             self.replacement_img = None
 
         # read images to embed if specified
-        embedded_img_paths = [ p.strip() for p in self.config["notification_email_embedded_imgs"].split(",") if p]
+        embedded_img_paths = [
+            p.strip() for p in cfg["notification_email_embedded_imgs"].split(
+                ",") if p]
         self.embedded_imgs = []
         for img_path in embedded_img_paths:
             # read image
             try:
                 img = MIMEImage(open(img_path, "rb").read())
             except IOError as e:
-                raise RuntimeError("error reading image: {}".format(e))
+                raise RuntimeError(f"error reading image: {e}")
             else:
-                img.add_header("Content-ID", "<{}>".format(basename(img_path)))
+                filename = basename(img_path)
+                img.add_header("Content-ID", f"<{filename}>")
                 self.embedded_imgs.append(img)
 
+    def get_email_body_soup(self, qid, msg):
+        "Extract and decode email body and return it as BeautifulSoup object."
+        # try to find the body part
+        self.logger.debug(f"{qid}: trying to find email body")
+        try:
+            body = msg.get_body(preferencelist=("html", "plain"))
+        except Exception as e:
+            self.logger.error(
+                f"{qid}: an error occured in "
+                f"email.message.EmailMessage.get_body: {e}")
+            body = None
 
-    def get_text(self, queueid, part):
+        if body:
-        "Get the mail text in html form from email part."
+            charset = body.get_content_charset() or "utf-8"
-        mimetype = part.get_content_type()
+            content = body.get_payload(decode=True)
-
+            try:
-        self.logger.debug("{}: extracting content of email text part".format(queueid))
+                content = content.decode(encoding=charset, errors="replace")
-        text = part.get_payload(decode=True)
+            except LookupError:
-
+                self.logger.info(
-        if mimetype == EMailNotification._plain_text:
+                    f"{qid}: unknown encoding '{charset}', "
-            self.logger.debug("{}: content mimetype is {}, converting to {}".format(queueid, mimetype, self._html_text))
+                    f"falling back to UTF-8")
-            text = re.sub(r"^(.*)$", r"\1<br/>\n", text.decode(), flags=re.MULTILINE)
+                content = content.decode("utf-8", errors="replace")
+            content_type = body.get_content_type()
+            if content_type == "text/plain":
+                # convert text/plain to text/html
+                self.logger.debug(
+                    f"{qid}: content type is {content_type}, "
+                    f"converting to text/html")
+                content = re.sub(r"^(.*)$", r"\1<br/>",
+                                 escape(content, quote=False),
+                                 flags=re.MULTILINE)
+            else:
+                self.logger.debug(
+                    f"{qid}: content type is {content_type}")
         else:
-            self.logger.debug("{}: content mimetype is {}".format(queueid, mimetype))
+            self.logger.error(
+                f"{qid}: unable to find email body")
+            content = "ERROR: unable to find email body"
 
-        return BeautifulSoup(text, "lxml")
+        # create BeautifulSoup object
+        length = len(content)
+        self.logger.debug(
+            f"{qid}: trying to create BeatufilSoup object with "
+            f"parser lib {self.parser_lib}, "
+            f"text length is {length} bytes")
+        soup = BeautifulSoup(content, self.parser_lib)
+        self.logger.debug(
+            f"{qid}: sucessfully created BeautifulSoup object")
 
-    def get_text_multipart(self, queueid, msg, preferred=_html_text):
-        "Get the mail text of a multipart email in html form."
-        soup = None
-
-        for part in msg.get_payload():
-            mimetype = part.get_content_type()
-            if mimetype in [EMailNotification._plain_text, EMailNotification._html_text]:
-                soup = self.get_text(queueid, part)
-            elif mimetype.startswith("multipart"):
-                soup = self.get_text_multipart(queueid, part, preferred)
-
-            if soup != None and mimetype == preferred:
-                break
         return soup
 
-    def sanitize(self, queueid, soup):
+    def sanitize(self, qid, soup):
         "Sanitize mail html text."
-        self.logger.debug("{}: sanitizing email text".format(queueid))
+        self.logger.debug(f"{qid}: sanitizing email text")
 
         # completly remove bad elements
         for element in soup(EMailNotification._bad_tags):
-            self.logger.debug("{}: removing dangerous tag '{}' and its content".format(queueid, element.name))
+            self.logger.debug(
+                f"{qid}: removing dangerous tag '{element.name}' "
+                f"and its content")
             element.extract()
 
         # remove not whitelisted elements, but keep their content
         for element in soup.find_all(True):
             if element.name not in EMailNotification._good_tags:
-                self.logger.debug("{}: removing tag '{}', keep its content".format(queueid, element.name))
+                self.logger.debug(
+                    f"{qid}: removing tag '{element.name}', keep its content")
                 element.replaceWithChildren()
 
         # remove not whitelisted attributes
         for element in soup.find_all(True):
-            for attribute in element.attrs.keys():
+            for attribute in list(element.attrs.keys()):
-                if attribute not in EMailNotification.good_attributes:
+                if attribute not in EMailNotification._good_attributes:
                     if element.name == "a" and attribute == "href":
-                        self.logger.debug("{}: setting attribute href to '#' on tag '{}'".format(queueid, element.name))
+                        self.logger.debug(
+                            f"{qid}: setting attribute href to '#' "
+                            f"on tag '{element.name}'")
                         element["href"] = "#"
                     else:
-                        self.logger.debug("{}: removing attribute '{}' from tag '{}'".format(queueid, attribute, element.name))
+                        self.logger.debug(
+                            f"{qid}: removing attribute '{attribute}' "
+                            f"from tag '{element.name}'")
                         del(element.attrs[attribute])
         return soup
 
-    def get_html_text_part(self, queueid, msg):
+    def notify(self, qid, storage_id, mailfrom, recipients, headers, fp,
-        "Get the mail text of an email in html form."
+               subgroups=None, named_subgroups=None, synchronous=False):
-        soup = None
-        mimetype = msg.get_content_type()
-
-        self.logger.debug("{}: trying to find text part of email".format(queueid))
-        if mimetype in [EMailNotification._plain_text, EMailNotification._html_text]:
-            soup = self.get_text(queueid, msg)
-        elif mimetype.startswith("multipart"):
-            soup = self.get_text_multipart(queueid, msg)
-
-        if soup == None:
-            self.logger.error("{}: unable to extract text part of email".format(queueid))
-            text = "ERROR: unable to extract text from email body"
-            soup = BeautifulSoup(text, "lxml", "UTF-8")
-
-        return soup
-
-    def notify(self, queueid, quarantine_id, mailfrom, recipients, headers, fp, subgroups=None, named_subgroups=None, synchronous=False):
         "Notify recipients via email."
-        super(EMailNotification, self).notify(queueid, quarantine_id, mailfrom, recipients, headers, fp, subgroups, named_subgroups, synchronous)
+        super(
+            EMailNotification,
+            self).notify(
+            qid,
+            storage_id,
+            mailfrom,
+            recipients,
+            headers,
+            fp,
+            subgroups,
+            named_subgroups,
+            synchronous)
 
-        # extract html text from email
+        # extract body from email
-        self.logger.debug("{}: extraction email text from original email".format(queueid))
+        soup = self.get_email_body_soup(
-        soup = self.get_html_text_part(queueid, email.message_from_binary_file(fp))
+            qid, email.message_from_binary_file(fp, policy=default_policy))
 
         # replace picture sources
         image_replaced = False
-        if self.replacement_img:
+        if self.strip_images:
+            self.logger.debug(
+                f"{qid}: looking for images to strip")
             for element in soup("img"):
                 if "src" in element.attrs.keys():
-                    self.logger.debug("{}: replacing image: {}".format(queueid, element["src"]))
+                    self.logger.debug(
+                        f"{qid}: strip image: {element['src']}")
+                element.extract()
+        elif self.replacement_img:
+            self.logger.debug(
+                f"{qid}: looking for images to replace")
+            for element in soup("img"):
+                if "src" in element.attrs.keys():
+                    self.logger.debug(
+                        f"{qid}: replacing image: {element['src']}")
                 element["src"] = "cid:removed_for_security_reasons"
                 image_replaced = True
 
         # sanitizing email text of original email
-        sanitized_text = self.sanitize(queueid, soup)
+        sanitized_text = self.sanitize(qid, soup)
         del soup
 
         # sending email notifications
         for recipient in recipients:
-            self.logger.debug("{}: generating notification email for '{}'".format(queueid, recipient))
+            self.logger.debug(
-            self.logger.debug("{}: parsing email template".format(queueid))
+                f"{qid}: generating notification email for '{recipient}'")
+            self.logger.debug(f"{qid}: parsing email template")
 
             # generate dict containing all template variables
-            variables = defaultdict(str,
+            variables = defaultdict(
+                    str,
                     EMAIL_HTML_TEXT=sanitized_text,
-                    EMAIL_FROM=escape(headers["from"]),
+                    EMAIL_FROM=escape(headers["from"], quote=False),
-                    EMAIL_ENVELOPE_FROM=escape(mailfrom),
+                    EMAIL_ENVELOPE_FROM=escape(mailfrom, quote=False),
-                    EMAIL_TO=escape(recipient),
+                    EMAIL_ENVELOPE_FROM_URL=escape(quote(mailfrom),
-                    EMAIL_SUBJECT=escape(headers["subject"]),
+                                                   quote=False),
-                    EMAIL_QUARANTINE_ID=quarantine_id)
+                    EMAIL_TO=escape(headers["to"], quote=False),
+                    EMAIL_ENVELOPE_TO=escape(recipient, quote=False),
+                    EMAIL_ENVELOPE_TO_URL=escape(quote(recipient)),
+                    EMAIL_SUBJECT=escape(headers["subject"], quote=False),
+                    EMAIL_QUARANTINE_ID=storage_id)
 
             if subgroups:
                 number = 0
                 for subgroup in subgroups:
-                    variables["SUBGROUP_{}".format(number)] = escape(subgroup)
+                    variables[f"SUBGROUP_{number}"] = escape(subgroup,
+                                                             quote=False)
             if named_subgroups:
-                for key, value in named_subgroups.items(): named_subgroups[key] = escape(value)
+                for key, value in named_subgroups.items():
+                    named_subgroups[key] = escape(value, quote=False)
                 variables.update(named_subgroups)
 
             # parse template
             htmltext = self.template.format_map(variables)
 
             msg = MIMEMultipart('related')
-            msg["Subject"] = self.subject.format_map(variables)
+            msg["From"] = self.from_header.format_map(
-            msg["From"] = "<{}>".format(self.from_header.format_map(variables))
+                defaultdict(str, EMAIL_FROM=headers["from"]))
-            msg["To"] = "<{}>".format(recipient)
+            msg["To"] = headers["to"]
+            msg["Subject"] = self.subject.format_map(
+                defaultdict(str, EMAIL_SUBJECT=headers["subject"]))
             msg["Date"] = email.utils.formatdate()
             msg.attach(MIMEText(htmltext, "html", 'UTF-8'))
 
             if image_replaced:
-                self.logger.debug("{}: attaching notification_replacement_img".format(queueid))
+                self.logger.debug(
+                    f"{qid}: attaching notification_replacement_img")
                 msg.attach(self.replacement_img)
 
             for img in self.embedded_imgs:
-                self.logger.debug("{}: attaching imgage".format(queueid))
+                self.logger.debug(f"{qid}: attaching imgage")
                 msg.attach(img)
 
-            self.logger.debug("{}: sending notification email to: {}".format(queueid, recipient))
+            self.logger.debug(
+                f"{qid}: sending notification email to: {recipient}")
             if synchronous:
                 try:
-                    mailer.smtp_send(self.smtp_host, self.smtp_port, self.mailfrom, recipient, msg.as_string())
+                    mailer.smtp_send(self.smtp_host, self.smtp_port,
+                                     self.mailfrom, recipient, msg.as_string())
                 except Exception as e:
-                    raise RuntimeError("error while sending email to '{}': {}".format(recipient, e))
+                    raise RuntimeError(
+                        f"error while sending email to '{recipient}': {e}")
             else:
-                mailer.sendmail(self.smtp_host, self.smtp_port, queueid, self.mailfrom, recipient, msg.as_string(), "notification email")
+                mailer.sendmail(self.smtp_host, self.smtp_port, qid,
+                                self.mailfrom, recipient, msg.as_string(),
+                                "notification email")
 
 
 # list of notification types and their related notification classes

pyquarantine/quarantines.py

@@ -1,267 +0,0 @@
-# PyQuarantine-Milter is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-# 
-# PyQuarantine-Milter is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the 
-# GNU General Public License for more details.
-# 
-# You should have received a copy of the GNU General Public License
-# along with PyQuarantineMilter.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-import json
-import logging
-import os
-
-from calendar import timegm
-from datetime import datetime
-from glob import glob
-from shutil import copyfileobj
-from time import gmtime
-
-from pyquarantine import mailer
-
-
-class BaseQuarantine(object):
-    "Quarantine base class"
-    def __init__(self, global_config, config, configtest=False):
-        self.name = config["name"]
-        self.global_config = global_config
-        self.config = config
-        self.logger = logging.getLogger(__name__)
-
-    def add(self, queueid, mailfrom, recipients, headers, fp, subgroups=None, named_subgroups=None):
-        "Add email to quarantine."
-        fp.seek(0)
-        return ""
-
-    def find(self, mailfrom=None, recipients=None, older_than=None):
-        "Find emails in quarantine."
-        return
-    
-    def get_metadata(self, quarantine_id):
-        "Return metadata of quarantined email."
-        return
-
-    def delete(self, quarantine_id, recipient=None):
-        "Delete email from quarantine."
-        return
-
-    def notify(self, quarantine_id, recipient=None):
-        "Notify recipient about email in quarantine."
-        if not self.config["notification_obj"]:
-            raise RuntimeError("notification type is set to None, unable to send notifications")
-        return
-
-    def release(self, quarantine_id, recipient=None):
-        "Release email from quarantine."
-        return
-
-
-class FileQuarantine(BaseQuarantine):
-    "Quarantine class to store mails on filesystem."
-    def __init__(self, global_config, config, configtest=False):
-        super(FileQuarantine, self).__init__(global_config, config, configtest)
-
-        # check if mandatory options are present in config
-        for option in ["quarantine_directory"]:
-            if option not in self.config.keys() and option in self.global_config.keys():
-                self.config[option] = self.global_config[option]
-            if option not in self.config.keys():
-                raise RuntimeError("mandatory option '{}' not present in config section '{}' or 'global'".format(option, self.name))
-        self.directory = self.config["quarantine_directory"]
-
-        # check if quarantine directory exists and is writable
-        if not os.path.isdir(self.directory) or not os.access(self.directory, os.W_OK):
-            raise RuntimeError("file quarantine directory '{}' does not exist or is not writable".format(self.directory))
-        self._metadata_suffix = ".metadata"
-
-    def _save_datafile(self, quarantine_id, fp):
-        datafile = os.path.join(self.directory, quarantine_id)
-        try:
-            with open(datafile, "wb") as f:
-                copyfileobj(fp, f)
-        except IOError as e:
-            raise RuntimeError("unable save data file: {}".format(e))
-
-    def _save_metafile(self, quarantine_id, metadata):
-        metafile = os.path.join(self.directory, "{}{}".format(quarantine_id, self._metadata_suffix))
-        try:
-            with open(metafile, "w") as f:
-                json.dump(metadata, f, indent=2)
-        except IOError as e:
-            raise RuntimeError("unable to save metadata file: {}".format(e))
-
-    def _remove(self, quarantine_id):
-        datafile = os.path.join(self.directory, quarantine_id)
-        metafile = "{}{}".format(datafile, self._metadata_suffix)
-
-        try:
-            os.remove(metafile)
-        except IOError as e:
-            raise RuntimeError("unable to remove metadata file: {}".format(e))
-
-        try:
-            os.remove(datafile)
-        except IOError as e:
-            raise RuntimeError("unable to remove data file: {}".format(e))
-
-    def add(self, queueid, mailfrom, recipients, headers, fp, subgroups=None, named_subgroups=None):
-        "Add email to file quarantine and return quarantine-id."
-        super(FileQuarantine, self).add(queueid, mailfrom, recipients, headers, fp, subgroups, named_subgroups)
-        quarantine_id = "{}_{}".format(datetime.now().strftime("%Y%m%d%H%M%S"), queueid)
-
-        # save mail
-        self._save_datafile(quarantine_id, fp)
-
-        # save metadata
-        metadata = {
-            "mailfrom": mailfrom,
-            "recipients": recipients,
-            "headers": headers,
-            "date": timegm(gmtime()),
-            "queue_id": queueid,
-            "subgroups": subgroups,
-            "named_subgroups": named_subgroups
-        }
-        try:
-            self._save_metafile(quarantine_id, metadata)
-        except RuntimeError as e:
-            datafile = os.path.join(self.directory, quarantine_id)
-            os.remove(datafile)
-            raise e
-
-        return quarantine_id
-
-    def get_metadata(self, quarantine_id):
-        "Return metadata of quarantined email."
-        super(FileQuarantine, self).get_metadata(quarantine_id)
-
-        metafile = os.path.join(self.directory, "{}{}".format(quarantine_id, self._metadata_suffix))
-        if not os.path.isfile(metafile):
-            raise RuntimeError("invalid quarantine id '{}'".format(quarantine_id))
-
-        try:
-            with open(metafile, "r") as f:
-                metadata = json.load(f)
-        except IOError as e:
-            raise RuntimeError("unable to read metadata file: {}".format(e))
-        except json.JSONDecodeError as e:
-            raise RuntimeError("invalid meta file '{}': {}".format(metafile, e))
-
-        return metadata
-
-    def find(self, mailfrom=None, recipients=None, older_than=None):
-        "Find emails in quarantine."
-        super(FileQuarantine, self).find(mailfrom, recipients, older_than)
-        if type(mailfrom) == str: mailfrom = [mailfrom]
-        if type(recipients) == str: recipients = [recipients]
-
-        emails = {}
-        metafiles = glob(os.path.join(self.directory, "*{}".format(self._metadata_suffix)))
-        for metafile in metafiles:
-            if not os.path.isfile(metafile): continue
-
-            quarantine_id = os.path.basename(metafile[:-len(self._metadata_suffix)])
-            metadata = self.get_metadata(quarantine_id)
-            if older_than != None:
-                if timegm(gmtime()) - metadata["date"] < (older_than * 24 * 3600):
-                    continue
-
-            if mailfrom != None:
-                if metadata["mailfrom"] not in mailfrom:
-                    continue
-
-            if recipients != None:
-                if len(recipients) == 1 and recipients[0] not in metadata["recipients"]:
-                    continue
-                elif len(set(recipients + metadata["recipients"])) == len(recipients + metadata["recipients"]):
-                    continue
-
-            emails[quarantine_id] = metadata
-
-        return emails
-
-    def delete(self, quarantine_id, recipient=None):
-        "Delete email in quarantine."
-        super(FileQuarantine, self).delete(quarantine_id, recipient)
-
-        try:
-            metadata = self.get_metadata(quarantine_id)
-        except RuntimeError as e:
-            raise RuntimeError("unable to delete email: {}".format(e))
-
-        if recipient == None:
-            self._remove(quarantine_id)
-        else:
-            if recipient not in metadata["recipients"]:
-                raise RuntimeError("invalid recipient '{}'".format(recipient))
-
-            metadata["recipients"].remove(recipient)
-            if not metadata["recipients"]:
-                self._remove(quarantine_id)
-            else:
-                self._save_metafile(quarantine_id, metadata)
-
-    def notify(self, quarantine_id, recipient=None):
-        "Notify recipient about email in quarantine."
-        super(FileQuarantine, self).notify(quarantine_id, recipient)
-
-        try:
-            metadata = self.get_metadata(quarantine_id)
-        except RuntimeError as e:
-            raise RuntimeError("unable to release email: {}".format(e))
-
-        if recipient != None:
-            if recipient not in metadata["recipients"]:
-                raise RuntimeError("invalid recipient '{}'".format(recipient))
-            recipients = [recipient]
-        else:
-            recipients = metadata["recipients"]
-
-        datafile = os.path.join(self.directory, quarantine_id)
-        try:
-            with open(datafile, "rb") as fp:
-                self.config["notification_obj"].notify(metadata["queue_id"], quarantine_id, metadata["mailfrom"], recipients, metadata["headers"], fp,
-                        metadata["subgroups"], metadata["named_subgroups"], synchronous=True)
-        except IOError as e:
-            raise(RuntimeError("unable to read data file: {}".format(e)))
-
-
-    def release(self, quarantine_id, recipient=None):
-        "Release email from quarantine."
-        super(FileQuarantine, self).release(quarantine_id, recipient)
-
-        try:
-            metadata = self.get_metadata(quarantine_id)
-        except RuntimeError as e:
-            raise RuntimeError("unable to release email: {}".format(e))
-
-        if recipient != None:
-            if recipient not in metadata["recipients"]:
-                raise RuntimeError("invalid recipient '{}'".format(recipient))
-            recipients = [recipient]
-        else:
-            recipients = metadata["recipients"]
-
-        datafile = os.path.join(self.directory, quarantine_id)
-        try:
-            with open(datafile, "rb") as f:
-                mail = f.read()
-        except IOError as e:
-            raise RuntimeError("unable to read data file: {}".format(e))
-
-        for recipient in recipients:
-            try:
-                mailer.smtp_send(self.config["smtp_host"], self.config["smtp_port"], metadata["mailfrom"], recipient, mail)
-            except Exception as e:
-                raise RuntimeError("error while sending email to '{}': {}".format(recipient, e))
-
-            self.delete(quarantine_id, recipient)
-
-
-# list of quarantine types and their related quarantine classes
-TYPES = {"file": FileQuarantine}

pyquarantine/run.py

@@ -22,17 +22,38 @@ import sys
 
 import pyquarantine
 
+from pyquarantine import __version__ as version
+
 
 def main():
     "Run PyQuarantine-Milter."
     # parse command line
-    parser = argparse.ArgumentParser(description="PyQuarantine milter daemon",
+    parser = argparse.ArgumentParser(
-            formatter_class=lambda prog: argparse.HelpFormatter(prog, max_help_position=45, width=140))
+        description="PyQuarantine milter daemon",
-    parser.add_argument("-c", "--config", help="List of config files to read.", nargs="+",
+        formatter_class=lambda prog: argparse.HelpFormatter(
-            default=pyquarantine.QuarantineMilter.get_configfiles())
+            prog, max_help_position=45, width=140))
-    parser.add_argument("-s", "--socket", help="Socket used to communicate with the MTA.", required=True)
+    parser.add_argument(
-    parser.add_argument("-d", "--debug", help="Log debugging messages.", action="store_true")
+        "-c", "--config",
-    parser.add_argument("-t", "--test", help="Check configuration.", action="store_true")
+        help="List of config files to read.",
+        nargs="+",
+        default=pyquarantine.QuarantineMilter.get_cfg_files())
+    parser.add_argument(
+        "-s", "--socket",
+        help="Socket used to communicate with the MTA.",
+        default="inet:8899@127.0.0.1")
+    parser.add_argument(
+        "-d", "--debug",
+        help="Log debugging messages.",
+        action="store_true")
+    parser.add_argument(
+        "-t", "--test",
+        help="Check configuration.",
+        action="store_true")
+    parser.add_argument(
+        "-v", "--version",
+        help="Print version.",
+        action="version",
+        version=f"%(prog)s ({version})")
     args = parser.parse_args()
 
     # setup logging
@@ -41,63 +62,64 @@ def main():
     syslog_name = logname
     if args.debug:
         loglevel = logging.DEBUG
-        logname = "{}[%(name)s]".format(logname)
+        logname = f"{logname}[%(name)s]"
-        syslog_name = "{}: [%(name)s] %(levelname)s".format(syslog_name)
+        syslog_name = f"{syslog_name}: [%(name)s] %(levelname)s"
 
     # set config files for milter class
-    pyquarantine.QuarantineMilter.set_configfiles(args.config)
+    pyquarantine.QuarantineMilter.set_cfg_files(args.config)
     root_logger = logging.getLogger()
     root_logger.setLevel(loglevel)
 
     # setup console log
     stdouthandler = logging.StreamHandler(sys.stdout)
     stdouthandler.setLevel(logging.DEBUG)
-    formatter = logging.Formatter("%(message)s".format(logname))
+    formatter = logging.Formatter("%(message)s")
     stdouthandler.setFormatter(formatter)
     root_logger.addHandler(stdouthandler)
     logger = logging.getLogger(__name__)
     if args.test:
         try:
-            pyquarantine.generate_milter_config(args.test)
+            pyquarantine.setup_milter(test=args.test)
             print("Configuration ok")
         except RuntimeError as e:
             logger.error(e)
             sys.exit(255)
         else:
             sys.exit(0)
-    formatter = logging.Formatter("%(asctime)s {}: [%(levelname)s] %(message)s".format(logname), datefmt="%Y-%m-%d %H:%M:%S")
+    formatter = logging.Formatter(
+        f"%(asctime)s {logname}: [%(levelname)s] %(message)s",
+        datefmt="%Y-%m-%d %H:%M:%S")
     stdouthandler.setFormatter(formatter)
 
     # setup syslog
-    sysloghandler = logging.handlers.SysLogHandler(address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_MAIL)
+    sysloghandler = logging.handlers.SysLogHandler(
+        address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_MAIL)
     sysloghandler.setLevel(loglevel)
-    formatter = logging.Formatter("{}: %(message)s".format(syslog_name))
+    formatter = logging.Formatter(f"{syslog_name}: %(message)s")
     sysloghandler.setFormatter(formatter)
     root_logger.addHandler(sysloghandler)
 
     logger.info("PyQuarantine-Milter starting")
     try:
         # generate milter config
-        global_config, config = pyquarantine.generate_milter_config()
+        pyquarantine.setup_milter()
     except RuntimeError as e:
         logger.error(e)
         sys.exit(255)
 
-    pyquarantine.QuarantineMilter.global_config = global_config
-    pyquarantine.QuarantineMilter.config = config
-
     # register to have the Milter factory create instances of your class:
     Milter.factory = pyquarantine.QuarantineMilter
     Milter.set_exception_policy(Milter.TEMPFAIL)
-    #Milter.set_flags(0)       # tell sendmail which features we use
 
     # run milter
     rc = 0
     try:
-        Milter.runmilter("pyquarantine-milter", socketname=args.socket, timeout=300)
+        Milter.runmilter("pyquarantine-milter", socketname=args.socket,
+                         timeout=300)
     except Milter.milter.error as e:
         logger.error(e)
         rc = 255
+
     pyquarantine.mailer.queue.put(None)
     logger.info("PyQuarantine-Milter terminated")
     sys.exit(rc)

pyquarantine/storages.py

@@ -0,0 +1,252 @@
+# PyQuarantine-Milter is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# PyQuarantine-Milter is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with PyQuarantineMilter.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import json
+import logging
+import os
+
+from calendar import timegm
+from datetime import datetime
+from glob import glob
+from shutil import copyfileobj
+from time import gmtime
+
+
+class BaseMailStorage(object):
+    "Mail storage base class"
+    storage_type = "base"
+
+    def __init__(self, name, global_cfg, cfg, test=False):
+        self.name = name
+        self.logger = logging.getLogger(__name__)
+
+    def add(self, qid, mailfrom, recipients, headers,
+            fp, subgroups=None, named_subgroups=None):
+        "Add email to storage."
+        fp.seek(0)
+        return ""
+
+    def find(self, mailfrom=None, recipients=None, older_than=None):
+        "Find emails in storage."
+        return
+
+    def get_metadata(self, storage_id):
+        "Return metadata of email in storage."
+        return
+
+    def delete(self, storage_id, recipients=None):
+        "Delete email from storage."
+        return
+
+    def get_mail(self, storage_id):
+        "Return a file pointer to the email and metadata."
+        return
+
+
+class FileMailStorage(BaseMailStorage):
+    "Storage class to store mails on filesystem."
+    storage_type = "file"
+
+    def __init__(self, name, global_cfg, cfg, test=False):
+        super(FileMailStorage, self).__init__(name, global_cfg, cfg, test)
+
+        defaults = {}
+        # check config
+
+        for opt in ["storage_directory"] + list(defaults.keys()):
+            if opt in cfg:
+                continue
+            if opt in global_cfg:
+                cfg[opt] = global_cfg[opt]
+            elif opt in defaults:
+                cfg[opt] = defaults[opt]
+            else:
+                raise RuntimeError(
+                    f"mandatory option '{opt}' not present in config "
+                    f"section '{self.name}' or 'global'")
+        self.directory = cfg["storage_directory"]
+
+        # check if quarantine directory exists and is writable
+        if not os.path.isdir(self.directory) or not os.access(
+                self.directory, os.W_OK):
+            raise RuntimeError(
+                f"file quarantine directory '{self.directory}' does "
+                f"not exist or is not writable")
+        self._metadata_suffix = ".metadata"
+
+    def _save_datafile(self, storage_id, fp):
+        datafile = os.path.join(self.directory, storage_id)
+        try:
+            with open(datafile, "wb") as f:
+                copyfileobj(fp, f)
+        except IOError as e:
+            raise RuntimeError(f"unable save data file: {e}")
+
+    def _save_metafile(self, storage_id, metadata):
+        metafile = os.path.join(
+            self.directory, f"{storage_id}{self._metadata_suffix}")
+        try:
+            with open(metafile, "w") as f:
+                json.dump(metadata, f, indent=2)
+        except IOError as e:
+            raise RuntimeError(f"unable to save metadata file: {e}")
+
+    def _remove(self, storage_id):
+        datafile = os.path.join(self.directory, storage_id)
+        metafile = f"{datafile}{self._metadata_suffix}"
+
+        try:
+            os.remove(metafile)
+        except IOError as e:
+            raise RuntimeError(f"unable to remove metadata file: {e}")
+
+        try:
+            os.remove(datafile)
+        except IOError as e:
+            raise RuntimeError(f"unable to remove data file: {e}")
+
+    def add(self, qid, mailfrom, recipients, headers,
+            fp, subgroups=None, named_subgroups=None):
+        "Add email to file storage and return storage id."
+        super(
+            FileMailStorage,
+            self).add(
+            qid,
+            mailfrom,
+            recipients,
+            headers,
+            fp,
+            subgroups,
+            named_subgroups)
+        timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
+        storage_id = f"{timestamp}_{qid}"
+
+        # save mail
+        self._save_datafile(storage_id, fp)
+
+        # save metadata
+        metadata = {
+            "mailfrom": mailfrom,
+            "recipients": recipients,
+            "headers": headers,
+            "date": timegm(gmtime()),
+            "queue_id": qid,
+            "subgroups": subgroups,
+            "named_subgroups": named_subgroups
+        }
+        try:
+            self._save_metafile(storage_id, metadata)
+        except RuntimeError as e:
+            datafile = os.path.join(self.directory, storage_id)
+            os.remove(datafile)
+            raise e
+
+        return storage_id
+
+    def get_metadata(self, storage_id):
+        "Return metadata of email in storage."
+        super(FileMailStorage, self).get_metadata(storage_id)
+
+        metafile = os.path.join(
+            self.directory, f"{storage_id}{self._metadata_suffix}")
+        if not os.path.isfile(metafile):
+            raise RuntimeError(
+                f"invalid storage id '{storage_id}'")
+
+        try:
+            with open(metafile, "r") as f:
+                metadata = json.load(f)
+        except IOError as e:
+            raise RuntimeError(f"unable to read metadata file: {e}")
+        except json.JSONDecodeError as e:
+            raise RuntimeError(
+                f"invalid metafile '{metafile}': {e}")
+
+        return metadata
+
+    def find(self, mailfrom=None, recipients=None, older_than=None):
+        "Find emails in storage."
+        super(FileMailStorage, self).find(mailfrom, recipients, older_than)
+        if isinstance(mailfrom, str):
+            mailfrom = [mailfrom]
+        if isinstance(recipients, str):
+            recipients = [recipients]
+
+        emails = {}
+        metafiles = glob(os.path.join(
+            self.directory, f"*{self._metadata_suffix}"))
+        for metafile in metafiles:
+            if not os.path.isfile(metafile):
+                continue
+
+            storage_id = os.path.basename(
+                metafile[:-len(self._metadata_suffix)])
+            metadata = self.get_metadata(storage_id)
+            if older_than is not None:
+                if timegm(gmtime()) - metadata["date"] < (older_than * 86400):
+                    continue
+
+            if mailfrom is not None:
+                if metadata["mailfrom"] not in mailfrom:
+                    continue
+
+            if recipients is not None:
+                if len(recipients) == 1 and \
+                        recipients[0] not in metadata["recipients"]:
+                    continue
+                elif len(set(recipients + metadata["recipients"])) == \
+                        len(recipients + metadata["recipients"]):
+                    continue
+
+            emails[storage_id] = metadata
+
+        return emails
+
+    def delete(self, storage_id, recipients=None):
+        "Delete email from storage."
+        super(FileMailStorage, self).delete(storage_id, recipients)
+
+        try:
+            metadata = self.get_metadata(storage_id)
+        except RuntimeError as e:
+            raise RuntimeError(f"unable to delete email: {e}")
+
+        if not recipients:
+            self._remove(storage_id)
+        else:
+            if type(recipients) == str:
+                recipients = [recipients]
+            for recipient in recipients:
+                if recipient not in metadata["recipients"]:
+                    raise RuntimeError(f"invalid recipient '{recipient}'")
+                metadata["recipients"].remove(recipient)
+                if not metadata["recipients"]:
+                    self._remove(storage_id)
+                else:
+                    self._save_metafile(storage_id, metadata)
+
+    def get_mail(self, storage_id):
+        super(FileMailStorage, self).get_mail(storage_id)
+
+        metadata = self.get_metadata(storage_id)
+        datafile = os.path.join(self.directory, storage_id)
+        try:
+            fp = open(datafile, "rb")
+        except IOError as e:
+            raise RuntimeError(f"unable to open email data file: {e}")
+        return (fp, metadata)
+
+
+# list of storage types and their related storage classes
+TYPES = {"file": FileMailStorage}

pyquarantine/whitelists.py

@@ -12,24 +12,25 @@
 # along with PyQuarantineMilter.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-import datetime
 import logging
 import peewee
 import re
-import sys
 
+from datetime import datetime
 from playhouse.db_url import connect
 
 
 class WhitelistBase(object):
     "Whitelist base class"
-    def __init__(self, global_config, config, configtest=False):
+
-        self.global_config = global_config
+    whitelist_type = "base"
-        self.config = config
+
-        self.configtest = configtest
+    def __init__(self, name, global_cfg, cfg, test=False):
-        self.name = config["name"]
+        self.name = name
+        self.test = test
         self.logger = logging.getLogger(__name__)
-        self.valid_entry_regex = re.compile(r"^[a-zA-Z0-9_.+-]*?(@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+)?$")
+        self.valid_entry_regex = re.compile(
+            r"^[a-zA-Z0-9_.=+-]*?(@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+)?$")
 
     def check(self, mailfrom, recipient):
         "Check if mailfrom/recipient combination is whitelisted."
@@ -56,45 +57,67 @@ class WhitelistBase(object):
 class WhitelistModel(peewee.Model):
     mailfrom = peewee.CharField()
     recipient = peewee.CharField()
-    created = peewee.DateTimeField(default=datetime.datetime.now)
+    created = peewee.DateTimeField(default=datetime.now)
-    last_used = peewee.DateTimeField(default=datetime.datetime.now)
+    last_used = peewee.DateTimeField(default=datetime.now)
     comment = peewee.TextField(default="")
     permanent = peewee.BooleanField(default=False)
 
 
 class Meta(object):
     indexes = (
-        (('mailfrom', 'recipient'), True), # trailing comma is mandatory if only one index should be created
+        # trailing comma is mandatory if only one index should be created
+        (('mailfrom', 'recipient'), True),
     )
 
 
 class DatabaseWhitelist(WhitelistBase):
     "Whitelist class to store whitelist in a database"
+    whitelist_type = "db"
     _db_connections = {}
     _db_tables = {}
 
-    def __init__(self, global_config, config, configtest=False):
+    def __init__(self, name, global_cfg, cfg, test=False):
-        super(DatabaseWhitelist, self).__init__(global_config, config, configtest)
+        super(
+            DatabaseWhitelist,
+            self).__init__(
+            global_cfg,
+            cfg,
+            test)
 
-        # check if mandatory options are present in config
+        defaults = {}
-        for option in ["whitelist_db_connection", "whitelist_db_table"]:
-            if option not in self.config.keys() and option in self.global_config.keys():
-                self.config[option] = self.global_config[option]
-            if option not in self.config.keys():
-                raise RuntimeError("mandatory option '{}' not present in config section '{}' or 'global'".format(option, self.name))
 
-        tablename = self.config["whitelist_db_table"]
+        # check config
-        connection_string = self.config["whitelist_db_connection"]
+        for opt in ["whitelist_db_connection",
+                    "whitelist_db_table"] + list(defaults.keys()):
+            if opt in cfg:
+                continue
+            if opt in global_cfg:
+                cfg[opt] = global_cfg[opt]
+            elif opt in defaults:
+                cfg[opt] = defaults[opt]
+            else:
+                raise RuntimeError(
+                    f"mandatory option '{opt}' not present in config "
+                    f"section '{self.name}' or 'global'")
+
+        tablename = cfg["whitelist_db_table"]
+        connection_string = cfg["whitelist_db_connection"]
 
         if connection_string in DatabaseWhitelist._db_connections.keys():
             db = DatabaseWhitelist._db_connections[connection_string]
         else:
             try:
                 # connect to database
-                self.logger.debug("connecting to database '{}'".format(re.sub(r"(.*?://.*?):.*?(@.*)", r"\1:<PASSWORD>\2", connection_string)))
+                conn = re.sub(
+                    r"(.*?://.*?):.*?(@.*)",
+                    r"\1:<PASSWORD>\2",
+                    connection_string)
+                self.logger.debug(
+                    f"connecting to database '{conn}'")
                 db = connect(connection_string)
             except Exception as e:
-                raise RuntimeError("unable to connect to database: {}".format(e))
+                raise RuntimeError(
+                    f"unable to connect to database: {e}")
 
             DatabaseWhitelist._db_connections[connection_string] = db
 
@@ -102,7 +125,7 @@ class DatabaseWhitelist(WhitelistBase):
         self.meta = Meta
         self.meta.database = db
         self.meta.table_name = tablename
-        self.model = type("WhitelistModel_{}".format(self.name), (WhitelistModel,), {
+        self.model = type(f"WhitelistModel_{self.name}", (WhitelistModel,), {
             "Meta": self.meta
         })
 
@@ -111,11 +134,12 @@ class DatabaseWhitelist(WhitelistBase):
 
         if tablename not in DatabaseWhitelist._db_tables[connection_string]:
             DatabaseWhitelist._db_tables[connection_string].append(tablename)
-            if not self.configtest:
+            if not self.test:
                 try:
                     db.create_tables([self.model])
                 except Exception as e:
-                    raise RuntimeError("unable to initialize table '{}': {}".format(tablename, e))
+                    raise RuntimeError(
+                        f"unable to initialize table '{tablename}': {e}")
 
     def _entry_to_dict(self, entry):
         result = {}
@@ -144,23 +168,30 @@ class DatabaseWhitelist(WhitelistBase):
         super(DatabaseWhitelist, self).check(mailfrom, recipient)
 
         # generate list of possible mailfroms
-        self.logger.debug("query database for whitelist entries from <{}> to <{}>".format(mailfrom, recipient))
+        self.logger.debug(
+            f"query database for whitelist entries from <{mailfrom}> "
+            f"to <{recipient}>")
         mailfroms = [""]
         if "@" in mailfrom and not mailfrom.startswith("@"):
-            mailfroms.append("@{}".format(mailfrom.split("@")[1]))
+            domain = mailfrom.split("@")[1]
+            mailfroms.append(f"@{domain}")
         mailfroms.append(mailfrom)
 
         # generate list of possible recipients
         recipients = [""]
         if "@" in recipient and not recipient.startswith("@"):
-            recipients.append("@{}".format(recipient.split("@")[1]))
+            domain = recipient.split("@")[1]
+            recipients.append(f"@{domain}")
         recipients.append(recipient)
 
         # query the database
         try:
-            entries = list(self.model.select().where(self.model.mailfrom.in_(mailfroms), self.model.recipient.in_(recipients)))
+            entries = list(
+                self.model.select().where(
+                    self.model.mailfrom.in_(mailfroms),
+                    self.model.recipient.in_(recipients)))
         except Exception as e:
-            raise RuntimeError("unable to query database: {}".format(e))
+            raise RuntimeError(f"unable to query database: {e}")
 
         if not entries:
             # no whitelist entry found
@@ -171,7 +202,7 @@ class DatabaseWhitelist(WhitelistBase):
 
         # use entry with the highest weight
         entry = entries[0]
-        entry.last_used = datetime.datetime.now()
+        entry.last_used = datetime.now()
         entry.save()
         result = {}
         for entry in entries:
@@ -183,38 +214,51 @@ class DatabaseWhitelist(WhitelistBase):
         "Find whitelist entries."
         super(DatabaseWhitelist, self).find(mailfrom, recipients, older_than)
 
-        if type(mailfrom) == str: mailfrom = [mailfrom]
+        if isinstance(mailfrom, str):
-        if type(recipients) == str: recipients = [recipients]
+            mailfrom = [mailfrom]
+        if isinstance(recipients, str):
+            recipients = [recipients]
 
         entries = {}
         try:
             for entry in list(self.model.select()):
-                if older_than != None: 
+                if older_than is not None:
-                    if (datetime.datetime.now() - entry.last_used).total_seconds() < (older_than * 24 * 3600):
+                    delta = (datetime.now() - entry.last_used).total_seconds()
+                    if delta < (older_than * 86400):
                         continue
 
-                if mailfrom != None:
+                if mailfrom is not None:
                     if entry.mailfrom not in mailfrom:
                         continue
 
-                if recipients != None:
+                if recipients is not None:
                     if entry.recipient not in recipients:
                         continue
 
                 entries.update(self._entry_to_dict(entry))
         except Exception as e:
-            raise RuntimeError("unable to query database: {}".format(e))
+            raise RuntimeError(f"unable to query database: {e}")
 
         return entries
 
     def add(self, mailfrom, recipient, comment, permanent):
         "Add entry to whitelist."
-        super(DatabaseWhitelist, self).add(mailfrom, recipient, comment, permanent)
+        super(
+            DatabaseWhitelist,
+            self).add(
+            mailfrom,
+            recipient,
+            comment,
+            permanent)
 
         try:
-            self.model.create(mailfrom=mailfrom, recipient=recipient, comment=comment, permanent=permanent)
+            self.model.create(
+                mailfrom=mailfrom,
+                recipient=recipient,
+                comment=comment,
+                permanent=permanent)
         except Exception as e:
-            raise RuntimeError("unable to add entry to database: {}".format(e))
+            raise RuntimeError(f"unable to add entry to database: {e}")
 
     def delete(self, whitelist_id):
         "Delete entry from whitelist."
@@ -224,7 +268,8 @@ class DatabaseWhitelist(WhitelistBase):
             query = self.model.delete().where(self.model.id == whitelist_id)
             deleted = query.execute()
         except Exception as e:
-            raise RuntimeError("unable to delete entry from database: {}".format(e))
+            raise RuntimeError(
+                f"unable to delete entry from database: {e}")
 
         if deleted == 0:
             raise RuntimeError("invalid whitelist id")
@@ -239,15 +284,20 @@ class WhitelistCache(object):
             self.check(whitelist, mailfrom, recipient)
 
     def check(self, whitelist, mailfrom, recipient):
-        if whitelist not in self.cache.keys(): self.cache[whitelist] = {}
+        if whitelist not in self.cache.keys():
-        if recipient not in self.cache[whitelist].keys(): self.cache[whitelist][recipient] = None
+            self.cache[whitelist] = {}
-        if self.cache[whitelist][recipient] == None:
+        if recipient not in self.cache[whitelist].keys():
-            self.cache[whitelist][recipient] = whitelist.check(mailfrom, recipient)
+            self.cache[whitelist][recipient] = None
+        if self.cache[whitelist][recipient] is None:
+            self.cache[whitelist][recipient] = whitelist.check(
+                mailfrom, recipient)
         return self.cache[whitelist][recipient]
 
-    def get_whitelisted_recipients(self, whitelist, mailfrom, recipients):
+    def get_recipients(self, whitelist, mailfrom, recipients):
         self.load(whitelist, mailfrom, recipients)
-        return list(filter(lambda x: self.cache[whitelist][x], self.cache[whitelist].keys()))
+        return list(filter(
+            lambda x: self.cache[whitelist][x],
+            self.cache[whitelist].keys()))
 
 
 # list of whitelist types and their related whitelist classes

setup.cfg

@@ -0,0 +1,2 @@
+[metadata]
+version = attr: pyquarantine.__version__

setup.py

@@ -4,8 +4,8 @@ def read_file(fname):
     with open(fname, 'r') as f:
         return f.read()
 
+
 setup(name = "pyquarantine",
-    version = "0.0.1",
     author = "Thomas Oettli",
     author_email = "spacefreak@noop.ch",
     description = "A pymilter based sendmail/postfix pre-queue filter.",
@@ -19,7 +19,7 @@ setup(name = "pyquarantine",
         #   3 - Alpha
         #   4 - Beta
         #   5 - Production/Stable
-        "Development Status :: 3 - Alpha",
+        "Development Status :: 4 - Beta",
         "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
         "Operating System :: OS Independent",
         "Programming Language :: Python",
@@ -32,6 +32,6 @@ setup(name = "pyquarantine",
             "pyquarantine=pyquarantine.cli:main"
         ]
     },
-    install_requires = ["pymilter", "netaddr", "beautifulsoup4", "peewee"],
+    install_requires = ["pymilter", "netaddr", "beautifulsoup4[lxml]", "peewee"],
-    python_requires = ">=3"
+    python_requires = ">=3.6"
 )