massive refactoring of the source

README.md

@@ -2,7 +2,7 @@
 A pymilter based sendmail/postfix pre-queue filter with the ability to add, remove and modify e-mail headers.  
 The project is currently in beta status, but it is already used in a productive enterprise environment which processes about a million e-mails per month.  
 
-The basic idea is to define rules with conditions and modifications which are processed when all conditions are true.
+The basic idea is to define rules with conditions and actions which are processed when all conditions are true.
 
 ## Dependencies
 Pymodmilter is depending on these python packages, but they are installed automatically if you are working with pip.
@@ -10,7 +10,7 @@ Pymodmilter is depending on these python packages, but they are installed automa
 * [netaddr](https://github.com/drkjam/netaddr/)
 
 ## Installation
-* Install pymodmilter with pip and copy the example configuration file.
+* Install pymodmilter with pip and copy the example config file.
 ```sh
 pip install pymodmilter
 cp /etc/pymodmilter/pymodmilter.conf.example /etc/pymodmilter/pymodmilter.conf
@@ -18,96 +18,107 @@ cp /etc/pymodmilter/pymodmilter.conf.example /etc/pymodmilter/pymodmilter.conf
 * Modify /etc/pymodmilter/pymodmilter.conf according to your needs.
 
 ## Configuration options
-Pymodmilter uses a configuration file in JSON format. The options are described below. Make a copy of the [example configuration file](https://github.com/spacefreak86/pymodmilter/blob/master/docs/pymodmilter.conf.example) in the  [docs](https://github.com/spacefreak86/pymodmilter/tree/master/docs) folder to start with.  
+Pymodmilter uses a config file in JSON format. The config file has to be JSON valid with the exception of allowed comment lines starting with **#**. The options are described below.  
-Rules and modifications are processed in the given order.
+Rules and actions are processed in the given order.
 
 ### Global
-The following global configuration options are optional:
+Config options in **global** section:
-* **socket**  
+* **socket** (optional)  
-  The socket used to communicate with the MTA.
+  The socket used to communicate with the MTA. If it is not specified in the config, it has to be set as command line option.
-* **local_addrs**  
+* **local_addrs** (optional)  
-  A list of hosts and network addresses which are considered local. It is used to for the condition option [local](#Conditions). This option may be overriden within a rule object.
+  A list of hosts and network addresses which are considered local. It is used to for the condition option [local](#Conditions).  
-* **log**  
+  Default: **::1/128, 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16**
-  Enable or disable logging. This option may be overriden by a rule or modification object.
+* **loglevel**  (optional)
+  Set the log level. This option may be overriden by any rule or action object. Possible values are:
+  * **error**  
+  * **warning**  
+  * **info**  
+  * **debug**  
+  Default: **info**
+* **pretend** (optional)
+  Pretend actions, for test purposes. This option may be overriden by any rule or action object.
 
 ### Rules
-The following configuration options are mandatory for each rule:
+Config options for **rule** objects:
-* **modifications**  
+* **name**  (optional)  
-  A list of modification objects which are processed in the given order.
-
-The following configuration options are optional for each rule:
-* **name**  
   Name of the rule.  
-* **conditions**  
+  Default: **Rule #n**
-  A list of conditions which all have to be true to process the rule.
+* **actions**  
-* **local_addrs**  
+  A list of action objects which are processed in the given order.
+* **conditions** (optional)  
+  A list of conditions which all have to be true to process the actions.
+* **loglevel** (optional)  
   As described above in the [Global](#Global) section.
-* **log**  
+* **pretend** (optional)  
   As described above in the [Global](#Global) section.
-* **pretend**  
-  Just pretend to make the modifications, for test purposes.
 
-### Modifications
+### Actions
-The following configuration options are mandatory for each modification:
+Config options for **action** objects:
+* **name** (optional)  
+  Name of the action.
+  Default: **Action #n**
 * **type**  
-  Set the modification type. Possible values are:
+  Action type. Possible values are:
   * **add_header**
   * **del_header**
   * **mod_header**
   * **add_disclaimer**
+* **conditions** (optional)  
+  A list of conditions which all have to be true to process the action.
+* **pretend** (optional)  
+  Just pretend all actions of this rule, for test purposes.
+* **loglevel** (optional)  
+  As described above in the [Global](#Global) section.
 
-The following configuration options are mandatory based on the modification type in use.
+Config options for **add_header** actions:
-* **add_header**  
   * **header**  
     Name of the header.
   * **value**  
     Value of the header.
 
-* **del_header**  
+Config options for **del_header** actions:
   * **header**  
-    Regular expression to match against header lines.
+    Regular expression to match against header names.
+  * **value** (optional)
+    Regular expression to match against the headers value.
 
-* **mod_header**  
+Config options for **mod_header** actions:
   * **header**  
-    Regular expression to match against header lines.
+    Regular expression to match against header names.
-  * **search**  
+  * **search** (optional)  
-    Regular expression to match against the value of header lines. You may use subgroups or named subgroups (python syntax) to include parts of the original value in the new value.
+    Regular expression to match against header values. You may use subgroups or named subgroups (python syntax) to include parts of the original value in the new value.
   * **value**  
     New value of the header.
 
-* **add_disclaimer**
+Config options for **add_disclaimer** actions:
   * **action**  
     Action to perform with the disclaimer. Possible values are:
     * append
     * prepend
-  * **html_template**  
+  * **html_file**  
-    Path to a file that contains the html representation of the disclaimer.
+    Path to a file which contains the html representation of the disclaimer.
-  * **text_template**  
+  * **text_file**  
-    Path to a file that contains the text representation of the disclaimer.
+    Path to a file which contains the text representation of the disclaimer.
-  * **error_policy**  
+  * **error_policy** (optional)  
-    Set what should be done if the disclaimer could not be added (e.g. no body text found). Possible values are:
+    Set the error policy in case the disclaimer cannot be added (e.g. if no body part is present in the e-mail). Possible values are:
     * wrap  
-      The original e-mail will be attached to a new one containing the disclaimer.
+      A new e-mail body is generated with the disclaimer as body and the original e-mail attached.
     * ignore  
       Ignore the error and do nothing.
     * reject  
       Reject the e-mail.
+    Default: **wrap**
 
-The following configuration options are optional for each modification:
-* **name**  
-  Name of the modification.
-* **log**  
-  As described above in the global object section.
 
 ### Conditions
-The following condition options are optional:
+Config options for **conditions** objects:
-* **local**  
+* **local** (optional)  
   If set to true, the rule is only executed for e-mails originating from addresses defined in local_addrs and vice versa.
-* **hosts**  
+* **hosts** (optional)  
   A list of hosts and network addresses for which the rule should be executed.
-* **envfrom**  
+* **envfrom** (optional)  
   A regular expression to match against the evenlope-from addresses for which the rule should be executed.
-* **envto**  
+* **envto** (optional)  
   A regular expression to match against all evenlope-to addresses. All addresses must match to fulfill the condition.
 
 ## Developer information

docs/pymodmilter.conf.example

@@ -30,16 +30,23 @@
         #
         "local_addrs": ["127.0.0.0/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"],
 
-        # Option: log
+        # Option: loglevel
+        # Type:   String
+        # Notes:  Set loglevel for rules and actions.
+        # Value:  [ error | warning | info | debug ]
+        #
+        "loglevel": "info",
+
+        # Option: pretend
         # Type:   Bool
-        # Notes:  Enable or disable logging of rules and modifications.
+        # Notes:  Just pretend to do the actions, for test purposes.
         # Value:  [ true | false ]
         #
-        "log": true
+        "pretend": true
     },
 
     # Section: rules
-    # Notes:   Rules and related modifications.
+    # Notes:   Rules and related actions.
     #
     "rules": [
         {
@@ -85,10 +92,10 @@
                 "envto": "^postmaster@.+$"
             },
 
-            # Section: modifications
+            # Section: actions
-            # Notes:   Modifications of the rule.
+            # Notes:   Actions of the rule.
             #
-            "modifications": [
+            "actions": [
                 {
                     # Option: name
                     # Type:   String
@@ -116,7 +123,7 @@
                     # Notes:  Value of the header.
                     # Value:  [ VALUE ]
                     #
-                    "value": "true",
+                    "value": "true"
                 }, {
                     "name": "modify_subject",
 
@@ -127,7 +134,7 @@
                     # Notes:  Regular expression to match against header lines (e.g. Subject: Test-Subject).
                     # Value:  [ REGEX ]
                     #
-                    "header": "^Subject:",
+                    "header": "^Subject$",
 
                     # Option: search
                     # Type:   String
@@ -151,7 +158,7 @@
                     # Notes:  Regular expression to match against header lines (e.g. Subject: Test-Subject).
                     # Value:  [ REGEX ]
                     #
-                    "header": "^Received:"
+                    "header": "^Received$"
                 }, {
                     "name": "add_disclaimer",
 
@@ -164,19 +171,19 @@
                     #
                     "action": "prepend",
 
-                    # Option: html_template
+                    # Option: html_file
                     # Type:   String
-                    # Notes:  Path to a file that contains the html representation of the disclaimer.
+                    # Notes:  Path to a file which contains the html representation of the disclaimer.
                     # Value:  [ FILE_PATH ]
                     #
-                    "html_template": "/etc/pymodmilter/templates/disclaimer_html.template",
+                    "html_file": "/etc/pymodmilter/templates/disclaimer_html.template",
 
-                    # Option: text_template
+                    # Option: text_file
                     # Type:   String
-                    # Notes:  Path to a file that contains the text representation of the disclaimer.
+                    # Notes:  Path to a file which contains the text representation of the disclaimer.
                     # Value:  [ FILE_PATH ]
                     #
-                    "text_template": "/etc/pymodmilter/templates/disclaimer_text.template",
+                    "text_file": "/etc/pymodmilter/templates/disclaimer_text.template",
 
                     # Option: error_policy
                     # Type:   String
@@ -185,14 +192,7 @@
                     #
                     "error_policy": "wrap"
                 }
-            ],
+            ]
-
-            # Option: pretend
-            # Type:   Bool
-            # Notes:  Just pretend to do the modifications, for test purposes.
-            # Value:  [ true | false ]
-            #
-            "pretend": true
         }
     ]
 }

docs/templates/disclaimer_html.template

@@ -6,4 +6,4 @@
   </td>
  </tr>
 </table>
-<br /><br />
+<br/><br/>

pymodmilter/__init__.py

@@ -14,28 +14,23 @@
 
 __all__ = [
     "make_header",
-    "replace_illegal_chars",
+    "actions",
+    "conditions",
     "run",
     "version",
-    "Modification",
+    "CustomLogger",
     "Rule",
     "ModifyMilter"]
 
 import Milter
 import logging
-import logging.handlers
-import re
 
 from Milter.utils import parse_addr
-from bs4 import BeautifulSoup
-from copy import copy
 from email.charset import Charset
 from email.header import Header, decode_header
-from email import message_from_binary_file
-from email.message import MIMEPart
-from email.policy import default as default_policy, SMTP
 from io import BytesIO
-from netaddr import IPAddress, IPNetwork, AddrFormatError
+
+from pymodmilter.conditions import Conditions
 
 
 def make_header(decoded_seq, maxlinelen=None, header_name=None,
@@ -60,524 +55,101 @@ def make_header(decoded_seq, maxlinelen=None, header_name=None,
     return h
 
 
-def replace_illegal_chars(string):
+class CustomLogger(logging.LoggerAdapter):
-    """Replace illegal characters in header values."""
+    def process(self, msg, kwargs):
-    return string.replace(
+        if "name" in self.extra:
-        "\x00", "").replace(
+            msg = "{}: {}".format(self.extra["name"], msg)
-        "\r", "").replace(
-        "\n", "")
 
+        if "qid" in self.extra:
+            msg = "{}: {}".format(self.extra["qid"], msg)
 
-class Modification:
+        if self.logger.getEffectiveLevel() != logging.DEBUG:
-    """
+            msg = msg.replace("\n", "").replace("\r", "")
-    Modification to implement certain modifications on e-mails.
 
-    Each modification function returns the necessary changes for ModifyMilter
+        return msg, kwargs
-    so they can be applied to the email passing the MTA.
-    """
-
-    def __init__(self, name, mod_type, log, **params):
-        self.logger = logging.getLogger(__name__)
-        self.logger.debug(f"initializing modification '{name}'")
-        self.name = name
-        self.log = log
-        # needs for each modification type
-        self.types = {
-            "add_header": {
-                "needs": ["headers"]},
-            "del_header": {
-                "needs": ["headers"]},
-            "mod_header": {
-                "needs": ["headers"]},
-            "add_disclaimer": {
-                "needs": ["headers", "data"]}}
-
-        if mod_type not in self.types:
-            raise RuntimeError(
-                f"{self.name}: invalid modification type '{mod_type}'")
-
-        self.mod_type = mod_type
-
-        try:
-            if mod_type == "add_header":
-                self.header = params["header"]
-                self.value = params["value"]
-            elif mod_type in ["del_header", "mod_header"]:
-                try:
-                    self.header = re.compile(
-                        params["header"],
-                        re.MULTILINE + re.DOTALL + re.IGNORECASE)
-                except re.error as e:
-                    raise RuntimeError(
-                        f"{self.name}: unable to parse regex of "
-                        f"option 'header': {e}")
-
-                if mod_type == "mod_header":
-                    try:
-                        self.search = re.compile(
-                            params["search"],
-                            re.MULTILINE + re.DOTALL + re.IGNORECASE)
-                    except re.error as e:
-                        raise RuntimeError(
-                            f"{self.name}: unable to parse regex of "
-                            f"option 'search': {e}")
-
-                    self.value = params["value"]
-            elif mod_type == "add_disclaimer":
-                if params["action"] not in ["append", "prepend"]:
-                    raise RuntimeError(
-                        f"{self.name}: unknown action specified")
-
-                self.action = params["action"]
-
-                if params["error_policy"] not in ["wrap", "ignore", "reject"]:
-                    raise RuntimeError(
-                        f"{self.name}: unknown error_policy specified")
-
-                self.error_policy = params["error_policy"]
-
-                try:
-                    with open(params["html_template"], "r") as f:
-                        self.html = BeautifulSoup(f.read(), "html.parser")
-                        body = self.html.find('body')
-                        if body:
-                            # just use content within the body tag if present
-                            self.html = body
-                    with open(params["text_template"], "r") as f:
-                        self.text = f.read()
-                except IOError as e:
-                    raise RuntimeError(f"unable to read template: {e}")
-        except KeyError as e:
-            raise RuntimeError(
-                f"{self.name}: mandatory configuration option not found: {e}")
-
-    def needs(self):
-        """Return the needs of this modification to work."""
-        return self.types[self.mod_type]["needs"]
-
-    def add_header(self, qid, headers, header, value, pos=-1):
-        """Add header to email."""
-        hdr = f"{header}: {value}"
-        if self.log:
-            self.logger.info(
-                f"{qid}: {self.name}: add_header: {hdr[0:70]}")
-        else:
-            self.logger.debug(
-                f"{qid}: {self.name}: add_header: {hdr}")
-
-        headers.append((header, value))
-        params = [header, value, pos]
-        return [("add_header", *params)]
-
-    def mod_header(self, qid, headers, header, search, replace):
-        """Modify an email header."""
-        if isinstance(header, str):
-            header = re.compile(
-                header, re.MULTILINE + re.DOTALL + re.IGNORECASE)
-
-        if isinstance(search, str):
-            search = re.compile(
-                search, re.MULTILINE + re.DOTALL + re.IGNORECASE)
-
-        changes = []
-        index = 0
-        occurrences = {}
-        # iterate a copy of headers because headers may be modified
-        for name, value in headers.copy():
-            # keep track of the occurrence of each header
-            # needed by Milter.Base.chgheader
-            if name not in occurrences.keys():
-                occurrences[name] = 1
-            else:
-                occurrences[name] += 1
-
-            hdr = f"{name}: {value}"
-            if header.search(hdr):
-                new_value = search.sub(replace, value).strip()
-                if new_value == "":
-                    self.logger.warning(
-                        f"{qid}: {self.name}: mod_header: resulting value is "
-                        f"empty, skip modification")
-                elif value != new_value:
-                    old_hdr = hdr
-                    hdr = f"{name}: {new_value}"
-                    if self.log:
-                        self.logger.info(
-                            f"{qid}: {self.name}: mod_header: "
-                            f"{old_hdr[0:70]}: {hdr[0:70]}")
-                    else:
-                        self.logger.debug(
-                            f"{qid}: {self.name}: mod_header: "
-                            f"(occ. {occurrences[name]}): {old_hdr}: "
-                            f"{hdr}")
-
-                    headers[index] = (name, new_value)
-                    params = [name, new_value, occurrences[name]]
-                    changes.append(("mod_header", *params))
-
-            index += 1
-
-        return changes
-
-    def del_header(self, qid, headers, header):
-        """Delete an email header."""
-        if isinstance(header, str):
-            header = re.compile(
-                header, re.MULTILINE + re.DOTALL + re.IGNORECASE)
-
-        changes = []
-        index = 0
-        occurrences = {}
-        # iterate a copy of headers because headers may be modified
-        for name, value in headers.copy():
-            # keep track of the occurrence of each header,
-            # needed by Milter.Base.chgheader
-            if name not in occurrences.keys():
-                occurrences[name] = 1
-            else:
-                occurrences[name] += 1
-
-            hdr = f"{name}: {value}"
-            if header.search(hdr):
-                if self.log:
-                    self.logger.info(
-                        f"{qid}: {self.name}: del_header: "
-                        f"{hdr[0:70]}")
-                else:
-                    self.logger.debug(
-                        f"{qid}: {self.name}: del_header: "
-                        f"(occ. {occurrences[name]}): {hdr}")
-
-                del headers[index]
-                params = [name, "", occurrences[name]]
-                changes.append(("mod_header", *params))
-                index -= 1
-                occurrences[name] -= 1
-
-            index += 1
-
-        return changes
-
-    def add_disclaimer(self, qid, headers, fp, text_template, html_template,
-                       error_policy):
-        """Append or prepend a disclaimer to the email body."""
-        changes = []
-
-        fp.seek(0)
-        msg = message_from_binary_file(fp, policy=default_policy)
-
-        html_body = None
-        text_body = None
-        update_headers = False
-
-        try:
-            html_body = msg.get_body(preferencelist=("html"))
-            text_body = msg.get_body(preferencelist=("plain"))
-        except Exception as e:
-            self.logger.error(
-                f"{qid}: {self.name}: an error occured in "
-                f"email.message.EmailMessage.get_body: {e}")
-
-        if html_body is None and text_body is None:
-            if self.error_policy == "ignore":
-                self.logger.info(
-                    f"{qid}: {self.name}: unable to find email body, "
-                    f"ignore according to policy")
-                return changes
-            elif self.error_policy == "reject":
-                self.logger.info(
-                    f"{qid}: {self.name}: unable to find email body, "
-                    f"reject message according to policy")
-                return [
-                    ("reject", "Message rejected due to missing email body")]
-
-            self.logger.info(
-                f"{qid}: {self.name}: unable to find email body, "
-                f"wrapping original email in a new message envelope")
-            msg = MIMEPart()
-            msg.add_header("MIME-Version", "1.0")
-            msg.set_content(
-                "Please see the original email attached.")
-            msg.add_alternative(
-                "Please see the original email attached.",
-                subtype="html")
-            fp.seek(0)
-            msg.add_attachment(
-                fp.read(), maintype="plain", subtype="text",
-                filename=f"{qid}.eml")
-            html_body = msg.get_body(preferencelist=("html"))
-            text_body = msg.get_body(preferencelist=("plain"))
-            # content and mime headers may have to be updated because
-            # a new message has been created
-            update_headers = True
-        elif not msg.is_multipart():
-            # content and mime headers may have to be updated because
-            # we operate on a non-multipart email
-            update_headers = True
-
-        if text_body is not None:
-            if self.log:
-                self.logger.info(
-                    f"{qid}: {self.name}: {self.action} text disclaimer")
-            else:
-                self.logger.debug(
-                    f"{qid}: {self.name}: {self.action} text disclaimer")
-
-            text = text_body.get_content()
-            if self.action == "prepend":
-                text = f"{text_template}{text}"
-            else:
-                text = f"{text}{text_template}"
-
-            text_body.set_content(
-                text.encode(), maintype="text", subtype="plain")
-            text_body.set_param("charset", "UTF-8", header="Content-Type")
-
-        if html_body is not None:
-            if self.log:
-                self.logger.info(
-                    f"{qid}: {self.name}: {self.action} html disclaimer")
-            else:
-                self.logger.debug(
-                    f"{qid}: {self.name}: {self.action} html disclaimer")
-
-            soup = BeautifulSoup(html_body.get_content(), "html.parser")
-            body = soup.find('body')
-            if body:
-                # work within the body tag if it is present
-                soup = body
-
-            if self.action == "prepend":
-                soup.insert(0, copy(html_template))
-            else:
-                soup.append(html_template)
-
-            html_body.set_content(
-                str(soup).encode(), maintype="text", subtype="html")
-            html_body.set_param("charset", "UTF-8", header="Content-Type")
-
-        if update_headers:
-            for name, value in msg.items():
-                name_lower = name.lower()
-                if not name_lower.startswith("content-") and \
-                        name_lower != "mime-version":
-                    continue
-
-                defined = False
-                for n, v in headers:
-                    if n.lower() == name_lower:
-                        changes += self.mod_header(
-                            qid, headers, f"^{n}:", ".*", value)
-                        defined = True
-                        break
-
-                if not defined:
-                    changes += self.add_header(
-                        qid, headers, name, value)
-
-        msg = msg.as_string(policy=SMTP).encode("ascii", errors="replace")
-        fp.seek(0)
-        fp.write(msg)
-        fp.truncate()
-        body_pos = msg.find(b"\r\n\r\n") + 2
-        changes.append(("mod_body", body_pos))
-        return changes
-
-    def execute(self, qid, headers, fp):
-        """
-        Execute configured modification.
-        """
-        changes = []
-
-        if self.mod_type == "add_header":
-            changes = self.add_header(
-                qid, headers, self.header, self.value)
-        elif self.mod_type == "mod_header":
-            changes = self.mod_header(
-                qid, headers, self.header, self.search, self.value)
-        elif self.mod_type == "del_header":
-            changes = self.del_header(
-                qid, headers, self.header)
-        elif self.mod_type == "add_disclaimer":
-            changes = self.add_disclaimer(
-                qid, headers, fp, self.text,
-                self.html, self.error_policy)
-
-        return changes
 
 
 class Rule:
     """
-    Rule to implement multiple modifications on emails based on conditions.
+    Rule to implement multiple actions on emails.
     """
 
-    def __init__(self, name, modifications, local_addrs, log, conditions={},
+    def __init__(self, name, local_addrs, conditions, actions, pretend=False,
-                 pretend=False):
+                 loglevel=logging.INFO):
-        self.logger = logging.getLogger(__name__)
+        logger = logging.getLogger(name)
-        if pretend:
+        self.logger = CustomLogger(logger, {"name": name})
-            self.name = f"{name} (pretend)"
+        self.logger.setLevel(loglevel)
-        else:
-            self.name = name
 
-        self.logger.debug(f"initializing rule '{self.name}'")
+        if logger is None:
-        self.log = log
+            logger = logging.getLogger(__name__)
+
+        self.logger = CustomLogger(logger, {"name": name})
+        self.conditions = Conditions(
+            local_addrs=local_addrs,
+            args=conditions,
+            logger=self.logger)
+        self.actions = actions
         self.pretend = pretend
+
         self._needs = []
-        self._local_addrs = []
+        for action in actions:
-
+            for need in action.needs():
-        try:
-            for addr in local_addrs:
-                self._local_addrs.append(IPNetwork(addr))
-        except AddrFormatError as e:
-            raise RuntimeError(
-                f"{self.name}: unable to parse entry of "
-                f"option local_addrs: {e}")
-
-        self.conditions = {}
-        for option, value in conditions.items():
-            if option == "local":
-                self.conditions[option] = value
-                self.logger.debug(
-                    f"{self.name}: added condition: {option} = {value}")
-            elif option == "hosts":
-                self.conditions[option] = []
-                try:
-                    for host in value:
-                        self.conditions[option].append(IPNetwork(host))
-                except AddrFormatError as e:
-                    raise RuntimeError(
-                        f"{self.name}: unable to parse entry of "
-                        f"condition '{option}': {e}")
-
-                self.logger.debug(
-                    f"{self.name}: added condition: {option} = {value}")
-            elif option in ["envfrom", "envto"]:
-                try:
-                    self.conditions[option] = re.compile(value, re.IGNORECASE)
-                except re.error as e:
-                    raise RuntimeError(
-                        f"{self.name}: unable to parse regex of "
-                        f"condition '{option}': {e}")
-
-                self.logger.debug(
-                    f"{self.name}: added condition: {option} = {value}")
-
-        self.modifications = []
-        for mod_idx, mod in enumerate(modifications):
-            params = {}
-            if "name" not in mod:
-                mod["name"] = f"Modification #{mod_idx}"
-
-            if self.name:
-                params["name"] = f"{self.name}: {mod['name']}"
-            else:
-                params["name"] = mod["name"]
-
-            if "log" in mod:
-                params["log"] = mod["log"]
-            else:
-                params["log"] = self.log
-
-            if "type" in mod:
-                params["mod_type"] = mod["type"]
-            else:
-                raise RuntimeError(
-                    f"{params['name']}: mandatory config "
-                    f"option 'type' not found")
-
-            for param in [
-                    "header", "search", "value", "action", "html_template",
-                    "text_template", "error_policy"]:
-                if param in mod:
-                    params[param] = mod[param]
-
-            modification = Modification(**params)
-            for need in modification.needs():
                 if need not in self._needs:
                     self._needs.append(need)
 
-            self.modifications.append(modification)
+        self.logger.debug("needs: {}".format(", ".join(self._needs)))
-            self.logger.debug(
-                f"{self.name}: added modification: {mod['name']}")
-
-        self.logger.debug(
-            f"{self.name}: rule needs: {self._needs}")
 
     def needs(self):
         """Return the needs of this rule."""
         return self._needs
 
-    def ignore_host(self, host):
+    def ignores(self, host=None, envfrom=None, envto=None):
-        """Check if host is ignored by this rule."""
+        args = {}
-        ip = IPAddress(host)
 
-        if "local" in self.conditions:
+        if host is not None:
-            is_local = False
+            args["host"] = host
-            for addr in self._local_addrs:
-                if ip in addr:
-                    is_local = True
-                    break
 
-            if is_local != self.conditions["local"]:
+        if envfrom is not None:
-                return True
+            args["envfrom"] = envfrom
 
-        if "hosts" in self.conditions:
+        if envto is not None:
-            # check if host is in list
+            args["envto"] = envto
-            for accepted in self.conditions["hosts"]:
+
-                if ip in accepted:
+        if self.conditions.match(args):
+            for action in self.actions:
+                if action.conditions.match(args):
                     return False
 
-            return True
+        return True
 
-        return False
+    def execute(self, milter, pretend=None):
+        """Execute all actions of this rule."""
+        if pretend is None:
+            pretend = self.pretend
 
-    def ignore_envfrom(self, envfrom):
+        for action in self.actions:
-        """Check if envelope-from address is ignored by this rule."""
+            milter_action = action.execute(milter)
-        if "envfrom" in self.conditions:
+            if milter_action is not None:
-            if not self.conditions["envfrom"].search(envfrom):
+                return milter_action
-                return True
-
-        return False
-
-    def ignore_envto(self, envto):
-        """Check if envelope-to address is ignored by this rule."""
-        if "envto" in self.conditions:
-            if not isinstance(envto, set):
-                envto = set(envto)
-            for to in envto:
-                if not self.conditions["envto"].search(to):
-                    return True
-
-        return False
-
-    def execute(self, qid, headers, data):
-        """Execute all modifications of this rule."""
-        changes = []
-        if self.log:
-            self.logger.info(f"{qid}: executing rule '{self.name}'")
-        else:
-            self.logger.debug(f"{qid}: executing rule '{self.name}'")
-
-        for mod in self.modifications:
-            self.logger.debug(f"{qid}: executing modification '{mod.name}'")
-            changes += mod.execute(qid, headers, data)
-
-        if self.pretend:
-            changes = []
-        return changes
 
 
 class ModifyMilter(Milter.Base):
     """ModifyMilter based on Milter.Base to implement milter communication"""
 
     _rules = []
+    _loglevel = logging.INFO
 
     @staticmethod
     def set_rules(rules):
         ModifyMilter._rules = rules
 
+    def set_loglevel(level):
+        ModifyMilter._loglevel = level
+
     def __init__(self):
         self.logger = logging.getLogger(__name__)
+        self.logger.setLevel(ModifyMilter._loglevel)
+
         # save rules, it must not change during runtime
         self.rules = ModifyMilter._rules.copy()
 
@@ -585,13 +157,10 @@ class ModifyMilter(Milter.Base):
         self.logger.debug(
             f"accepted milter connection from {hostaddr[0]} "
             f"port {hostaddr[1]}")
-        ip = IPAddress(hostaddr[0])
 
         # remove rules which ignore this host
         for rule in self.rules.copy():
-            if rule.ignore_host(ip):
+            if rule.ignores(host=hostaddr[0]):
-                self.logger.debug(
-                    f"host {hostaddr[0]} is ignored by rule '{rule.name}'")
                 self.rules.remove(rule)
 
         if not self.rules:
@@ -605,10 +174,7 @@ class ModifyMilter(Milter.Base):
     def envfrom(self, mailfrom, *str):
         mailfrom = "@".join(parse_addr(mailfrom)).lower()
         for rule in self.rules.copy():
-            if rule.ignore_envfrom(mailfrom):
+            if rule.ignores(envfrom=mailfrom):
-                self.logger.debug(
-                    f"envelope-from {mailfrom} is ignored by "
-                    f"rule '{rule.name}'")
                 self.rules.remove(rule)
 
         if not self.rules:
@@ -628,10 +194,7 @@ class ModifyMilter(Milter.Base):
     def data(self):
         try:
             for rule in self.rules.copy():
-                if rule.ignore_envto(self.recipients):
+                if rule.ignores(envto=[*self.recipients]):
-                    self.logger.debug(
-                        f"envelope-to addresses are ignored by "
-                        f"rule '{rule.name}'")
                     self.rules.remove(rule)
 
             if not self.rules:
@@ -641,17 +204,18 @@ class ModifyMilter(Milter.Base):
                 return Milter.ACCEPT
 
             self.qid = self.getsymval('i')
-            self.logger.debug(f"{self.qid}: received queue-id from MTA")
+            self.logger = CustomLogger(self.logger, {"qid": self.qid})
-            self.headers = None
+            self.logger.debug("received queue-id from MTA")
+            self.fields = None
             self.fp = None
             for rule in self.rules:
-                if "headers" in rule.needs() and self.headers is None:
+                if "fields" in rule.needs() and self.fields is None:
-                    self.headers = []
+                    self.fields = []
 
-                if "data" in rule.needs() and self.fp is None:
+                if "body" in rule.needs() and self.fp is None:
                     self.fp = BytesIO()
 
-                if None not in [self.headers, self.fp]:
+                if None not in [self.fields, self.fp]:
                     break
 
         except Exception as e:
@@ -663,21 +227,15 @@ class ModifyMilter(Milter.Base):
 
     def header(self, name, value):
         try:
-            # remove surrogates from value
+            if self.fields is not None:
-            value = value.encode(
+                # remove surrogates from value
-                errors="surrogateescape").decode(errors="replace")
+                value = value.encode(
-            if self.fp is not None:
+                    errors="surrogateescape").decode(errors="replace")
-                self.fp.write(f"{name}: {value}\r\n".encode(
+                self.logger.debug(f"received header: {name}: {value}")
-                    encoding="ascii", errors="replace"))
-
-            if self.headers is not None:
-                self.logger.debug(f"{self.qid}: received header: "
-                                  f"{name}: {value}")
                 header = make_header(decode_header(value), errors="replace")
                 value = str(header).replace("\x00", "")
-                self.logger.debug(
+                self.logger.debug(f"decoded header: {name}: {value}")
-                    f"{self.qid}: decoded header: {name}: {value}")
+                self.fields.append((name, value))
-                self.headers.append((name, value))
         except Exception as e:
             self.logger.exception(
                 f"an exception occured in header function: {e}")
@@ -685,17 +243,6 @@ class ModifyMilter(Milter.Base):
 
         return Milter.CONTINUE
 
-    def eoh(self):
-        try:
-            if self.fp is not None:
-                self.fp.write(b"\r\n")
-        except Exception as e:
-            self.logger.exception(
-                f"an exception occured in eoh function: {e}")
-            return Milter.TEMPFAIL
-
-        return Milter.CONTINUE
-
     def body(self, chunk):
         try:
             if self.fp is not None:
@@ -709,43 +256,20 @@ class ModifyMilter(Milter.Base):
 
     def eom(self):
         try:
-            changes = []
             for rule in self.rules:
-                changes += rule.execute(self.qid, self.headers, self.fp)
+                milter_action = rule.execute(self)
 
-            mod_body_pos = None
+                if milter_action is not None:
-            for mod_type, *params in changes:
+                    if milter_action["action"] == "reject":
-                if mod_type in ["add_header", "mod_header", "del_header"]:
+                        self.setreply("554", "5.7.0", milter_action["reason"])
-                    header, value, occurrence = params
+                        return Milter.REJECT
-                    enc_value = replace_illegal_chars(
-                        Header(s=value).encode())
-                    if mod_type == "add_header":
-                        self.logger.debug(f"{self.qid}: milter: add "
-                                          f"header: {header}: {enc_value}")
-                        self.addheader(header, enc_value, occurrence)
-                    else:
-                        if enc_value == "":
-                            self.logger.debug(
-                                f"{self.qid}: milter: delete "
-                                f"header (occ. {occurrence}): "
-                                f"{header}")
-                        else:
-                            self.logger.debug(
-                                f"{self.qid}: milter: modify "
-                                f"header (occ. {occurrence}): "
-                                f"{header}: {enc_value}")
 
-                        self.chgheader(header, occurrence, enc_value)
+                    if milter_action["action"] == "accept":
-                elif mod_type == "mod_body":
+                        return Milter.ACCEPT
-                    mod_body_pos = params[0]
+
-                elif mod_type == "reject":
+                    if milter_action["action"] == "discard":
-                    self.setreply("554", "5.7.0", params[0])
+                        return Milter.DISCARD
-                    return Milter.REJECT
 
-            if mod_body_pos is not None:
-                self.fp.seek(mod_body_pos)
-                self.logger.debug(f"{self.qid}: milter: replace body")
-                self.replacebody(self.fp.read())
         except Exception as e:
             self.logger.exception(
                 f"an exception occured in eom function: {e}")

pymodmilter/actions.py

@@ -0,0 +1,456 @@
+# PyMod-Milter is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# PyMod-Milter is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with PyMod-Milter.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import logging
+import re
+
+from bs4 import BeautifulSoup
+from collections import defaultdict
+from copy import copy
+from email.header import Header
+from email.parser import BytesFeedParser
+from email.message import MIMEPart
+from email.policy import default as default_policy, SMTP
+
+from pymodmilter import CustomLogger, Conditions
+
+
+def _replace_illegal_chars(string):
+    """Replace illegal characters in header values."""
+    return string.replace(
+        "\x00", "").replace(
+        "\r", "").replace(
+        "\n", "")
+
+
+def add_header(field, value, milter, idx=-1, pretend=False,
+               logger=logging.getLogger(__name__)):
+    """Add a mail header field."""
+    header = f"{field}: {value}"
+    if logger.getEffectiveLevel() == logging.DEBUG:
+        logger.debug(f"add_header: {header}")
+    else:
+        logger.info(f"add_header: {header[0:70]}")
+
+    if idx == -1:
+        milter.fields.append((field, value))
+    else:
+        milter.fields.insert(idx, (field, value))
+
+    if pretend:
+        return
+
+    encoded_value = _replace_illegal_chars(
+        Header(s=value).encode())
+    milter.logger.debug(f"milter: addheader: {field}[{idx}]: {encoded_value}")
+    milter.addheader(field, encoded_value, idx)
+
+
+def mod_header(field, value, milter, search=None, pretend=False,
+               logger=logging.getLogger(__name__)):
+    """Change the value of a mail header field."""
+    if not isinstance(field, re.Pattern):
+        field = re.compile(field, re.IGNORECASE)
+
+    if search is not None and not isinstance(search, re.Pattern):
+        search = re.compile(search, re.MULTILINE + re.DOTALL + re.IGNORECASE)
+
+    occ = defaultdict(int)
+
+    for idx, (f, v) in enumerate(milter.fields):
+        occ[f] += 1
+
+        if not field.match(f):
+            continue
+
+        if search is not None:
+            new_v = search.sub(value, v).strip()
+        else:
+            new_v = value.strip()
+
+        if new_v == v:
+            continue
+
+        if not new_v:
+            logger.warning(
+                f"mod_header: resulting value is empty, "
+                f"skip modification")
+            continue
+
+        header = f"{f}: {v}"
+        new_header = f"{f}: {new_v}"
+        if logger.getEffectiveLevel() == logging.DEBUG:
+            logger.debug(f"mod_header: {header}: {new_header}")
+        else:
+            logger.info(f"mod_header: {header[0:70]}: {new_header[0:70]}")
+
+        milter.fields[idx] = (f, new_v)
+
+        if pretend:
+            continue
+
+        encoded_value = _replace_illegal_chars(
+            Header(s=new_v).encode())
+        milter.logger.debug(
+            f"milter: chgheader: {f}[{occ[f]}]: {encoded_value}")
+        milter.chgheader(f, occ[f], encoded_value)
+
+
+def del_header(field, milter, value=None, pretend=False,
+               logger=logging.getLogger(__name__)):
+    """Delete a mail header field."""
+    if not isinstance(field, re.Pattern):
+        field = re.compile(field, re.IGNORECASE)
+
+    if value is not None and not isinstance(value, re.Pattern):
+        value = re.compile(value, re.MULTILINE + re.DOTALL + re.IGNORECASE)
+
+    idx = -1
+    occ = defaultdict(int)
+
+    # iterate a copy of milter.fields because elements may get removed
+    # during iteration
+    for f, v in milter.fields.copy():
+        idx += 1
+        occ[f] += 1
+
+        if not field.match(f):
+            continue
+
+        if value is not None and not value.search(v):
+            continue
+
+        header = f"{f}: {v}"
+        if logger.getEffectiveLevel() == logging.DEBUG:
+            logger.debug(f"del_header: {header}")
+        else:
+            logger.info(f"del_header: {header[0:70]}")
+
+        del milter.fields[idx]
+
+        if not pretend:
+            encoded_value = ""
+            milter.logger.debug(
+                f"milter: chgheader: {f}[{occ[f]}]: {encoded_value}")
+            milter.chgheader(f, occ[f], encoded_value)
+
+        idx -= 1
+        occ[f] -= 1
+
+
+def _get_body_content(msg, body_type):
+    content = None
+    body_part = msg.get_body(preferencelist=(body_type))
+    if body_part is not None:
+        content = body_part.get_content()
+
+    return (body_part, content)
+
+
+def _wrap_message(milter):
+    msg = MIMEPart()
+    msg.add_header("MIME-Version", "1.0")
+
+    msg.set_content(
+        "Please see the original email attached.")
+    msg.add_alternative(
+        "Please see the original email attached.",
+        subtype="html")
+
+    data = b""
+    for field, value in milter.fields:
+        encoded_value = _replace_illegal_chars(
+            Header(s=value).encode())
+        data += field.encode("ascii", errors="replace")
+        data += b": "
+        data += encoded_value.encode("ascii", errors="replace")
+        data += b"\r\n"
+
+    milter.fp.seek(0)
+    data += b"\r\n" + milter.fp.read()
+
+    msg.add_attachment(
+        data, maintype="plain", subtype="text",
+        filename=f"{milter.qid}.eml")
+
+    return msg
+
+
+def _inject_body(milter, msg):
+    if not msg.is_multipart():
+        msg.make_mixed()
+
+    new_msg = MIMEPart()
+    new_msg.add_header("MIME-Version", "1.0")
+    new_msg.set_content("")
+    new_msg.add_alternative("", subtype="html")
+    new_msg.make_mixed()
+    for attachment in msg.iter_attachments():
+        new_msg.attach(attachment)
+
+    return new_msg
+
+
+def add_disclaimer(text, html, action, policy, milter, pretend=False,
+                   logger=logging.getLogger(__name__)):
+    """Append or prepend a disclaimer to the mail body."""
+    milter.fp.seek(0)
+    fp = BytesFeedParser(policy=default_policy)
+
+    for field, value in milter.fields:
+        field_lower = field.lower()
+        if not field_lower.startswith("content-") and \
+                field_lower != "mime-version":
+            continue
+        logger.debug(
+            f"feed content header to message object: {field}: {value}")
+        encoded_value = _replace_illegal_chars(
+            Header(s=value).encode())
+        fp.feed(field.encode("ascii", errors="replace"))
+        fp.feed(b": ")
+        fp.feed(encoded_value.encode("ascii", errors="replace"))
+        fp.feed(b"\r\n")
+
+    fp.feed(b"\r\n")
+    logger.debug(f"feed body to message object: {field}: {value}")
+    fp.feed(milter.fp.read())
+
+    logger.debug("parse message")
+    msg = fp.close()
+
+    text_content = None
+    html_content = None
+
+    try:
+        try:
+            logger.debug("try to find a plain and/or html body part")
+            text_body, text_content = _get_body_content(msg, "plain")
+            html_body, html_content = _get_body_content(msg, "html")
+            if text_content is None and html_content is None:
+                raise RuntimeError()
+        except RuntimeError:
+            logger.info(
+                "message does not contain any body part, "
+                "inject empty plain and html body parts")
+            msg = _inject_body(milter, msg)
+            text_body, text_content = _get_body_content(msg, "plain")
+            html_body, html_content = _get_body_content(msg, "html")
+            if text_content is None and html_content is None:
+                raise RuntimeError("no message body present after injecting")
+    except Exception as e:
+        logger.warning(e)
+        if policy == "ignore":
+            logger.info(
+                f"unable to add disclaimer to message body, "
+                f"ignore error according to policy")
+            return
+        elif policy == "reject":
+            logger.info(
+                f"unable to add disclaimer to message body, "
+                f"reject message according to policy")
+            return [
+                ("reject", "Message rejected due to error")]
+
+        logger.info("wrap original message in a new message envelope")
+        msg = _wrap_message(milter)
+        text_body, text_content = _get_body_content(msg, "plain")
+        html_body, html_content = _get_body_content(msg, "html")
+        if text_content is None and html_content is None:
+            raise Exception("no message body present after wrapping, "
+                            "give up ...")
+
+    if text_content is not None:
+        logger.info(f"{action} text disclaimer")
+
+        if action == "prepend":
+            content = f"{text}{text_content}"
+        else:
+            content = f"{text_content}{text}"
+
+        text_body.set_content(
+            content.encode(), maintype="text", subtype="plain")
+        text_body.set_param("charset", "UTF-8", header="Content-Type")
+
+    if html_content is not None:
+        logger.info(f"{action} html disclaimer")
+
+        soup = BeautifulSoup(html_content, "html.parser")
+
+        body = soup.find('body')
+        if body:
+            soup = body
+
+        if action == "prepend":
+            soup.insert(0, copy(html))
+        else:
+            soup.append(html)
+
+        html_body.set_content(
+            str(soup).encode(), maintype="text", subtype="html")
+        html_body.set_param("charset", "UTF-8", header="Content-Type")
+
+    try:
+        logger.debug("serialize message as bytes")
+        data = msg.as_bytes(policy=SMTP)
+    except Exception as e:
+        logger.waring(
+            f"unable to serialize message as bytes: {e}")
+        try:
+            logger.warning("try to serialize message as string")
+            data = msg.as_string(policy=SMTP)
+            data = data.encode("ascii", errors="replace")
+        except Exception as e:
+            raise e
+
+    body_pos = data.find(b"\r\n\r\n") + 4
+    milter.fp.seek(0)
+    milter.fp.write(data[body_pos:])
+    milter.fp.truncate()
+
+    if pretend:
+        return
+
+    logger.debug("milter: replacebody")
+    milter.replacebody(data[body_pos:])
+    del data
+
+    fields = {
+        "mime-version": {
+            "field": "MIME-Version",
+            "value": msg.get("MIME-Version"),
+            "modified": False},
+        "content-type": {
+            "field": "Content-Type",
+            "value": msg.get("Content-Type"),
+            "modified": False},
+        "content-transfer-encoding": {
+            "field": "Content-Transfer-Encoding",
+            "value": msg.get("Content-Transfer-Encoding"),
+            "modified": False}}
+
+    for field, value in milter.fields.copy():
+        field_lower = field.lower()
+        if field_lower in fields and fields[field_lower]["value"] is not None:
+            mod_header(field=f"^{field}$", value=fields[field_lower]["value"],
+                       milter=milter, pretend=pretend, logger=logger)
+            fields[field_lower]["modified"] = True
+
+        elif field_lower.startswith("content-"):
+            del_header(field=f"^{field}$", milter=milter,
+                       pretend=pretend, logger=logger)
+
+    for field in fields.values():
+        if not field["modified"] and field["value"] is not None:
+            add_header(field=field["field"], value=field["value"],
+                       milter=milter, pretend=pretend, logger=logger)
+
+
+class Action:
+    """Action to implement a pre-configured action to perform on e-mails."""
+    _types = {
+        "add_header": ["fields"],
+        "del_header": ["fields"],
+        "mod_header": ["fields"],
+        "add_disclaimer": ["fields", "body"]}
+
+    def __init__(self, name, local_addrs, conditions, action_type, args,
+                 loglevel=logging.INFO, pretend=False):
+        logger = logging.getLogger(name)
+        self.logger = CustomLogger(logger, {"name": name})
+        self.logger.setLevel(loglevel)
+
+        self.conditions = Conditions(
+            local_addrs=local_addrs,
+            args=conditions,
+            logger=self.logger)
+        self.pretend = pretend
+        self._args = {}
+
+        if action_type not in self._types:
+            raise RuntimeError(f"invalid action_type '{action_type}'")
+        self._needs = self._types[action_type]
+
+        try:
+            if action_type == "add_header":
+                self._func = add_header
+                self._args["field"] = args["header"]
+                self._args["value"] = args["value"]
+                if "idx" in args:
+                    self._args["idx"] = args["idx"]
+
+            elif action_type in ["mod_header", "del_header"]:
+                args["field"] = args["header"]
+                del args["header"]
+                regex_args = ["field"]
+
+                if action_type == "mod_header":
+                    self._func = mod_header
+                    self._args["value"] = args["value"]
+                    regex_args.append("search")
+                elif action_type == "del_header" and "value" in args:
+                    self._func = del_header
+                    regex_args.append("value")
+
+                for arg in regex_args:
+                    try:
+                        self._args[arg] = re.compile(
+                            args[arg],
+                            re.MULTILINE + re.DOTALL + re.IGNORECASE)
+                    except re.error as e:
+                        raise RuntimeError(
+                            f"unable to parse {arg} regex: {e}")
+
+            elif action_type == "add_disclaimer":
+                self._func = add_disclaimer
+                if args["action"] not in ["append", "prepend"]:
+                    raise RuntimeError(f"invalid action '{args['action']}'")
+
+                self._args["action"] = args["action"]
+
+                if args["error_policy"] not in ["wrap", "ignore", "reject"]:
+                    raise RuntimeError(f"invalid policy '{args['policy']}'")
+
+                self._args["policy"] = args["error_policy"]
+
+                try:
+                    with open(args["html_file"], "r") as f:
+                        html = BeautifulSoup(
+                            f.read(), "html.parser")
+                        body = html.find('body')
+                        if body:
+                            # just use content within the body tag if present
+                            html = body
+                        self._args["html"] = html
+                    with open(args["text_file"], "r") as f:
+                        self._args["text"] = f.read()
+                except IOError as e:
+                    raise RuntimeError(f"unable to read template: {e}")
+
+        except KeyError as e:
+            raise RuntimeError(
+                f"mandatory argument not found: {e}")
+
+    def needs(self):
+        """Return the needs of this action."""
+        return self._needs
+
+    def execute(self, milter, pretend=None):
+        """Execute configured action."""
+        if pretend is None:
+            pretend = self.pretend
+
+        logger = CustomLogger(self.logger, {"qid": milter.qid})
+
+        return self._func(
+            milter=milter, pretend=pretend, logger=logger, **self._args)

pymodmilter/conditions.py

@@ -0,0 +1,134 @@
+# PyMod-Milter is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# PyMod-Milter is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with PyMod-Milter.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import logging
+import re
+
+from netaddr import IPAddress, IPNetwork, AddrFormatError
+
+
+class Conditions:
+    """Conditions to implement conditions for rules and actions."""
+
+    def __init__(self, local_addrs, args, logger=None):
+        if logger is None:
+            logger = logging.getLogger(__name__)
+
+        self._local_addrs = []
+        self.logger = logger
+        self._args = {}
+
+        try:
+            for addr in local_addrs:
+                self._local_addrs.append(IPNetwork(addr))
+        except AddrFormatError as e:
+            raise RuntimeError(f"invalid address in local_addrs: {e}")
+
+        try:
+            if "local" in args:
+                logger.debug(f"condition: local = {args['local']}")
+                self._args["local"] = args["local"]
+
+            if "hosts" in args:
+                logger.debug(f"condition: hosts = {args['hosts']}")
+                self._args["hosts"] = []
+                try:
+                    for host in args["hosts"]:
+                        self._args["hosts"].append(IPNetwork(host))
+                except AddrFormatError as e:
+                    raise RuntimeError(f"invalid address in hosts: {e}")
+
+            if "envfrom" in args:
+                logger.debug(f"condition: envfrom = {args['envfrom']}")
+                try:
+                    self._args["envfrom"] = re.compile(
+                        args["envfrom"], re.IGNORECASE)
+                except re.error as e:
+                    raise RuntimeError(f"unable to parse envfrom regex: {e}")
+
+            if "envto" in args:
+                logger.debug(f"condition: envto = {args['envto']}")
+                try:
+                    self._args["envto"] = re.compile(
+                        args["envto"], re.IGNORECASE)
+                except re.error as e:
+                    raise RuntimeError(f"unable to parse envto regex: {e}")
+
+        except KeyError as e:
+            raise RuntimeError(
+                f"mandatory argument not found: {e}")
+
+    def match(self, args):
+        if "host" in args:
+            ip = IPAddress(args["host"])
+
+            if "local" in self._args:
+                is_local = False
+                for addr in self._local_addrs:
+                    if ip in addr:
+                        is_local = True
+                        break
+
+                if is_local != self._args["local"]:
+                    self.logger.debug(
+                        f"ignore host {args['host']}, "
+                        f"condition local does not match")
+                    return False
+
+                self.logger.debug(
+                    f"condition local matches for host {args['host']}")
+
+            if "hosts" in self._args:
+                found = False
+                for addr in self._args["hosts"]:
+                    if ip in addr:
+                        found = True
+                        break
+
+                if not found:
+                    self.logger.debug(
+                        f"ignore host {args['host']}, "
+                        f"condition hosts does not match")
+                    return False
+
+                self.logger.debug(
+                    f"condition hosts matches for host {args['host']}")
+
+        if "envfrom" in args and "envfrom" in self._args:
+            if not self._args["envfrom"].match(args["envfrom"]):
+                self.logger.debug(
+                    f"ignore envelope-from address {args['envfrom']}, "
+                    f"condition envfrom does not match")
+                return False
+
+            self.logger.debug(
+                f"condition envfrom matches for "
+                f"envelope-from address {args['envfrom']}")
+
+        if "envto" in args and "envto" in self._args:
+            if not isinstance(args["envto"], list):
+                args["envto"] = [args["envto"]]
+
+            for envto in args["envto"]:
+                if not self._args["envto"].match(envto):
+                    self.logger.debug(
+                        f"ignore envelope-to address {args['envto']}, "
+                        f"condition envto does not match")
+                    return False
+
+            self.logger.debug(
+                f"condition envto matches for "
+                f"envelope-to address {args['envto']}")
+
+        return True

pymodmilter/run.py

@@ -12,7 +12,6 @@
 # along with PyMod-Milter.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-
 import Milter
 import argparse
 import logging
@@ -24,77 +23,103 @@ from re import sub
 
 from pymodmilter import Rule, ModifyMilter
 from pymodmilter.version import __version__ as version
+from pymodmilter.actions import Action
 
 
 def main():
     "Run PyMod-Milter."
-    # parse command line
     parser = argparse.ArgumentParser(
         description="PyMod milter daemon",
         formatter_class=lambda prog: argparse.HelpFormatter(
             prog, max_help_position=45, width=140))
+
     parser.add_argument(
         "-c", "--config", help="Config file to read.",
         default="/etc/pymodmilter/pymodmilter.conf")
+
     parser.add_argument(
         "-s",
         "--socket",
         help="Socket used to communicate with the MTA.",
         default="")
+
     parser.add_argument(
         "-d",
         "--debug",
         help="Log debugging messages.",
         action="store_true")
+
     parser.add_argument(
         "-t",
         "--test",
         help="Check configuration.",
         action="store_true")
+
     parser.add_argument(
         "-v", "--version",
         help="Print version.",
         action="version",
         version=f"%(prog)s ({version})")
+
     args = parser.parse_args()
 
-    # setup logging
+    loglevels = {
-    loglevel = logging.INFO
+        "error": logging.ERROR,
-    logname = "pymodmilter"
+        "warning": logging.WARNING,
-    syslog_name = logname
+        "info": logging.INFO,
-    if args.debug:
+        "debug": logging.DEBUG
-        loglevel = logging.DEBUG
+    }
-        logname = f"{logname}[%(name)s]"
-        syslog_name = f"{syslog_name}: [%(name)s] %(levelname)s"
 
     root_logger = logging.getLogger()
-    root_logger.setLevel(loglevel)
+    root_logger.setLevel(logging.DEBUG)
 
     # setup console log
     stdouthandler = logging.StreamHandler(sys.stdout)
-    stdouthandler.setLevel(logging.DEBUG)
+    stdouthandler.setFormatter(
-    formatter = logging.Formatter("%(message)s")
+        logging.Formatter("%(asctime)s - %(levelname)s: %(message)s"))
-    stdouthandler.setFormatter(formatter)
     root_logger.addHandler(stdouthandler)
+
+    # setup syslog
+    sysloghandler = logging.handlers.SysLogHandler(
+        address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_MAIL)
+    sysloghandler.setFormatter(
+        logging.Formatter("pymodmilter: %(message)s"))
+    root_logger.addHandler(sysloghandler)
+
     logger = logging.getLogger(__name__)
 
+    if not args.debug:
+        logger.setLevel(logging.INFO)
+
     try:
-        # read config file
-        logger.debug("parsing config file")
         try:
             with open(args.config, "r") as fh:
-                config = loads(
+                config = sub(r"(?m)^\s*#.*\n?", "", fh.read())
-                    sub(r"(?m)^\s*#.*\n?", "", fh.read()))
+                config = loads(config)
         except Exception as e:
+            for num, line in enumerate(config.splitlines()):
+                logger.error(f"{num+1}: {line}")
             raise RuntimeError(
                 f"unable to parse config file: {e}")
 
-        logger.debug("preparing milter configuration ...")
-
-        # default values for global config if not set
         if "global" not in config:
             config["global"] = {}
 
+        if "loglevel" not in config["global"]:
+            config["global"]["loglevel"] = "info"
+
+        if args.debug:
+            loglevel = logging.DEBUG
+        else:
+            loglevel = loglevels[config["global"]["loglevel"]]
+
+        logger.setLevel(loglevel)
+
+        logger.debug("prepar milter configuration")
+
+        if "pretend" not in config["global"]:
+            config["global"]["pretend"] = False
+
         if args.socket:
             socket = args.socket
         elif "socket" in config["global"]:
@@ -104,64 +129,110 @@ def main():
                 f"listening socket is neither specified on the command line "
                 f"nor in the configuration file")
 
-        if "local_addrs" not in config["global"]:
+        if "local_addrs" in config["global"]:
-            config["global"]["local_addrs"] = [
+            local_addrs = config["global"]["local_addrs"]
+        else:
+            local_addrs = [
+                "::1/128",
                 "127.0.0.0/8",
                 "10.0.0.0/8",
                 "172.16.0.0/12",
                 "192.168.0.0/16"]
 
-        if "log" not in config["global"]:
+        if "rules" not in config:
-            config["global"]["log"] = True
+            raise RuntimeError(
-
+                f"mandatory config section 'rules' not found")
-        if "pretend" not in config["global"]:
-            config["global"]["pretend"] = False
-
-        # check if mandatory sections are present in config
-        for section in ["rules"]:
-            if section not in config:
-                raise RuntimeError(
-                    f"mandatory config section '{section}' not found")
 
         if not config["rules"]:
             raise RuntimeError("no rules configured")
 
-        rules = []
+        logger.debug("initialize rules ...")
-        # iterate configured rules
-        for rule_idx, rule in enumerate(config["rules"]):
-            params = {}
-            # set default values if not specified in config
-            if "name" in rule:
-                params["name"] = rule["name"]
-            else:
-                params["name"] = f"Rule #{rule_idx}"
 
-            if "log" in rule:
+        rules = []
-                params["log"] = rule["log"]
+        for rule_idx, rule in enumerate(config["rules"]):
+            if "name" in rule:
+                rule_name = rule["name"]
             else:
-                params["log"] = config["global"]["log"]
+                rule_name = f"Rule #{rule_idx}"
+
+            logger.debug(f"prepare rule {rule_name} ...")
+
+            if "actions" not in rule:
+                raise RuntimeError(
+                    f"{rule_name}: mandatory config "
+                    f"section 'actions' not found")
+
+            if not rule["actions"]:
+                raise RuntimeError("{rule_name}: no actions configured")
+
+            if args.debug:
+                rule_loglevel = logging.DEBUG
+            elif "loglevel" in rule:
+                rule_loglevel = loglevels[rule["loglevel"]]
+            else:
+                rule_loglevel = loglevels[config["global"]["loglevel"]]
 
             if "pretend" in rule:
-                params["pretend"] = rule["pretend"]
+                rule_pretend = rule["pretend"]
             else:
-                params["pretend"] = config["global"]["pretend"]
+                rule_pretend = config["global"]["pretend"]
 
-            if "local_addrs" in rule:
+            actions = []
-                params["local_addrs"] = rule["local_addrs"]
+            for action_idx, action in enumerate(rule["actions"]):
-            else:
+                if "name" in action:
-                params["local_addrs"] = config["global"]["local_addrs"]
+                    action_name = action["name"]
+                else:
+                    action_name = f"Action #{action_idx}"
 
-            if "conditions" in rule:
+                if args.debug:
-                params["conditions"] = rule["conditions"]
+                    action_loglevel = logging.DEBUG
+                elif "loglevel" in action:
+                    action_loglevel = loglevels[action["loglevel"]]
+                else:
+                    action_loglevel = rule_loglevel
 
-            if "modifications" in rule:
+                if "pretend" in action:
-                params["modifications"] = rule["modifications"]
+                    action_pretend = action["pretend"]
-            else:
+                else:
-                raise RuntimeError(
+                    action_pretend = rule_pretend
-                    f"{rule['name']}: mandatory config section "
-                    f"'modifications' not found")
 
-            rules.append(Rule(**params))
+                if "type" not in action:
+                    raise RuntimeError(
+                        f"{rule_name}: {action_name}: mandatory config "
+                        f"section 'actions' not found")
+
+                if "conditions" not in action:
+                    action["conditions"] = {}
+
+                try:
+                    actions.append(
+                        Action(
+                            name=action_name,
+                            local_addrs=local_addrs,
+                            conditions=action["conditions"],
+                            action_type=action["type"],
+                            args=action,
+                            loglevel=action_loglevel,
+                            pretend=action_pretend))
+                except RuntimeError as e:
+                    logger.error(f"{action_name}: {e}")
+                    sys.exit(253)
+
+            if "conditions" not in rule:
+                rule["conditions"] = {}
+
+            try:
+                rules.append(
+                    Rule(
+                        name=rule_name,
+                        local_addrs=local_addrs,
+                        conditions=rule["conditions"],
+                        actions=actions,
+                        loglevel=rule_loglevel,
+                        pretend=rule_pretend))
+            except RuntimeError as e:
+                logger.error(f"{rule_name}: {e}")
+                sys.exit(254)
 
     except RuntimeError as e:
         logger.error(e)
@@ -171,27 +242,22 @@ def main():
         print("Configuration ok")
         sys.exit(0)
 
-    # change log format for runtime
+    # setup console log for runtime
-    formatter = logging.Formatter(
+    formatter = logging.Formatter("%(asctime)s - %(levelname)s: %(message)s")
-        f"%(asctime)s {logname}: [%(levelname)s] %(message)s",
-        datefmt="%Y-%m-%d %H:%M:%S")
     stdouthandler.setFormatter(formatter)
-
+    stdouthandler.setLevel(logging.DEBUG)
-    # setup syslog
-    sysloghandler = logging.handlers.SysLogHandler(
-        address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_MAIL)
-    sysloghandler.setLevel(loglevel)
-    formatter = logging.Formatter(f"{syslog_name}: %(message)s")
-    sysloghandler.setFormatter(formatter)
-    root_logger.addHandler(sysloghandler)
 
     logger.info("pymodmilter starting")
     ModifyMilter.set_rules(rules)
+    ModifyMilter.set_loglevel(loglevels[config["global"]["loglevel"]])
 
     # register milter factory class
     Milter.factory = ModifyMilter
     Milter.set_exception_policy(Milter.TEMPFAIL)
 
+    if args.debug:
+        Milter.setdbg(1)
+
     rc = 0
     try:
         Milter.runmilter("pymodmilter", socketname=socket, timeout=30)