diff --git a/README.md b/README.md
index d90e816..e103d79 100644
--- a/README.md
+++ b/README.md
@@ -2,7 +2,7 @@
A pymilter based sendmail/postfix pre-queue filter with the ability to add, remove and modify e-mail headers.
The project is currently in beta status, but it is already used in a productive enterprise environment which processes about a million e-mails per month.
-The basic idea is to define rules with conditions and modifications which are processed when all conditions are true.
+The basic idea is to define rules with conditions and actions which are processed when all conditions are true.
## Dependencies
Pymodmilter is depending on these python packages, but they are installed automatically if you are working with pip.
@@ -10,7 +10,7 @@ Pymodmilter is depending on these python packages, but they are installed automa
* [netaddr](https://github.com/drkjam/netaddr/)
## Installation
-* Install pymodmilter with pip and copy the example configuration file.
+* Install pymodmilter with pip and copy the example config file.
```sh
pip install pymodmilter
cp /etc/pymodmilter/pymodmilter.conf.example /etc/pymodmilter/pymodmilter.conf
@@ -18,96 +18,107 @@ cp /etc/pymodmilter/pymodmilter.conf.example /etc/pymodmilter/pymodmilter.conf
* Modify /etc/pymodmilter/pymodmilter.conf according to your needs.
## Configuration options
-Pymodmilter uses a configuration file in JSON format. The options are described below. Make a copy of the [example configuration file](https://github.com/spacefreak86/pymodmilter/blob/master/docs/pymodmilter.conf.example) in the [docs](https://github.com/spacefreak86/pymodmilter/tree/master/docs) folder to start with.
-Rules and modifications are processed in the given order.
+Pymodmilter uses a config file in JSON format. The config file has to be JSON valid with the exception of allowed comment lines starting with **#**. The options are described below.
+Rules and actions are processed in the given order.
### Global
-The following global configuration options are optional:
-* **socket**
- The socket used to communicate with the MTA.
-* **local_addrs**
- A list of hosts and network addresses which are considered local. It is used to for the condition option [local](#Conditions). This option may be overriden within a rule object.
-* **log**
- Enable or disable logging. This option may be overriden by a rule or modification object.
+Config options in **global** section:
+* **socket** (optional)
+ The socket used to communicate with the MTA. If it is not specified in the config, it has to be set as command line option.
+* **local_addrs** (optional)
+ A list of hosts and network addresses which are considered local. It is used to for the condition option [local](#Conditions).
+ Default: **::1/128, 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16**
+* **loglevel** (optional)
+ Set the log level. This option may be overriden by any rule or action object. Possible values are:
+ * **error**
+ * **warning**
+ * **info**
+ * **debug**
+ Default: **info**
+* **pretend** (optional)
+ Pretend actions, for test purposes. This option may be overriden by any rule or action object.
### Rules
-The following configuration options are mandatory for each rule:
-* **modifications**
- A list of modification objects which are processed in the given order.
-
-The following configuration options are optional for each rule:
-* **name**
- Name of the rule.
-* **conditions**
- A list of conditions which all have to be true to process the rule.
-* **local_addrs**
+Config options for **rule** objects:
+* **name** (optional)
+ Name of the rule.
+ Default: **Rule #n**
+* **actions**
+ A list of action objects which are processed in the given order.
+* **conditions** (optional)
+ A list of conditions which all have to be true to process the actions.
+* **loglevel** (optional)
As described above in the [Global](#Global) section.
-* **log**
+* **pretend** (optional)
As described above in the [Global](#Global) section.
-* **pretend**
- Just pretend to make the modifications, for test purposes.
-### Modifications
-The following configuration options are mandatory for each modification:
+### Actions
+Config options for **action** objects:
+* **name** (optional)
+ Name of the action.
+ Default: **Action #n**
* **type**
- Set the modification type. Possible values are:
+ Action type. Possible values are:
* **add_header**
* **del_header**
* **mod_header**
* **add_disclaimer**
+* **conditions** (optional)
+ A list of conditions which all have to be true to process the action.
+* **pretend** (optional)
+ Just pretend all actions of this rule, for test purposes.
+* **loglevel** (optional)
+ As described above in the [Global](#Global) section.
-The following configuration options are mandatory based on the modification type in use.
-* **add_header**
+Config options for **add_header** actions:
* **header**
Name of the header.
* **value**
Value of the header.
-* **del_header**
+Config options for **del_header** actions:
* **header**
- Regular expression to match against header lines.
+ Regular expression to match against header names.
+ * **value** (optional)
+ Regular expression to match against the headers value.
-* **mod_header**
+Config options for **mod_header** actions:
* **header**
- Regular expression to match against header lines.
- * **search**
- Regular expression to match against the value of header lines. You may use subgroups or named subgroups (python syntax) to include parts of the original value in the new value.
+ Regular expression to match against header names.
+ * **search** (optional)
+ Regular expression to match against header values. You may use subgroups or named subgroups (python syntax) to include parts of the original value in the new value.
* **value**
New value of the header.
-* **add_disclaimer**
+Config options for **add_disclaimer** actions:
* **action**
Action to perform with the disclaimer. Possible values are:
* append
* prepend
- * **html_template**
- Path to a file that contains the html representation of the disclaimer.
- * **text_template**
- Path to a file that contains the text representation of the disclaimer.
- * **error_policy**
- Set what should be done if the disclaimer could not be added (e.g. no body text found). Possible values are:
+ * **html_file**
+ Path to a file which contains the html representation of the disclaimer.
+ * **text_file**
+ Path to a file which contains the text representation of the disclaimer.
+ * **error_policy** (optional)
+ Set the error policy in case the disclaimer cannot be added (e.g. if no body part is present in the e-mail). Possible values are:
* wrap
- The original e-mail will be attached to a new one containing the disclaimer.
+ A new e-mail body is generated with the disclaimer as body and the original e-mail attached.
* ignore
Ignore the error and do nothing.
* reject
Reject the e-mail.
+ Default: **wrap**
-The following configuration options are optional for each modification:
-* **name**
- Name of the modification.
-* **log**
- As described above in the global object section.
### Conditions
-The following condition options are optional:
-* **local**
+Config options for **conditions** objects:
+* **local** (optional)
If set to true, the rule is only executed for e-mails originating from addresses defined in local_addrs and vice versa.
-* **hosts**
+* **hosts** (optional)
A list of hosts and network addresses for which the rule should be executed.
-* **envfrom**
+* **envfrom** (optional)
A regular expression to match against the evenlope-from addresses for which the rule should be executed.
-* **envto**
+* **envto** (optional)
A regular expression to match against all evenlope-to addresses. All addresses must match to fulfill the condition.
## Developer information
diff --git a/docs/pymodmilter.conf.example b/docs/pymodmilter.conf.example
index 51874ee..1afaa43 100644
--- a/docs/pymodmilter.conf.example
+++ b/docs/pymodmilter.conf.example
@@ -30,16 +30,23 @@
#
"local_addrs": ["127.0.0.0/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"],
- # Option: log
+ # Option: loglevel
+ # Type: String
+ # Notes: Set loglevel for rules and actions.
+ # Value: [ error | warning | info | debug ]
+ #
+ "loglevel": "info",
+
+ # Option: pretend
# Type: Bool
- # Notes: Enable or disable logging of rules and modifications.
+ # Notes: Just pretend to do the actions, for test purposes.
# Value: [ true | false ]
#
- "log": true
+ "pretend": true
},
# Section: rules
- # Notes: Rules and related modifications.
+ # Notes: Rules and related actions.
#
"rules": [
{
@@ -85,10 +92,10 @@
"envto": "^postmaster@.+$"
},
- # Section: modifications
- # Notes: Modifications of the rule.
+ # Section: actions
+ # Notes: Actions of the rule.
#
- "modifications": [
+ "actions": [
{
# Option: name
# Type: String
@@ -116,7 +123,7 @@
# Notes: Value of the header.
# Value: [ VALUE ]
#
- "value": "true",
+ "value": "true"
}, {
"name": "modify_subject",
@@ -127,7 +134,7 @@
# Notes: Regular expression to match against header lines (e.g. Subject: Test-Subject).
# Value: [ REGEX ]
#
- "header": "^Subject:",
+ "header": "^Subject$",
# Option: search
# Type: String
@@ -151,7 +158,7 @@
# Notes: Regular expression to match against header lines (e.g. Subject: Test-Subject).
# Value: [ REGEX ]
#
- "header": "^Received:"
+ "header": "^Received$"
}, {
"name": "add_disclaimer",
@@ -164,19 +171,19 @@
#
"action": "prepend",
- # Option: html_template
+ # Option: html_file
# Type: String
- # Notes: Path to a file that contains the html representation of the disclaimer.
+ # Notes: Path to a file which contains the html representation of the disclaimer.
# Value: [ FILE_PATH ]
#
- "html_template": "/etc/pymodmilter/templates/disclaimer_html.template",
+ "html_file": "/etc/pymodmilter/templates/disclaimer_html.template",
- # Option: text_template
+ # Option: text_file
# Type: String
- # Notes: Path to a file that contains the text representation of the disclaimer.
+ # Notes: Path to a file which contains the text representation of the disclaimer.
# Value: [ FILE_PATH ]
#
- "text_template": "/etc/pymodmilter/templates/disclaimer_text.template",
+ "text_file": "/etc/pymodmilter/templates/disclaimer_text.template",
# Option: error_policy
# Type: String
@@ -185,14 +192,7 @@
#
"error_policy": "wrap"
}
- ],
-
- # Option: pretend
- # Type: Bool
- # Notes: Just pretend to do the modifications, for test purposes.
- # Value: [ true | false ]
- #
- "pretend": true
+ ]
}
]
}
diff --git a/docs/templates/disclaimer_html.template b/docs/templates/disclaimer_html.template
index eec02a0..827f148 100644
--- a/docs/templates/disclaimer_html.template
+++ b/docs/templates/disclaimer_html.template
@@ -6,4 +6,4 @@
-
+
diff --git a/pymodmilter/__init__.py b/pymodmilter/__init__.py
index 3fe5b69..1b255e6 100644
--- a/pymodmilter/__init__.py
+++ b/pymodmilter/__init__.py
@@ -14,28 +14,23 @@
__all__ = [
"make_header",
- "replace_illegal_chars",
+ "actions",
+ "conditions",
"run",
"version",
- "Modification",
+ "CustomLogger",
"Rule",
"ModifyMilter"]
import Milter
import logging
-import logging.handlers
-import re
from Milter.utils import parse_addr
-from bs4 import BeautifulSoup
-from copy import copy
from email.charset import Charset
from email.header import Header, decode_header
-from email import message_from_binary_file
-from email.message import MIMEPart
-from email.policy import default as default_policy, SMTP
from io import BytesIO
-from netaddr import IPAddress, IPNetwork, AddrFormatError
+
+from pymodmilter.conditions import Conditions
def make_header(decoded_seq, maxlinelen=None, header_name=None,
@@ -60,524 +55,101 @@ def make_header(decoded_seq, maxlinelen=None, header_name=None,
return h
-def replace_illegal_chars(string):
- """Replace illegal characters in header values."""
- return string.replace(
- "\x00", "").replace(
- "\r", "").replace(
- "\n", "")
+class CustomLogger(logging.LoggerAdapter):
+ def process(self, msg, kwargs):
+ if "name" in self.extra:
+ msg = "{}: {}".format(self.extra["name"], msg)
+ if "qid" in self.extra:
+ msg = "{}: {}".format(self.extra["qid"], msg)
-class Modification:
- """
- Modification to implement certain modifications on e-mails.
+ if self.logger.getEffectiveLevel() != logging.DEBUG:
+ msg = msg.replace("\n", "").replace("\r", "")
- Each modification function returns the necessary changes for ModifyMilter
- so they can be applied to the email passing the MTA.
- """
-
- def __init__(self, name, mod_type, log, **params):
- self.logger = logging.getLogger(__name__)
- self.logger.debug(f"initializing modification '{name}'")
- self.name = name
- self.log = log
- # needs for each modification type
- self.types = {
- "add_header": {
- "needs": ["headers"]},
- "del_header": {
- "needs": ["headers"]},
- "mod_header": {
- "needs": ["headers"]},
- "add_disclaimer": {
- "needs": ["headers", "data"]}}
-
- if mod_type not in self.types:
- raise RuntimeError(
- f"{self.name}: invalid modification type '{mod_type}'")
-
- self.mod_type = mod_type
-
- try:
- if mod_type == "add_header":
- self.header = params["header"]
- self.value = params["value"]
- elif mod_type in ["del_header", "mod_header"]:
- try:
- self.header = re.compile(
- params["header"],
- re.MULTILINE + re.DOTALL + re.IGNORECASE)
- except re.error as e:
- raise RuntimeError(
- f"{self.name}: unable to parse regex of "
- f"option 'header': {e}")
-
- if mod_type == "mod_header":
- try:
- self.search = re.compile(
- params["search"],
- re.MULTILINE + re.DOTALL + re.IGNORECASE)
- except re.error as e:
- raise RuntimeError(
- f"{self.name}: unable to parse regex of "
- f"option 'search': {e}")
-
- self.value = params["value"]
- elif mod_type == "add_disclaimer":
- if params["action"] not in ["append", "prepend"]:
- raise RuntimeError(
- f"{self.name}: unknown action specified")
-
- self.action = params["action"]
-
- if params["error_policy"] not in ["wrap", "ignore", "reject"]:
- raise RuntimeError(
- f"{self.name}: unknown error_policy specified")
-
- self.error_policy = params["error_policy"]
-
- try:
- with open(params["html_template"], "r") as f:
- self.html = BeautifulSoup(f.read(), "html.parser")
- body = self.html.find('body')
- if body:
- # just use content within the body tag if present
- self.html = body
- with open(params["text_template"], "r") as f:
- self.text = f.read()
- except IOError as e:
- raise RuntimeError(f"unable to read template: {e}")
- except KeyError as e:
- raise RuntimeError(
- f"{self.name}: mandatory configuration option not found: {e}")
-
- def needs(self):
- """Return the needs of this modification to work."""
- return self.types[self.mod_type]["needs"]
-
- def add_header(self, qid, headers, header, value, pos=-1):
- """Add header to email."""
- hdr = f"{header}: {value}"
- if self.log:
- self.logger.info(
- f"{qid}: {self.name}: add_header: {hdr[0:70]}")
- else:
- self.logger.debug(
- f"{qid}: {self.name}: add_header: {hdr}")
-
- headers.append((header, value))
- params = [header, value, pos]
- return [("add_header", *params)]
-
- def mod_header(self, qid, headers, header, search, replace):
- """Modify an email header."""
- if isinstance(header, str):
- header = re.compile(
- header, re.MULTILINE + re.DOTALL + re.IGNORECASE)
-
- if isinstance(search, str):
- search = re.compile(
- search, re.MULTILINE + re.DOTALL + re.IGNORECASE)
-
- changes = []
- index = 0
- occurrences = {}
- # iterate a copy of headers because headers may be modified
- for name, value in headers.copy():
- # keep track of the occurrence of each header
- # needed by Milter.Base.chgheader
- if name not in occurrences.keys():
- occurrences[name] = 1
- else:
- occurrences[name] += 1
-
- hdr = f"{name}: {value}"
- if header.search(hdr):
- new_value = search.sub(replace, value).strip()
- if new_value == "":
- self.logger.warning(
- f"{qid}: {self.name}: mod_header: resulting value is "
- f"empty, skip modification")
- elif value != new_value:
- old_hdr = hdr
- hdr = f"{name}: {new_value}"
- if self.log:
- self.logger.info(
- f"{qid}: {self.name}: mod_header: "
- f"{old_hdr[0:70]}: {hdr[0:70]}")
- else:
- self.logger.debug(
- f"{qid}: {self.name}: mod_header: "
- f"(occ. {occurrences[name]}): {old_hdr}: "
- f"{hdr}")
-
- headers[index] = (name, new_value)
- params = [name, new_value, occurrences[name]]
- changes.append(("mod_header", *params))
-
- index += 1
-
- return changes
-
- def del_header(self, qid, headers, header):
- """Delete an email header."""
- if isinstance(header, str):
- header = re.compile(
- header, re.MULTILINE + re.DOTALL + re.IGNORECASE)
-
- changes = []
- index = 0
- occurrences = {}
- # iterate a copy of headers because headers may be modified
- for name, value in headers.copy():
- # keep track of the occurrence of each header,
- # needed by Milter.Base.chgheader
- if name not in occurrences.keys():
- occurrences[name] = 1
- else:
- occurrences[name] += 1
-
- hdr = f"{name}: {value}"
- if header.search(hdr):
- if self.log:
- self.logger.info(
- f"{qid}: {self.name}: del_header: "
- f"{hdr[0:70]}")
- else:
- self.logger.debug(
- f"{qid}: {self.name}: del_header: "
- f"(occ. {occurrences[name]}): {hdr}")
-
- del headers[index]
- params = [name, "", occurrences[name]]
- changes.append(("mod_header", *params))
- index -= 1
- occurrences[name] -= 1
-
- index += 1
-
- return changes
-
- def add_disclaimer(self, qid, headers, fp, text_template, html_template,
- error_policy):
- """Append or prepend a disclaimer to the email body."""
- changes = []
-
- fp.seek(0)
- msg = message_from_binary_file(fp, policy=default_policy)
-
- html_body = None
- text_body = None
- update_headers = False
-
- try:
- html_body = msg.get_body(preferencelist=("html"))
- text_body = msg.get_body(preferencelist=("plain"))
- except Exception as e:
- self.logger.error(
- f"{qid}: {self.name}: an error occured in "
- f"email.message.EmailMessage.get_body: {e}")
-
- if html_body is None and text_body is None:
- if self.error_policy == "ignore":
- self.logger.info(
- f"{qid}: {self.name}: unable to find email body, "
- f"ignore according to policy")
- return changes
- elif self.error_policy == "reject":
- self.logger.info(
- f"{qid}: {self.name}: unable to find email body, "
- f"reject message according to policy")
- return [
- ("reject", "Message rejected due to missing email body")]
-
- self.logger.info(
- f"{qid}: {self.name}: unable to find email body, "
- f"wrapping original email in a new message envelope")
- msg = MIMEPart()
- msg.add_header("MIME-Version", "1.0")
- msg.set_content(
- "Please see the original email attached.")
- msg.add_alternative(
- "Please see the original email attached.",
- subtype="html")
- fp.seek(0)
- msg.add_attachment(
- fp.read(), maintype="plain", subtype="text",
- filename=f"{qid}.eml")
- html_body = msg.get_body(preferencelist=("html"))
- text_body = msg.get_body(preferencelist=("plain"))
- # content and mime headers may have to be updated because
- # a new message has been created
- update_headers = True
- elif not msg.is_multipart():
- # content and mime headers may have to be updated because
- # we operate on a non-multipart email
- update_headers = True
-
- if text_body is not None:
- if self.log:
- self.logger.info(
- f"{qid}: {self.name}: {self.action} text disclaimer")
- else:
- self.logger.debug(
- f"{qid}: {self.name}: {self.action} text disclaimer")
-
- text = text_body.get_content()
- if self.action == "prepend":
- text = f"{text_template}{text}"
- else:
- text = f"{text}{text_template}"
-
- text_body.set_content(
- text.encode(), maintype="text", subtype="plain")
- text_body.set_param("charset", "UTF-8", header="Content-Type")
-
- if html_body is not None:
- if self.log:
- self.logger.info(
- f"{qid}: {self.name}: {self.action} html disclaimer")
- else:
- self.logger.debug(
- f"{qid}: {self.name}: {self.action} html disclaimer")
-
- soup = BeautifulSoup(html_body.get_content(), "html.parser")
- body = soup.find('body')
- if body:
- # work within the body tag if it is present
- soup = body
-
- if self.action == "prepend":
- soup.insert(0, copy(html_template))
- else:
- soup.append(html_template)
-
- html_body.set_content(
- str(soup).encode(), maintype="text", subtype="html")
- html_body.set_param("charset", "UTF-8", header="Content-Type")
-
- if update_headers:
- for name, value in msg.items():
- name_lower = name.lower()
- if not name_lower.startswith("content-") and \
- name_lower != "mime-version":
- continue
-
- defined = False
- for n, v in headers:
- if n.lower() == name_lower:
- changes += self.mod_header(
- qid, headers, f"^{n}:", ".*", value)
- defined = True
- break
-
- if not defined:
- changes += self.add_header(
- qid, headers, name, value)
-
- msg = msg.as_string(policy=SMTP).encode("ascii", errors="replace")
- fp.seek(0)
- fp.write(msg)
- fp.truncate()
- body_pos = msg.find(b"\r\n\r\n") + 2
- changes.append(("mod_body", body_pos))
- return changes
-
- def execute(self, qid, headers, fp):
- """
- Execute configured modification.
- """
- changes = []
-
- if self.mod_type == "add_header":
- changes = self.add_header(
- qid, headers, self.header, self.value)
- elif self.mod_type == "mod_header":
- changes = self.mod_header(
- qid, headers, self.header, self.search, self.value)
- elif self.mod_type == "del_header":
- changes = self.del_header(
- qid, headers, self.header)
- elif self.mod_type == "add_disclaimer":
- changes = self.add_disclaimer(
- qid, headers, fp, self.text,
- self.html, self.error_policy)
-
- return changes
+ return msg, kwargs
class Rule:
"""
- Rule to implement multiple modifications on emails based on conditions.
+ Rule to implement multiple actions on emails.
"""
- def __init__(self, name, modifications, local_addrs, log, conditions={},
- pretend=False):
- self.logger = logging.getLogger(__name__)
- if pretend:
- self.name = f"{name} (pretend)"
- else:
- self.name = name
+ def __init__(self, name, local_addrs, conditions, actions, pretend=False,
+ loglevel=logging.INFO):
+ logger = logging.getLogger(name)
+ self.logger = CustomLogger(logger, {"name": name})
+ self.logger.setLevel(loglevel)
- self.logger.debug(f"initializing rule '{self.name}'")
- self.log = log
+ if logger is None:
+ logger = logging.getLogger(__name__)
+
+ self.logger = CustomLogger(logger, {"name": name})
+ self.conditions = Conditions(
+ local_addrs=local_addrs,
+ args=conditions,
+ logger=self.logger)
+ self.actions = actions
self.pretend = pretend
+
self._needs = []
- self._local_addrs = []
-
- try:
- for addr in local_addrs:
- self._local_addrs.append(IPNetwork(addr))
- except AddrFormatError as e:
- raise RuntimeError(
- f"{self.name}: unable to parse entry of "
- f"option local_addrs: {e}")
-
- self.conditions = {}
- for option, value in conditions.items():
- if option == "local":
- self.conditions[option] = value
- self.logger.debug(
- f"{self.name}: added condition: {option} = {value}")
- elif option == "hosts":
- self.conditions[option] = []
- try:
- for host in value:
- self.conditions[option].append(IPNetwork(host))
- except AddrFormatError as e:
- raise RuntimeError(
- f"{self.name}: unable to parse entry of "
- f"condition '{option}': {e}")
-
- self.logger.debug(
- f"{self.name}: added condition: {option} = {value}")
- elif option in ["envfrom", "envto"]:
- try:
- self.conditions[option] = re.compile(value, re.IGNORECASE)
- except re.error as e:
- raise RuntimeError(
- f"{self.name}: unable to parse regex of "
- f"condition '{option}': {e}")
-
- self.logger.debug(
- f"{self.name}: added condition: {option} = {value}")
-
- self.modifications = []
- for mod_idx, mod in enumerate(modifications):
- params = {}
- if "name" not in mod:
- mod["name"] = f"Modification #{mod_idx}"
-
- if self.name:
- params["name"] = f"{self.name}: {mod['name']}"
- else:
- params["name"] = mod["name"]
-
- if "log" in mod:
- params["log"] = mod["log"]
- else:
- params["log"] = self.log
-
- if "type" in mod:
- params["mod_type"] = mod["type"]
- else:
- raise RuntimeError(
- f"{params['name']}: mandatory config "
- f"option 'type' not found")
-
- for param in [
- "header", "search", "value", "action", "html_template",
- "text_template", "error_policy"]:
- if param in mod:
- params[param] = mod[param]
-
- modification = Modification(**params)
- for need in modification.needs():
+ for action in actions:
+ for need in action.needs():
if need not in self._needs:
self._needs.append(need)
- self.modifications.append(modification)
- self.logger.debug(
- f"{self.name}: added modification: {mod['name']}")
-
- self.logger.debug(
- f"{self.name}: rule needs: {self._needs}")
+ self.logger.debug("needs: {}".format(", ".join(self._needs)))
def needs(self):
"""Return the needs of this rule."""
return self._needs
- def ignore_host(self, host):
- """Check if host is ignored by this rule."""
- ip = IPAddress(host)
+ def ignores(self, host=None, envfrom=None, envto=None):
+ args = {}
- if "local" in self.conditions:
- is_local = False
- for addr in self._local_addrs:
- if ip in addr:
- is_local = True
- break
+ if host is not None:
+ args["host"] = host
- if is_local != self.conditions["local"]:
- return True
+ if envfrom is not None:
+ args["envfrom"] = envfrom
- if "hosts" in self.conditions:
- # check if host is in list
- for accepted in self.conditions["hosts"]:
- if ip in accepted:
+ if envto is not None:
+ args["envto"] = envto
+
+ if self.conditions.match(args):
+ for action in self.actions:
+ if action.conditions.match(args):
return False
- return True
+ return True
- return False
+ def execute(self, milter, pretend=None):
+ """Execute all actions of this rule."""
+ if pretend is None:
+ pretend = self.pretend
- def ignore_envfrom(self, envfrom):
- """Check if envelope-from address is ignored by this rule."""
- if "envfrom" in self.conditions:
- if not self.conditions["envfrom"].search(envfrom):
- return True
-
- return False
-
- def ignore_envto(self, envto):
- """Check if envelope-to address is ignored by this rule."""
- if "envto" in self.conditions:
- if not isinstance(envto, set):
- envto = set(envto)
- for to in envto:
- if not self.conditions["envto"].search(to):
- return True
-
- return False
-
- def execute(self, qid, headers, data):
- """Execute all modifications of this rule."""
- changes = []
- if self.log:
- self.logger.info(f"{qid}: executing rule '{self.name}'")
- else:
- self.logger.debug(f"{qid}: executing rule '{self.name}'")
-
- for mod in self.modifications:
- self.logger.debug(f"{qid}: executing modification '{mod.name}'")
- changes += mod.execute(qid, headers, data)
-
- if self.pretend:
- changes = []
- return changes
+ for action in self.actions:
+ milter_action = action.execute(milter)
+ if milter_action is not None:
+ return milter_action
class ModifyMilter(Milter.Base):
"""ModifyMilter based on Milter.Base to implement milter communication"""
_rules = []
+ _loglevel = logging.INFO
@staticmethod
def set_rules(rules):
ModifyMilter._rules = rules
+ def set_loglevel(level):
+ ModifyMilter._loglevel = level
+
def __init__(self):
self.logger = logging.getLogger(__name__)
+ self.logger.setLevel(ModifyMilter._loglevel)
+
# save rules, it must not change during runtime
self.rules = ModifyMilter._rules.copy()
@@ -585,13 +157,10 @@ class ModifyMilter(Milter.Base):
self.logger.debug(
f"accepted milter connection from {hostaddr[0]} "
f"port {hostaddr[1]}")
- ip = IPAddress(hostaddr[0])
# remove rules which ignore this host
for rule in self.rules.copy():
- if rule.ignore_host(ip):
- self.logger.debug(
- f"host {hostaddr[0]} is ignored by rule '{rule.name}'")
+ if rule.ignores(host=hostaddr[0]):
self.rules.remove(rule)
if not self.rules:
@@ -605,10 +174,7 @@ class ModifyMilter(Milter.Base):
def envfrom(self, mailfrom, *str):
mailfrom = "@".join(parse_addr(mailfrom)).lower()
for rule in self.rules.copy():
- if rule.ignore_envfrom(mailfrom):
- self.logger.debug(
- f"envelope-from {mailfrom} is ignored by "
- f"rule '{rule.name}'")
+ if rule.ignores(envfrom=mailfrom):
self.rules.remove(rule)
if not self.rules:
@@ -628,10 +194,7 @@ class ModifyMilter(Milter.Base):
def data(self):
try:
for rule in self.rules.copy():
- if rule.ignore_envto(self.recipients):
- self.logger.debug(
- f"envelope-to addresses are ignored by "
- f"rule '{rule.name}'")
+ if rule.ignores(envto=[*self.recipients]):
self.rules.remove(rule)
if not self.rules:
@@ -641,17 +204,18 @@ class ModifyMilter(Milter.Base):
return Milter.ACCEPT
self.qid = self.getsymval('i')
- self.logger.debug(f"{self.qid}: received queue-id from MTA")
- self.headers = None
+ self.logger = CustomLogger(self.logger, {"qid": self.qid})
+ self.logger.debug("received queue-id from MTA")
+ self.fields = None
self.fp = None
for rule in self.rules:
- if "headers" in rule.needs() and self.headers is None:
- self.headers = []
+ if "fields" in rule.needs() and self.fields is None:
+ self.fields = []
- if "data" in rule.needs() and self.fp is None:
+ if "body" in rule.needs() and self.fp is None:
self.fp = BytesIO()
- if None not in [self.headers, self.fp]:
+ if None not in [self.fields, self.fp]:
break
except Exception as e:
@@ -663,21 +227,15 @@ class ModifyMilter(Milter.Base):
def header(self, name, value):
try:
- # remove surrogates from value
- value = value.encode(
- errors="surrogateescape").decode(errors="replace")
- if self.fp is not None:
- self.fp.write(f"{name}: {value}\r\n".encode(
- encoding="ascii", errors="replace"))
-
- if self.headers is not None:
- self.logger.debug(f"{self.qid}: received header: "
- f"{name}: {value}")
+ if self.fields is not None:
+ # remove surrogates from value
+ value = value.encode(
+ errors="surrogateescape").decode(errors="replace")
+ self.logger.debug(f"received header: {name}: {value}")
header = make_header(decode_header(value), errors="replace")
value = str(header).replace("\x00", "")
- self.logger.debug(
- f"{self.qid}: decoded header: {name}: {value}")
- self.headers.append((name, value))
+ self.logger.debug(f"decoded header: {name}: {value}")
+ self.fields.append((name, value))
except Exception as e:
self.logger.exception(
f"an exception occured in header function: {e}")
@@ -685,17 +243,6 @@ class ModifyMilter(Milter.Base):
return Milter.CONTINUE
- def eoh(self):
- try:
- if self.fp is not None:
- self.fp.write(b"\r\n")
- except Exception as e:
- self.logger.exception(
- f"an exception occured in eoh function: {e}")
- return Milter.TEMPFAIL
-
- return Milter.CONTINUE
-
def body(self, chunk):
try:
if self.fp is not None:
@@ -709,43 +256,20 @@ class ModifyMilter(Milter.Base):
def eom(self):
try:
- changes = []
for rule in self.rules:
- changes += rule.execute(self.qid, self.headers, self.fp)
+ milter_action = rule.execute(self)
- mod_body_pos = None
- for mod_type, *params in changes:
- if mod_type in ["add_header", "mod_header", "del_header"]:
- header, value, occurrence = params
- enc_value = replace_illegal_chars(
- Header(s=value).encode())
- if mod_type == "add_header":
- self.logger.debug(f"{self.qid}: milter: add "
- f"header: {header}: {enc_value}")
- self.addheader(header, enc_value, occurrence)
- else:
- if enc_value == "":
- self.logger.debug(
- f"{self.qid}: milter: delete "
- f"header (occ. {occurrence}): "
- f"{header}")
- else:
- self.logger.debug(
- f"{self.qid}: milter: modify "
- f"header (occ. {occurrence}): "
- f"{header}: {enc_value}")
+ if milter_action is not None:
+ if milter_action["action"] == "reject":
+ self.setreply("554", "5.7.0", milter_action["reason"])
+ return Milter.REJECT
- self.chgheader(header, occurrence, enc_value)
- elif mod_type == "mod_body":
- mod_body_pos = params[0]
- elif mod_type == "reject":
- self.setreply("554", "5.7.0", params[0])
- return Milter.REJECT
+ if milter_action["action"] == "accept":
+ return Milter.ACCEPT
+
+ if milter_action["action"] == "discard":
+ return Milter.DISCARD
- if mod_body_pos is not None:
- self.fp.seek(mod_body_pos)
- self.logger.debug(f"{self.qid}: milter: replace body")
- self.replacebody(self.fp.read())
except Exception as e:
self.logger.exception(
f"an exception occured in eom function: {e}")
diff --git a/pymodmilter/actions.py b/pymodmilter/actions.py
new file mode 100644
index 0000000..c9a1119
--- /dev/null
+++ b/pymodmilter/actions.py
@@ -0,0 +1,456 @@
+# PyMod-Milter is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# PyMod-Milter is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with PyMod-Milter. If not, see .
+#
+
+import logging
+import re
+
+from bs4 import BeautifulSoup
+from collections import defaultdict
+from copy import copy
+from email.header import Header
+from email.parser import BytesFeedParser
+from email.message import MIMEPart
+from email.policy import default as default_policy, SMTP
+
+from pymodmilter import CustomLogger, Conditions
+
+
+def _replace_illegal_chars(string):
+ """Replace illegal characters in header values."""
+ return string.replace(
+ "\x00", "").replace(
+ "\r", "").replace(
+ "\n", "")
+
+
+def add_header(field, value, milter, idx=-1, pretend=False,
+ logger=logging.getLogger(__name__)):
+ """Add a mail header field."""
+ header = f"{field}: {value}"
+ if logger.getEffectiveLevel() == logging.DEBUG:
+ logger.debug(f"add_header: {header}")
+ else:
+ logger.info(f"add_header: {header[0:70]}")
+
+ if idx == -1:
+ milter.fields.append((field, value))
+ else:
+ milter.fields.insert(idx, (field, value))
+
+ if pretend:
+ return
+
+ encoded_value = _replace_illegal_chars(
+ Header(s=value).encode())
+ milter.logger.debug(f"milter: addheader: {field}[{idx}]: {encoded_value}")
+ milter.addheader(field, encoded_value, idx)
+
+
+def mod_header(field, value, milter, search=None, pretend=False,
+ logger=logging.getLogger(__name__)):
+ """Change the value of a mail header field."""
+ if not isinstance(field, re.Pattern):
+ field = re.compile(field, re.IGNORECASE)
+
+ if search is not None and not isinstance(search, re.Pattern):
+ search = re.compile(search, re.MULTILINE + re.DOTALL + re.IGNORECASE)
+
+ occ = defaultdict(int)
+
+ for idx, (f, v) in enumerate(milter.fields):
+ occ[f] += 1
+
+ if not field.match(f):
+ continue
+
+ if search is not None:
+ new_v = search.sub(value, v).strip()
+ else:
+ new_v = value.strip()
+
+ if new_v == v:
+ continue
+
+ if not new_v:
+ logger.warning(
+ f"mod_header: resulting value is empty, "
+ f"skip modification")
+ continue
+
+ header = f"{f}: {v}"
+ new_header = f"{f}: {new_v}"
+ if logger.getEffectiveLevel() == logging.DEBUG:
+ logger.debug(f"mod_header: {header}: {new_header}")
+ else:
+ logger.info(f"mod_header: {header[0:70]}: {new_header[0:70]}")
+
+ milter.fields[idx] = (f, new_v)
+
+ if pretend:
+ continue
+
+ encoded_value = _replace_illegal_chars(
+ Header(s=new_v).encode())
+ milter.logger.debug(
+ f"milter: chgheader: {f}[{occ[f]}]: {encoded_value}")
+ milter.chgheader(f, occ[f], encoded_value)
+
+
+def del_header(field, milter, value=None, pretend=False,
+ logger=logging.getLogger(__name__)):
+ """Delete a mail header field."""
+ if not isinstance(field, re.Pattern):
+ field = re.compile(field, re.IGNORECASE)
+
+ if value is not None and not isinstance(value, re.Pattern):
+ value = re.compile(value, re.MULTILINE + re.DOTALL + re.IGNORECASE)
+
+ idx = -1
+ occ = defaultdict(int)
+
+ # iterate a copy of milter.fields because elements may get removed
+ # during iteration
+ for f, v in milter.fields.copy():
+ idx += 1
+ occ[f] += 1
+
+ if not field.match(f):
+ continue
+
+ if value is not None and not value.search(v):
+ continue
+
+ header = f"{f}: {v}"
+ if logger.getEffectiveLevel() == logging.DEBUG:
+ logger.debug(f"del_header: {header}")
+ else:
+ logger.info(f"del_header: {header[0:70]}")
+
+ del milter.fields[idx]
+
+ if not pretend:
+ encoded_value = ""
+ milter.logger.debug(
+ f"milter: chgheader: {f}[{occ[f]}]: {encoded_value}")
+ milter.chgheader(f, occ[f], encoded_value)
+
+ idx -= 1
+ occ[f] -= 1
+
+
+def _get_body_content(msg, body_type):
+ content = None
+ body_part = msg.get_body(preferencelist=(body_type))
+ if body_part is not None:
+ content = body_part.get_content()
+
+ return (body_part, content)
+
+
+def _wrap_message(milter):
+ msg = MIMEPart()
+ msg.add_header("MIME-Version", "1.0")
+
+ msg.set_content(
+ "Please see the original email attached.")
+ msg.add_alternative(
+ "Please see the original email attached.",
+ subtype="html")
+
+ data = b""
+ for field, value in milter.fields:
+ encoded_value = _replace_illegal_chars(
+ Header(s=value).encode())
+ data += field.encode("ascii", errors="replace")
+ data += b": "
+ data += encoded_value.encode("ascii", errors="replace")
+ data += b"\r\n"
+
+ milter.fp.seek(0)
+ data += b"\r\n" + milter.fp.read()
+
+ msg.add_attachment(
+ data, maintype="plain", subtype="text",
+ filename=f"{milter.qid}.eml")
+
+ return msg
+
+
+def _inject_body(milter, msg):
+ if not msg.is_multipart():
+ msg.make_mixed()
+
+ new_msg = MIMEPart()
+ new_msg.add_header("MIME-Version", "1.0")
+ new_msg.set_content("")
+ new_msg.add_alternative("", subtype="html")
+ new_msg.make_mixed()
+ for attachment in msg.iter_attachments():
+ new_msg.attach(attachment)
+
+ return new_msg
+
+
+def add_disclaimer(text, html, action, policy, milter, pretend=False,
+ logger=logging.getLogger(__name__)):
+ """Append or prepend a disclaimer to the mail body."""
+ milter.fp.seek(0)
+ fp = BytesFeedParser(policy=default_policy)
+
+ for field, value in milter.fields:
+ field_lower = field.lower()
+ if not field_lower.startswith("content-") and \
+ field_lower != "mime-version":
+ continue
+ logger.debug(
+ f"feed content header to message object: {field}: {value}")
+ encoded_value = _replace_illegal_chars(
+ Header(s=value).encode())
+ fp.feed(field.encode("ascii", errors="replace"))
+ fp.feed(b": ")
+ fp.feed(encoded_value.encode("ascii", errors="replace"))
+ fp.feed(b"\r\n")
+
+ fp.feed(b"\r\n")
+ logger.debug(f"feed body to message object: {field}: {value}")
+ fp.feed(milter.fp.read())
+
+ logger.debug("parse message")
+ msg = fp.close()
+
+ text_content = None
+ html_content = None
+
+ try:
+ try:
+ logger.debug("try to find a plain and/or html body part")
+ text_body, text_content = _get_body_content(msg, "plain")
+ html_body, html_content = _get_body_content(msg, "html")
+ if text_content is None and html_content is None:
+ raise RuntimeError()
+ except RuntimeError:
+ logger.info(
+ "message does not contain any body part, "
+ "inject empty plain and html body parts")
+ msg = _inject_body(milter, msg)
+ text_body, text_content = _get_body_content(msg, "plain")
+ html_body, html_content = _get_body_content(msg, "html")
+ if text_content is None and html_content is None:
+ raise RuntimeError("no message body present after injecting")
+ except Exception as e:
+ logger.warning(e)
+ if policy == "ignore":
+ logger.info(
+ f"unable to add disclaimer to message body, "
+ f"ignore error according to policy")
+ return
+ elif policy == "reject":
+ logger.info(
+ f"unable to add disclaimer to message body, "
+ f"reject message according to policy")
+ return [
+ ("reject", "Message rejected due to error")]
+
+ logger.info("wrap original message in a new message envelope")
+ msg = _wrap_message(milter)
+ text_body, text_content = _get_body_content(msg, "plain")
+ html_body, html_content = _get_body_content(msg, "html")
+ if text_content is None and html_content is None:
+ raise Exception("no message body present after wrapping, "
+ "give up ...")
+
+ if text_content is not None:
+ logger.info(f"{action} text disclaimer")
+
+ if action == "prepend":
+ content = f"{text}{text_content}"
+ else:
+ content = f"{text_content}{text}"
+
+ text_body.set_content(
+ content.encode(), maintype="text", subtype="plain")
+ text_body.set_param("charset", "UTF-8", header="Content-Type")
+
+ if html_content is not None:
+ logger.info(f"{action} html disclaimer")
+
+ soup = BeautifulSoup(html_content, "html.parser")
+
+ body = soup.find('body')
+ if body:
+ soup = body
+
+ if action == "prepend":
+ soup.insert(0, copy(html))
+ else:
+ soup.append(html)
+
+ html_body.set_content(
+ str(soup).encode(), maintype="text", subtype="html")
+ html_body.set_param("charset", "UTF-8", header="Content-Type")
+
+ try:
+ logger.debug("serialize message as bytes")
+ data = msg.as_bytes(policy=SMTP)
+ except Exception as e:
+ logger.waring(
+ f"unable to serialize message as bytes: {e}")
+ try:
+ logger.warning("try to serialize message as string")
+ data = msg.as_string(policy=SMTP)
+ data = data.encode("ascii", errors="replace")
+ except Exception as e:
+ raise e
+
+ body_pos = data.find(b"\r\n\r\n") + 4
+ milter.fp.seek(0)
+ milter.fp.write(data[body_pos:])
+ milter.fp.truncate()
+
+ if pretend:
+ return
+
+ logger.debug("milter: replacebody")
+ milter.replacebody(data[body_pos:])
+ del data
+
+ fields = {
+ "mime-version": {
+ "field": "MIME-Version",
+ "value": msg.get("MIME-Version"),
+ "modified": False},
+ "content-type": {
+ "field": "Content-Type",
+ "value": msg.get("Content-Type"),
+ "modified": False},
+ "content-transfer-encoding": {
+ "field": "Content-Transfer-Encoding",
+ "value": msg.get("Content-Transfer-Encoding"),
+ "modified": False}}
+
+ for field, value in milter.fields.copy():
+ field_lower = field.lower()
+ if field_lower in fields and fields[field_lower]["value"] is not None:
+ mod_header(field=f"^{field}$", value=fields[field_lower]["value"],
+ milter=milter, pretend=pretend, logger=logger)
+ fields[field_lower]["modified"] = True
+
+ elif field_lower.startswith("content-"):
+ del_header(field=f"^{field}$", milter=milter,
+ pretend=pretend, logger=logger)
+
+ for field in fields.values():
+ if not field["modified"] and field["value"] is not None:
+ add_header(field=field["field"], value=field["value"],
+ milter=milter, pretend=pretend, logger=logger)
+
+
+class Action:
+ """Action to implement a pre-configured action to perform on e-mails."""
+ _types = {
+ "add_header": ["fields"],
+ "del_header": ["fields"],
+ "mod_header": ["fields"],
+ "add_disclaimer": ["fields", "body"]}
+
+ def __init__(self, name, local_addrs, conditions, action_type, args,
+ loglevel=logging.INFO, pretend=False):
+ logger = logging.getLogger(name)
+ self.logger = CustomLogger(logger, {"name": name})
+ self.logger.setLevel(loglevel)
+
+ self.conditions = Conditions(
+ local_addrs=local_addrs,
+ args=conditions,
+ logger=self.logger)
+ self.pretend = pretend
+ self._args = {}
+
+ if action_type not in self._types:
+ raise RuntimeError(f"invalid action_type '{action_type}'")
+ self._needs = self._types[action_type]
+
+ try:
+ if action_type == "add_header":
+ self._func = add_header
+ self._args["field"] = args["header"]
+ self._args["value"] = args["value"]
+ if "idx" in args:
+ self._args["idx"] = args["idx"]
+
+ elif action_type in ["mod_header", "del_header"]:
+ args["field"] = args["header"]
+ del args["header"]
+ regex_args = ["field"]
+
+ if action_type == "mod_header":
+ self._func = mod_header
+ self._args["value"] = args["value"]
+ regex_args.append("search")
+ elif action_type == "del_header" and "value" in args:
+ self._func = del_header
+ regex_args.append("value")
+
+ for arg in regex_args:
+ try:
+ self._args[arg] = re.compile(
+ args[arg],
+ re.MULTILINE + re.DOTALL + re.IGNORECASE)
+ except re.error as e:
+ raise RuntimeError(
+ f"unable to parse {arg} regex: {e}")
+
+ elif action_type == "add_disclaimer":
+ self._func = add_disclaimer
+ if args["action"] not in ["append", "prepend"]:
+ raise RuntimeError(f"invalid action '{args['action']}'")
+
+ self._args["action"] = args["action"]
+
+ if args["error_policy"] not in ["wrap", "ignore", "reject"]:
+ raise RuntimeError(f"invalid policy '{args['policy']}'")
+
+ self._args["policy"] = args["error_policy"]
+
+ try:
+ with open(args["html_file"], "r") as f:
+ html = BeautifulSoup(
+ f.read(), "html.parser")
+ body = html.find('body')
+ if body:
+ # just use content within the body tag if present
+ html = body
+ self._args["html"] = html
+ with open(args["text_file"], "r") as f:
+ self._args["text"] = f.read()
+ except IOError as e:
+ raise RuntimeError(f"unable to read template: {e}")
+
+ except KeyError as e:
+ raise RuntimeError(
+ f"mandatory argument not found: {e}")
+
+ def needs(self):
+ """Return the needs of this action."""
+ return self._needs
+
+ def execute(self, milter, pretend=None):
+ """Execute configured action."""
+ if pretend is None:
+ pretend = self.pretend
+
+ logger = CustomLogger(self.logger, {"qid": milter.qid})
+
+ return self._func(
+ milter=milter, pretend=pretend, logger=logger, **self._args)
diff --git a/pymodmilter/conditions.py b/pymodmilter/conditions.py
new file mode 100644
index 0000000..c0fbf9a
--- /dev/null
+++ b/pymodmilter/conditions.py
@@ -0,0 +1,134 @@
+# PyMod-Milter is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# PyMod-Milter is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with PyMod-Milter. If not, see .
+#
+
+import logging
+import re
+
+from netaddr import IPAddress, IPNetwork, AddrFormatError
+
+
+class Conditions:
+ """Conditions to implement conditions for rules and actions."""
+
+ def __init__(self, local_addrs, args, logger=None):
+ if logger is None:
+ logger = logging.getLogger(__name__)
+
+ self._local_addrs = []
+ self.logger = logger
+ self._args = {}
+
+ try:
+ for addr in local_addrs:
+ self._local_addrs.append(IPNetwork(addr))
+ except AddrFormatError as e:
+ raise RuntimeError(f"invalid address in local_addrs: {e}")
+
+ try:
+ if "local" in args:
+ logger.debug(f"condition: local = {args['local']}")
+ self._args["local"] = args["local"]
+
+ if "hosts" in args:
+ logger.debug(f"condition: hosts = {args['hosts']}")
+ self._args["hosts"] = []
+ try:
+ for host in args["hosts"]:
+ self._args["hosts"].append(IPNetwork(host))
+ except AddrFormatError as e:
+ raise RuntimeError(f"invalid address in hosts: {e}")
+
+ if "envfrom" in args:
+ logger.debug(f"condition: envfrom = {args['envfrom']}")
+ try:
+ self._args["envfrom"] = re.compile(
+ args["envfrom"], re.IGNORECASE)
+ except re.error as e:
+ raise RuntimeError(f"unable to parse envfrom regex: {e}")
+
+ if "envto" in args:
+ logger.debug(f"condition: envto = {args['envto']}")
+ try:
+ self._args["envto"] = re.compile(
+ args["envto"], re.IGNORECASE)
+ except re.error as e:
+ raise RuntimeError(f"unable to parse envto regex: {e}")
+
+ except KeyError as e:
+ raise RuntimeError(
+ f"mandatory argument not found: {e}")
+
+ def match(self, args):
+ if "host" in args:
+ ip = IPAddress(args["host"])
+
+ if "local" in self._args:
+ is_local = False
+ for addr in self._local_addrs:
+ if ip in addr:
+ is_local = True
+ break
+
+ if is_local != self._args["local"]:
+ self.logger.debug(
+ f"ignore host {args['host']}, "
+ f"condition local does not match")
+ return False
+
+ self.logger.debug(
+ f"condition local matches for host {args['host']}")
+
+ if "hosts" in self._args:
+ found = False
+ for addr in self._args["hosts"]:
+ if ip in addr:
+ found = True
+ break
+
+ if not found:
+ self.logger.debug(
+ f"ignore host {args['host']}, "
+ f"condition hosts does not match")
+ return False
+
+ self.logger.debug(
+ f"condition hosts matches for host {args['host']}")
+
+ if "envfrom" in args and "envfrom" in self._args:
+ if not self._args["envfrom"].match(args["envfrom"]):
+ self.logger.debug(
+ f"ignore envelope-from address {args['envfrom']}, "
+ f"condition envfrom does not match")
+ return False
+
+ self.logger.debug(
+ f"condition envfrom matches for "
+ f"envelope-from address {args['envfrom']}")
+
+ if "envto" in args and "envto" in self._args:
+ if not isinstance(args["envto"], list):
+ args["envto"] = [args["envto"]]
+
+ for envto in args["envto"]:
+ if not self._args["envto"].match(envto):
+ self.logger.debug(
+ f"ignore envelope-to address {args['envto']}, "
+ f"condition envto does not match")
+ return False
+
+ self.logger.debug(
+ f"condition envto matches for "
+ f"envelope-to address {args['envto']}")
+
+ return True
diff --git a/pymodmilter/run.py b/pymodmilter/run.py
index 924ab2a..d5a3666 100644
--- a/pymodmilter/run.py
+++ b/pymodmilter/run.py
@@ -12,7 +12,6 @@
# along with PyMod-Milter. If not, see .
#
-
import Milter
import argparse
import logging
@@ -24,77 +23,103 @@ from re import sub
from pymodmilter import Rule, ModifyMilter
from pymodmilter.version import __version__ as version
+from pymodmilter.actions import Action
def main():
"Run PyMod-Milter."
- # parse command line
parser = argparse.ArgumentParser(
description="PyMod milter daemon",
formatter_class=lambda prog: argparse.HelpFormatter(
prog, max_help_position=45, width=140))
+
parser.add_argument(
"-c", "--config", help="Config file to read.",
default="/etc/pymodmilter/pymodmilter.conf")
+
parser.add_argument(
"-s",
"--socket",
help="Socket used to communicate with the MTA.",
default="")
+
parser.add_argument(
"-d",
"--debug",
help="Log debugging messages.",
action="store_true")
+
parser.add_argument(
"-t",
"--test",
help="Check configuration.",
action="store_true")
+
parser.add_argument(
"-v", "--version",
help="Print version.",
action="version",
version=f"%(prog)s ({version})")
+
args = parser.parse_args()
- # setup logging
- loglevel = logging.INFO
- logname = "pymodmilter"
- syslog_name = logname
- if args.debug:
- loglevel = logging.DEBUG
- logname = f"{logname}[%(name)s]"
- syslog_name = f"{syslog_name}: [%(name)s] %(levelname)s"
+ loglevels = {
+ "error": logging.ERROR,
+ "warning": logging.WARNING,
+ "info": logging.INFO,
+ "debug": logging.DEBUG
+ }
root_logger = logging.getLogger()
- root_logger.setLevel(loglevel)
+ root_logger.setLevel(logging.DEBUG)
# setup console log
stdouthandler = logging.StreamHandler(sys.stdout)
- stdouthandler.setLevel(logging.DEBUG)
- formatter = logging.Formatter("%(message)s")
- stdouthandler.setFormatter(formatter)
+ stdouthandler.setFormatter(
+ logging.Formatter("%(asctime)s - %(levelname)s: %(message)s"))
root_logger.addHandler(stdouthandler)
+
+ # setup syslog
+ sysloghandler = logging.handlers.SysLogHandler(
+ address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_MAIL)
+ sysloghandler.setFormatter(
+ logging.Formatter("pymodmilter: %(message)s"))
+ root_logger.addHandler(sysloghandler)
+
logger = logging.getLogger(__name__)
+ if not args.debug:
+ logger.setLevel(logging.INFO)
+
try:
- # read config file
- logger.debug("parsing config file")
try:
with open(args.config, "r") as fh:
- config = loads(
- sub(r"(?m)^\s*#.*\n?", "", fh.read()))
+ config = sub(r"(?m)^\s*#.*\n?", "", fh.read())
+ config = loads(config)
except Exception as e:
+ for num, line in enumerate(config.splitlines()):
+ logger.error(f"{num+1}: {line}")
raise RuntimeError(
f"unable to parse config file: {e}")
- logger.debug("preparing milter configuration ...")
-
- # default values for global config if not set
if "global" not in config:
config["global"] = {}
+ if "loglevel" not in config["global"]:
+ config["global"]["loglevel"] = "info"
+
+ if args.debug:
+ loglevel = logging.DEBUG
+ else:
+ loglevel = loglevels[config["global"]["loglevel"]]
+
+ logger.setLevel(loglevel)
+
+ logger.debug("prepar milter configuration")
+
+ if "pretend" not in config["global"]:
+ config["global"]["pretend"] = False
+
if args.socket:
socket = args.socket
elif "socket" in config["global"]:
@@ -104,64 +129,110 @@ def main():
f"listening socket is neither specified on the command line "
f"nor in the configuration file")
- if "local_addrs" not in config["global"]:
- config["global"]["local_addrs"] = [
+ if "local_addrs" in config["global"]:
+ local_addrs = config["global"]["local_addrs"]
+ else:
+ local_addrs = [
+ "::1/128",
"127.0.0.0/8",
"10.0.0.0/8",
"172.16.0.0/12",
"192.168.0.0/16"]
- if "log" not in config["global"]:
- config["global"]["log"] = True
-
- if "pretend" not in config["global"]:
- config["global"]["pretend"] = False
-
- # check if mandatory sections are present in config
- for section in ["rules"]:
- if section not in config:
- raise RuntimeError(
- f"mandatory config section '{section}' not found")
+ if "rules" not in config:
+ raise RuntimeError(
+ f"mandatory config section 'rules' not found")
if not config["rules"]:
raise RuntimeError("no rules configured")
- rules = []
- # iterate configured rules
- for rule_idx, rule in enumerate(config["rules"]):
- params = {}
- # set default values if not specified in config
- if "name" in rule:
- params["name"] = rule["name"]
- else:
- params["name"] = f"Rule #{rule_idx}"
+ logger.debug("initialize rules ...")
- if "log" in rule:
- params["log"] = rule["log"]
+ rules = []
+ for rule_idx, rule in enumerate(config["rules"]):
+ if "name" in rule:
+ rule_name = rule["name"]
else:
- params["log"] = config["global"]["log"]
+ rule_name = f"Rule #{rule_idx}"
+
+ logger.debug(f"prepare rule {rule_name} ...")
+
+ if "actions" not in rule:
+ raise RuntimeError(
+ f"{rule_name}: mandatory config "
+ f"section 'actions' not found")
+
+ if not rule["actions"]:
+ raise RuntimeError("{rule_name}: no actions configured")
+
+ if args.debug:
+ rule_loglevel = logging.DEBUG
+ elif "loglevel" in rule:
+ rule_loglevel = loglevels[rule["loglevel"]]
+ else:
+ rule_loglevel = loglevels[config["global"]["loglevel"]]
if "pretend" in rule:
- params["pretend"] = rule["pretend"]
+ rule_pretend = rule["pretend"]
else:
- params["pretend"] = config["global"]["pretend"]
+ rule_pretend = config["global"]["pretend"]
- if "local_addrs" in rule:
- params["local_addrs"] = rule["local_addrs"]
- else:
- params["local_addrs"] = config["global"]["local_addrs"]
+ actions = []
+ for action_idx, action in enumerate(rule["actions"]):
+ if "name" in action:
+ action_name = action["name"]
+ else:
+ action_name = f"Action #{action_idx}"
- if "conditions" in rule:
- params["conditions"] = rule["conditions"]
+ if args.debug:
+ action_loglevel = logging.DEBUG
+ elif "loglevel" in action:
+ action_loglevel = loglevels[action["loglevel"]]
+ else:
+ action_loglevel = rule_loglevel
- if "modifications" in rule:
- params["modifications"] = rule["modifications"]
- else:
- raise RuntimeError(
- f"{rule['name']}: mandatory config section "
- f"'modifications' not found")
+ if "pretend" in action:
+ action_pretend = action["pretend"]
+ else:
+ action_pretend = rule_pretend
- rules.append(Rule(**params))
+ if "type" not in action:
+ raise RuntimeError(
+ f"{rule_name}: {action_name}: mandatory config "
+ f"section 'actions' not found")
+
+ if "conditions" not in action:
+ action["conditions"] = {}
+
+ try:
+ actions.append(
+ Action(
+ name=action_name,
+ local_addrs=local_addrs,
+ conditions=action["conditions"],
+ action_type=action["type"],
+ args=action,
+ loglevel=action_loglevel,
+ pretend=action_pretend))
+ except RuntimeError as e:
+ logger.error(f"{action_name}: {e}")
+ sys.exit(253)
+
+ if "conditions" not in rule:
+ rule["conditions"] = {}
+
+ try:
+ rules.append(
+ Rule(
+ name=rule_name,
+ local_addrs=local_addrs,
+ conditions=rule["conditions"],
+ actions=actions,
+ loglevel=rule_loglevel,
+ pretend=rule_pretend))
+ except RuntimeError as e:
+ logger.error(f"{rule_name}: {e}")
+ sys.exit(254)
except RuntimeError as e:
logger.error(e)
@@ -171,27 +242,22 @@ def main():
print("Configuration ok")
sys.exit(0)
- # change log format for runtime
- formatter = logging.Formatter(
- f"%(asctime)s {logname}: [%(levelname)s] %(message)s",
- datefmt="%Y-%m-%d %H:%M:%S")
+ # setup console log for runtime
+ formatter = logging.Formatter("%(asctime)s - %(levelname)s: %(message)s")
stdouthandler.setFormatter(formatter)
-
- # setup syslog
- sysloghandler = logging.handlers.SysLogHandler(
- address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_MAIL)
- sysloghandler.setLevel(loglevel)
- formatter = logging.Formatter(f"{syslog_name}: %(message)s")
- sysloghandler.setFormatter(formatter)
- root_logger.addHandler(sysloghandler)
+ stdouthandler.setLevel(logging.DEBUG)
logger.info("pymodmilter starting")
ModifyMilter.set_rules(rules)
+ ModifyMilter.set_loglevel(loglevels[config["global"]["loglevel"]])
# register milter factory class
Milter.factory = ModifyMilter
Milter.set_exception_policy(Milter.TEMPFAIL)
+ if args.debug:
+ Milter.setdbg(1)
+
rc = 0
try:
Milter.runmilter("pymodmilter", socketname=socket, timeout=30)