headers are getting corrupted, remove surrogates again

Signed-off-by: Thomas Oettli <spacefreak@noop.ch>

MANIFEST.in

@@ -1,3 +1,3 @@
 include LICENSE README.md
-recursive-include docs *
+recursive-include pyquarantine/docs *
-recursive-include misc *
+recursive-include pyquarantine/misc *

README.md

@@ -1,151 +1,446 @@
 # pyquarantine-milter
-A pymilter based sendmail/postfix pre-queue filter with the ability to quarantine e-mails, sending notifications
+A pymilter based sendmail/postfix pre-queue filter with the ability to ...
-to recipients and respond with a milter-action (ACCEPT, DISCARD or REJECT).
+* modify e-mail headers (add, modify, delete)
+* store e-mails
+* send e-mail notifications
+* append and prepend disclaimers to e-mail text parts
+* quarantine e-mails (store e-mail, optionally notify receivers)
 
-It is useful in many cases because of its felxible configuration and the ability to handle any number of quarantines.
+It is useful in many cases due to its flexible configuration and the ability to handle any number of quarantines and modifications sequential and conditional. Storages and whitelists used by quarantines can be managed with the built-in CLI.  
-The MTA can check e-mail headers using regular expressions to determine if and which quarantine to use. 
-Each quarantine can be configured with a quarantine type, notification type, whitelist and an action to respond with.
 
-Addionally, pyquarantine-milter provides a sanitized, harmless version of the text parts of e-mails, which can be embedded in e-mail notifications. This makes it easier for users to decide, if a match is a false-positive or not. If a matching quarantine provides a quarantine ID of the original e-mail, it is also available as a template variable. This is useful if you want to add links to a webservice to notification e-mails, to give your users the ability to release e-mails or whitelist the from-address for example. The webservice then releases the e-mail from the quarantine.
+Addionally, pyquarantine-milter provides a sanitized, harmless version of the text parts of e-mails as template variable, which can be embedded in e-mail notifications. This makes it easier for users to decide, if a match is a false-positive or not.  
+It is also possible to use any metavariable as template variable (e.g. storage ID, envelope-from address, ...). This may be used to give your users the ability to release e-mails or whitelist the from-address for example. A webservice then releases the e-mail from the quarantine.  
 
-The project is currently in alpha status, but will soon be used in a productive enterprise environment and possibly existing bugs will be fixed.
+The project is currently in beta status, but it is already used in a productive enterprise environment that processes about a million e-mails per month.
 
-## Requirements
+## Dependencies
-* pymilter <https://pythonhosted.org/pymilter/>
+pyquarantine is depending on these python packages, they are installed automatically if you are working with pip.
-* netaddr <https://github.com/drkjam/netaddr/>
+* [jsonschema](https://github.com/Julian/jsonschema)
-* peewee <https://github.com/coleifer/peewee/>
+* [pymilter](https://github.com/sdgathman/pymilter)
-* BeautifulSoup <https://www.crummy.com/software/BeautifulSoup/>
+* [netaddr](https://github.com/drkjam/netaddr)
+* [peewee](https://github.com/coleifer/peewee)
+* [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/)
+
+## Installation
+```sh
+# install pyquarantine with pip.
+pip install pyquarantine
+
+# install service files, default config and templates
+pyquarantine-milter --install
+
+# copy default config file
+cp /etc/pyquarantine/pyquarantine.conf.default /etc/pyquarantine/pyquarantine.conf
+
+# Check the validity of the your config file.
+pyquarantine-milter -t
+```
+## Autostart
+The following init systems are supported.
+
+### systemd
+```sh
+# start the daemon at boot time
+systemctl enable pyquarantine-milter.service
+
+# start the daemon immediately
+systemctl start pyquarantine-milter.service
+```
+
+### OpenRC (Gentoo)
+```sh
+# start the daemon at boot time
+rc-update add pyquarantine-milter default
+
+# start the daemon immediately
+rc-service pyquarantine-milter start
+```
 
 ## Configuration
-The pyquarantine module uses an INI-style configuration file. The sections are described below. If you have to specify a path in the config, you can always use a relative path to the last loaded config file.
+pyquarantine uses a config file in JSON format. It has to be JSON valid with the exception of allowed comment lines starting with **#**.  
 
-### Section "global"
+The basic idea is to configure rules that contain actions. Both rules and actions may have conditions. An example of using rules is separating incoming and outgoing e-mails using the local condition. Rules and actions are always processed in the given order.  
-Any available configuration option can be set in the global section as default instead of in a quarantine section.  
 
-The following configuration options are mandatory in the global section:
+### Global
-* **quarantines**  
+Global config options:
-  Comma-separated, ordered list of active quarantines. For each, there must be a section of the same name in the configuration.
+* **socket** (optional)  
-* **preferred_quarantin_action**  
+  Socket used to communicate with the MTA. If it is not specified in the config, it has to be set as command line option.
-  Defines which quarantine action should be preferred if multiple quarantines are matching for multiple recipients.  
+* **local_addrs** (optional, default: [fe80::/64, ::1/128, 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16])  
-  If at least one recipient receives the original e-mail due to whitelisting, the action is always ACCEPT.
+  List of hosts and network addresses which are considered local. It is used for the condition option [local](#Conditions).  
-  Possible values are:
+* **loglevel**  (optional, default: "info")  
-  * **first**
+  Set the log level. This option may be overriden by any rule or action object.  
-  * **last**
+  Possible values:
+  * **error**  
+  * **warning**  
+  * **info**  
+  * **debug**  
+* **pretend** (optional, default: false)  
+  Pretend actions, for test purposes. This option may be overriden by any rule or action object.  
+* **rules**  
+  List of rule objects.
 
-### Quarantine sections
+### Rule
-The following configuration options are mandatory in each quarantine section:
+Config options for rule objects:
-* **regex**  
+* **name**  
-  Case insensitive regular expression to filter e-mail headers.
+  Name of the rule.  
-* **quarantine_type**  
+* **actions**  
-  One of the quarantine-types described below.
+  List of action objects.
+* **conditions** (optional)  
+  See section [Conditions](#Conditions).
+* **loglevel** (optional)  
+  See section [Global](#Global).
+* **pretend** (optional)  
+  See section [Global](#Global).
+
+### Action
+Config options for action objects:
+* **name**  
+  Name of the action.
+* **type**  
+  See section [Action types](#Action-types).
+* **options**  
+  Options depending on the action type, see section [Action types](#Action-types).
+* **conditions** (optional)  
+  See section [Conditions](#Conditions).
+* **loglevel** (optional)  
+  See section [Global](#Global).
+* **pretend** (optional)  
+  See section [Global](#Global).
+
+### Conditions
+Config options for conditions objects:
+* **local** (optional)  
+  Matches outgoing e-mails (sender address matches **local_addrs**) if set to **true** or matches incoming e-mails if set to **false**.
+* **hosts** (optional)  
+  Matches e-mails originating from the given list of hosts and network addresses.
+* **envfrom** (optional)  
+  Matches e-mails for which the envelope-from address matches the given regular expression.
+* **envto** (optional)  
+  Matches e-mails for which all envelope-to addresses match the given regular expression.
+* **headers** (optional)  
+  Matches e-mails for which all regular expressions in the given list are matching at least one e-mail header.
+* **whitelist** (optional)  
+  Matches e-mails for which the whitelist has no entry for the envelope-from and envelope-to address combination, see section [Whitelist](#Whitelist) for details.
+* **var** (optional)  
+  Matches e-mails for which a previous action or condition has set the given metavariable.
+* **metavar** (optional)  
+  Prefix for the name of metavariables which are possibly provided by the **envfrom**, **envto** or **headers** condition. Meta variables will be provided if the regular expressions contain named subgroups, see [python.re](https://docs.python.org/3/library/re.html) for details.  
+  If not set, no metavariables will be provided.
+
+### Whitelist
+Config options for whitelist objects:
+* **type**  
+  See section [Whitelist types](#Whitelist-types).
+
+### Action types
+Available action types:
+##### add_header
+Add new header.  
+Options:
+* **field**  
+  Name of the header.
+* **value**  
+  Value of the header.
+
+##### del_header
+Delete header(s).  
+Options:
+* **field**  
+  Regular expression to match against header names.
+* **value** (optional)  
+  Regular expression to match against the headers value.
+
+##### mod_header
+Modify header(s).  
+Options:
+* **field**  
+  Regular expression to match against header names.
+* **search** (optional)  
+  Regular expression to match against header values. You may use subgroups or named subgroups (python syntax) to include parts of the original value in the new value.
+* **value**  
+  New value of the header.
+
+##### add_disclaimer
+Append or prepend disclaimer to text and/or html body parts.  
+Options:
 * **action**  
-  One of the actions described below.
+  Action to perform with the disclaimer.  
-* **notification_type**  
+  Possible values:
-  One of the notification types described below.
+  * append
-* **whitelist_type**  
+  * prepend
-  One of the whitelist types described below.
+* **html_template**  
+  Path to a file which contains the html representation of the disclaimer.
+* **text_template**  
+  Path to a file which contains the text representation of the disclaimer.
+* **error_policy** (optional, default: "wrap")  
+  Set the error policy in case the disclaimer cannot be added (e.g. if the html part cannot be parsed).  
+  Possible values:
+  * **wrap**  
+    A new e-mail body is generated with the disclaimer as body and the original e-mail attached.
+  * **ignore**  
+    Ignore the error and do nothing.
+  * **reject**  
+    Reject the e-mail.
+* **add_html_body** (optional, default: false)  
+  Generate a html body with the content of the text body if no html body is present.
+
+##### store
+Store e-mail.  
+Options:
+* **type**  
+  See section [Storage types](#Storage-types).
+* **original** (optional, default: false)  
+  If set to true, store the message as received by the MTA instead of storing the current state of the message, that may was modified already by other actions.
+* **metadata** (optional, default: false)  
+  Store metadata.
+* **metavar**  (optional)  
+  Prefix for the metavariable names. If not set, no metavariables will be provided.  
+  The storage provides the following metavariables:
+  * **ID** (the storage ID of the e-mail)  
+  * **DATAFILE** (path to the data file)  
+  * **METAFILE** (path to the meta file if **metadata** is set to **true**)  
+
+##### notify
+Send notification.  
+Options:
+* **type**  
+  See section [Notification types](#Notification-types).
+
+##### quarantine
+Quarantine e-mail.  
+Options:
+* **store**  
+  Options for e-mail storage, see action [store](#store).  
+  If the option **metadata** is not specificall set for this storage, it will be set to true.
 * **smtp_host**  
-  SMTP host used to release original e-mails from the quarantine.
+  SMTP host used to release e-mails from quarantine.
 * **smtp_port**  
-  SMTP port
+  SMTP port used to release e-mails from quarantine.
-
+* **notify** (optional)  
-The following configuration options are optional in each quarantine section:
+  Options for e-mail notifications, see action [notify](#notify).
-* **ignore_hosts**  
+* **milter_action** (optional)  
-  Comma-separated list of host and network addresses to be ignored by this quarantine.
+  Milter action to perform. If set, no further rules or actions will be processed.  
-* **reject_reason**  
+  Please think carefully what you set here or your MTA may do something you do not want it to do.  
-  Reason to return to the client if action is set to reject.
+  Possible values:
-
-
-### Quarantine types
-* **NONE**  
-  Original e-mails scrapped, sent to nirvana, black-holed or however you want to call it.
-
-* **FILE**  
-  Original e-mails are stored on the filesystem with a unique filename. The filename is available as a
-  template variable used in notifiaction templates.  
-  The following configuration options are mandatory for this quarantine type:
-  * **quarantine_directory**  
-    The directory in which quarantined e-mails are stored.
-
-
-### Notification types
-* **NONE**  
-  No quarantine notifications will be sent.
-
-* **EMAIL**  
-  Quarantine e-mail notifications are sent to recipients. The SMTP host and port, E-mail template, from-address and the subject are configurable for each quarantine. The templates must contain the notification e-mail text in HTML form.  
-
-  The following template variables are available:
-  * **{EMAIL_ENVELOPE_FROM}**  
-    E-mail from-address received by the milter.
-  * **{EMAIL_FROM}**  
-    Value of the from header of the original e-mail.
-  * **{EMAIL_TO}**  
-    E-mail recipient address of this notification.
-  * **{EMAIL_SUBJECT}**  
-    Configured e-mail subject.
-  * **{EMAIL_QUARANTINE_ID}**  
-    Quarantine-ID of the original e-mail if available, empty otherwise.
-  * **{EMAIL_HTML_TEXT}**  
-    Sanitized version of the e-mail text part of the original e-mail. Only harmless HTML tags and attributes are included. Images are optionally stripped or replaced with the image set by notification_email_replacement_img option.
-
-  Some template variables are only available if the regex of the matching quarantine contains subgroups or named subgroups (python syntax). This is useful to include information (e.g. virus names, spam points, ...) of the matching header within the notification.  
-  The following dynamic template variables are available:
-  * **{SUBGROUP_n}**  
-    Content of a subgroup, 'n' will be replaced by the index number of each subgroup, starting with 0.
-  * **{subgroup_name}**  
-    Content of a named subgroup, 'subgroup_name' will be replaced by its name.
-
-  The following configuration options are mandatory for this notification type:
-  * **notification_email_smtp_host**  
-    SMTP host used to send notification e-mails.
-  * **notification_email_smtp_port**  
-    SMTP port.
-  * **notification_email_envelope_from**  
-    Notification e-mail envelope from-address.
-  * **notification_email_from**  
-    Value of the notification e-mail from header. Optionally, you may use the EMAIL_FROM template variable described above.
-  * **notification_email_subject**  
-    Notification e-mail subject. Optionally, you may use the EMAIL_SUBJECT template variable described above.
-  * **notification_email_template**  
-    Path to the notification e-mail template. It is hold in memory during runtime.
-  * **notification_email_embedded_imgs**  
-    Comma-separated list of images to embed into the notification e-mail. The Content-ID of each image will be set to the filename, so you can reference it from the e-mail template. All images are hold in memory during runtime.  
-    Leave empty to disable.
-
-  The following configuration options are optional for this notification type:
-  * **notification_email_strip_images**  
-    Enable to strip images from e-mails. This option superseeds notification_email_replacement_img.
-  * **notification_email_replacement_img**  
-    Path to an image to replace images in e-mails. It is hold in memory during runtime.
-  * **notification_email_parser_lib**  
-    HTML parser library used to parse text part of emails.
-
-
-### Actions
-Every quarantine responds with a milter-action if an e-mail header matches the configured regular expression. Please think carefully what you set here or your MTA will do something you do not want.  
-The following actions are available:
   * **ACCEPT**  
     Tell the MTA to continue processing the e-mail.
-* **DISCARD**  
-  Tell the MTA to silently discard the e-mail.
   * **REJECT**  
     Tell the MTA to reject the e-mail.
+  * **DISCARD**  
+    Tell the MTA to silently discard the e-mail.
+* **reject_reason** (optional, default: "Message rejected")  
+  Reject message sent to MTA if milter_action is set to reject.
+* **whitelist** (optional)  
+  Options for a whitelist object, see section [Whitelist](#Whitelist).
 
+### Storage types
+Available storage types:
+##### file
+File storage.  
+Options:
+* **directory**  
+  Directory used to store e-mails.
+* **metadata** (optional, default: false)  
+  Store metadata file.
+* **mode**  (optional, default: system default)  
+  File mode when new files are created.
+
+### Notification types
+Available notification types:
+##### email
+Generate an e-mail notification based on a template and send it to the original recipient.  
+Available template variables:
+* **{ENVELOPE_FROM}**  
+  Sender address received by the milter.
+* **{ENVELOPE_FROM_URL}**  
+  Like ENVELOPE_FROM, but URL encoded.
+* **{ENVELOPE_TO}**  
+  Recipient address of this notification.
+* **{ENVELOPE_TO_URL}**  
+  Like ENVELOPE_TO, but URL encoded.
+* **{FROM}**  
+  Value of the FROM header of the e-mail.
+* **{TO}**  
+  Value of the TO header of the e-mail.
+* **{SUBJECT}**  
+  Configured e-mail notification subject.
+* **{HTML_TEXT}**  
+  Sanitized version of the e-mail text part of the e-mail. Only harmless HTML tags and attributes are included. Images are optionally stripped or replaced with the image set by **repl_img** option.
+
+Additionally, every metavariable set by previous conditions or actions are also available as template variables. This is useful to include additional information (e.g. virus names, spam points, ...) within the notification.  
+
+Options:
+* **smtp_host**  
+  SMTP host used to send notifications.
+* **smtp_port**  
+  SMTP port used to send notifications.
+* **envelope_from**  
+  Envelope-From address.
+* **from_header**  
+  Value of the From header. You may use the template variable **{FROM}**.
+* **subject**  
+  Subject of the notification e-mail. You  may use the template variable **{SUBJECT}**.
+* **template**  
+  Path to the HTML template.  
+* **strip_imgs** (optional, default: false)  
+  Strip images from e-mail. This option superseeds **repl_img**.
+* **repl_img** (optional)  
+  Image used to replace all images in the e-mail HTML part.
+* **embed_imgs** (optional)  
+  List of images to embed into the notification e-mail. The Content-ID of each image will be set to the filename, so you can reference it from the e-mail template.
 
 ### Whitelist types
-* **NONE**  
+Available whitelist types:
-  No whitelist will be used.
+##### db
-
+Whitelist stored in database. The table is created automatically if it does not exist yet.  
-* **DB**  
+Options:
-  A database whitelist will be used. All database types supported by peewee are available.  
+* **connection**  
-
+  Database connection string, see [Peewee Playhouse Extension](https://docs.peewee-orm.com/en/latest/peewee/playhouse.html#db-url).
-  The following configuration options are mandatory for this whitelist type:
+* **table**  
-  * **whitelist_db_connection**  
-  Database connection string (e.g. mysql://user:password@host:port).  
-
-  * **whitelist_db_table**  
   Database table to use.
 
+### Integration with MTA
+For integration with Postfix, see [Postix Milter Readme](http://www.postfix.org/MILTER_README.html).  
+For integration with sendmail, see [Pymilter Sendmail Readme](https://pythonhosted.org/pymilter/milter_api/installation.html#config).
+
+## Examples
+Here are some config examples.
+
+### Virus and spam quarantine for incoming e-mails
+In this example it is assumed, that another milter (e.g. Amavisd or Rspamd) adds headers to spam and virus e-mails.
+```json
+{
+    "socket": "unix:/tmp/pyquarantine.sock",
+    "rules": [
+        {
+            "name": "inbound",
+            "conditions": {
+                "local": false
+            },
+            "actions": [
+                {
+                    "name": "virus",
+                    "type": "quarantine",
+                    "conditions": {
+                        "headers": ["^X-Virus: Yes"],
+                    },
+                    "options": {
+                        "store": {
+                            "type": "file",
+                            "directory": "/mnt/data/quarantine/virus",
+                        },
+                        "notify": {
+                            "type": "email",
+                            "smtp_host": "localhost",
+                            "smtp_port": 2525,
+                            "envelope_from": "notifications@example.com",
+                            "from_header": "{FROM}",
+                            "subject": "[VIRUS] {SUBJECT}",
+                            "template": "/etc/pyquarantine/templates/notification.template",
+                            "repl_img": "/etc/pyquarantine/templates/removed.png"
+                        },
+                        "smtp_host": "localhost",
+                        "smtp_port": 2525,
+                        "milter_action": "REJECT",
+                        "reject_reason": "Message rejected due to virus"
+                    }
+                }, {
+                    "name": "spam",
+                    "type": "quarantine",
+                    "conditions": {
+                        "headers": ["^X-Spam: Yes"]
+                    },
+                    "options": {
+                        "store": {
+                            "type": "file",
+                            "directory": "/mnt/data/quarantine/spam",
+                        },
+                        "notify": {
+                            "type": "email",
+                            "smtp_host": "localhost",
+                            "smtp_port": 2525,
+                            "envelope_from": "notifications@example.com",
+                            "from_header": "{FROM}",
+                            "subject": "[SPAM] {SUBJECT}",
+                            "template": "/etc/pyquarantine/templates/notification.template",
+                            "repl_img": "/etc/pyquarantine/templates/removed.png"
+                        },
+                        "smtp_host": "localhost",
+                        "smtp_port": 2525,
+                        "milter_action": "DISCARD"
+                    }
+                }
+            ]
+        }
+    ]
+}
+```
+### Mark subject of incoming e-mails and remove the mark from outgoing e-mails
+```json
+{
+    "socket": "unix:/tmp/pyquarantine.sock",
+    "rules": [
+        {
+            "name": "inbound",
+            "conditions": {
+                "local": false
+            },
+            "actions": [
+                {
+                    "name": "add_subject_prefix",
+                    "type": "mod_header",
+                    "options": {
+                        "field": "^(Subject|Thread-Topic)$",
+                        "search": "^(?P<subject>.*)",
+                        "value": "[EXTERNAL] \\g<subject>"
+                    }
+                }
+            ]
+        }, {
+            "name": "outbound",
+            "conditions": {
+                "local": true
+            },
+            "actions": [
+                {
+                    "name": "remove_subject_prefix",
+                    "type": "mod_header",
+                    "options": {
+                        "field": "^(Subject|Thread-Topic)$",
+                        "search": "^(?P<prefix>.*)\\[EXTERNAL\\] (?P<suffix>.*)$",
+                        "value": "\\g<prefix>\\g<suffix>"
+                    }
+                }
+            ]
+        }
+    ]
+}
+```
+### Store an exact copy of all incoming e-mails in directory
+```json
+{
+    "socket": "unix:/tmp/pyquarantine.sock",
+    "rules": [
+        {
+            "name": "inbound",
+            "conditions": {
+                "local": false
+            },
+            "actions": [
+                {
+                    "name": "store_original",
+                    "type": "store",
+                    "options": {
+                        "type": "file",
+                        "directory": "/mnt/data/incoming",
+                        "original": true,
+                    }
+                }
+            ]
+        }
+    ]
+}
+```
+
 ## Developer information
 Everyone who wants to improve or extend this project is very welcome.

distribution/gentoo/mail-filter/pyquarantine-milter/pyquarantine-milter-9999.ebuild

@@ -0,0 +1,52 @@
+# Copyright 2020 Gentoo Authors
+# Distributed under the terms of the GNU General Public License v2
+
+EAPI=7
+PYTHON_COMPAT=( python3_{9..10} )
+DISTUTILS_USE_SETUPTOOLS=rdepend
+
+SCM=""
+if [ "${PV#9999}" != "${PV}" ] ; then
+	SCM="git-r3"
+	EGIT_REPO_URI="https://github.com/spacefreak86/${PN}"
+	EGIT_BRANCH="master"
+fi
+
+inherit ${SCM} distutils-r1 systemd
+
+DESCRIPTION="A pymilter based sendmail/postfix pre-queue filter."
+HOMEPAGE="https://github.com/spacefreak86/pyquarantine-milter"
+if [ "${PV#9999}" != "${PV}" ] ; then
+	SRC_URI=""
+	KEYWORDS=""
+	# Needed for tests
+	S="${WORKDIR}/${PN}"
+	EGIT_CHECKOUT_DIR="${S}"
+else
+	SRC_URI="https://github.com/spacefreak86/${PN}/archive/${PV}.tar.gz -> ${P}.tar.gz"
+	KEYWORDS="amd64 x86"
+fi
+
+LICENSE="GPL-3"
+SLOT="0"
+IUSE="+lxml systemd"
+
+RDEPEND="
+	dev-python/beautifulsoup4[${PYTHON_USEDEP}]
+	dev-python/jsonschema[${PYTHON_USEDEP}]
+	lxml? ( dev-python/lxml[${PYTHON_USEDEP}] )
+	dev-python/netaddr[${PYTHON_USEDEP}]
+	dev-python/peewee[${PYTHON_USEDEP}]
+	>=dev-python/pymilter-1.5[${PYTHON_USEDEP}]"
+
+python_install_all() {
+	distutils-r1_python_install_all
+	use systemd && systemd_dounit pyquarantine/misc/systemd/${PN}.service
+	newinitd pyquarantine/misc/openrc/${PN}.initd ${PN}
+	newconfd pyquarantine/misc/openrc/${PN}.confd ${PN}
+}
+
+pkg_postinst() {
+	elog "You will need to set up your /etc/pyquarantine/pyquarantine.conf file before"
+	elog "running pyquarantine-milter for the first time."
+}

distribution/pypi/build.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+set -e
+set -x
+PYTHON=$(which python)
+
+script_dir=$(dirname "$(readlink -f -- "$BASH_SOURCE")")
+pkg_dir=$(realpath "${script_dir}"/../..)
+
+cd "${pkg_dir}"
+${PYTHON} setup.py clean
+${PYTHON} setup.py sdist bdist_wheel

distribution/pypi/distribute.sh

@@ -0,0 +1,17 @@
+#!/bin/bash
+set -e
+
+TWINE=$(which twine)
+
+script_dir=$(dirname "$(readlink -f -- "$BASH_SOURCE")")
+pkg_dir=$(realpath "${script_dir}/../..")
+
+cd "${pkg_dir}/dist"
+ls -la
+msg="Select version to distribute (cancel with CTRL+C):"
+echo "${msg}"
+select version in $(find . -maxdepth 1 -type f -name "pyquarantine-*.*.*.tar.gz" -printf "%f\n" | sed "s#\.tar\.gz##g"); do
+  [ -n "${version}" ] && break
+  echo -e "\ninvalid choice\n\n${msg}"
+done
+${TWINE} upload "${version}"{.tar.gz,-*.whl}

docs/pyquarantine.conf.example

@@ -1,180 +0,0 @@
-# This is an example /etc/pyquarantine/pyquarantine.conf file.
-# Copy it into place before use.
-#
-# Comments: use '#' for comment lines and ';' (following a space) for inline comments.
-#
-#           If an option is not present in a quarantine section, it will be read from
-#           the global section.
-#
-
-
-[global]
-
-# Option: quarantines
-# Notes:  Set active quarantines (comma-separated).
-#         Each active quarantine must have a section with the same name below.
-#         The quarantine name 'global' is forbidden and will be ignored.
-#         Quarantine names must be unique.
-# Values: [ ACTIVE ]
-#
-quarantines = spam
-
-# Option: quarantine_action_precedence
-# Notes:  Set if the action of the first or the last matching quarantine should
-#         be used if multiple recipients match multiple quarantines. If an original
-#         email is delivered to at least one recipient due to whitelisting, the
-#         email will always be accepted.
-# Values: [ first | last ]
-#
-preferred_quarantine_action = last
-
-
-[spam]
-# Option: ignore_hosts
-# Notes:  Set a list of host and network addresses to be ignored by this quarantine.
-#         All the common host/network notations are supported, including IPv6.
-# Value:  [ HOST ]
-#
-ignore_hosts = 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
-
-# Option: regex
-# Notes:  Set the case insensitive regular expression to match against email headers.
-#         If the regex matches any of the email headers, the email
-#         will be processed by this quarantine.
-# Values: [ REGEX ]
-#
-regex = ^X-Spam-Flag: YES
-
-# Option: smtp_host
-# Notes:  Set the SMTP host. It will be used to (re-)inject emails.
-# Values: [ HOSTNAME | IP_ADDRESS ]
-#
-smtp_host = 127.0.0.1
-
-# Option: smtp_port
-# Notes:  Set the SMTP port.
-# Values: [ PORT ]
-#
-smtp_port = 25
-
-# Option: quarantine_type
-# Notes:  Set the quarantine type.
-# Values: [ file | none ]
-#
-quarantine_type = file
-
-# Option: quarantine_directory
-# Notes:  Set the directory to store quarantined emails.
-#         This option is needed by quarantine type 'file'.
-# Values: [ DIRECTORY ]
-#
-quarantine_directory = /var/lib/pyquarantine/spam
-
-# Option: action
-# Notes:  Set the milter action to perform if email is processed by this quarantine.
-# Values: [ accept | discard | reject ]
-#
-action = discard
-
-# Option: reject_reason
-# Notes:  Optionally set the reason to return if action is set to reject.
-# Values: [ REASON ]
-#
-reject_reason = Message rejected
-
-# Option: notification
-# Notes:  Set the notification type.
-# Values: [ email | none ]
-#
-notification_type = email
-
-# Option: notification_email_smtp_host
-# Notes:  Set the SMTP host. It will be used to send notification e-mails.
-# Values: [ HOSTNAME | IP_ADDRESS ]
-#
-notification_email_smtp_host = 127.0.0.1
-
-# Option: notification_email_smtp_port
-# Notes:  Set the SMTP port.
-# Values: [ PORT ]
-#
-notification_email_smtp_port = 25
-
-# Option: notification_email_envelope_from
-# Notes:  Set the envelope-from address used when sending notification emails.
-#         This option is needed by notification type 'email'.
-# Values: [ ENVELOPE_FROM_ADDRESS ]
-#
-notification_email_envelope_from = notification@domain.tld
-
-# Option: notification_email_from
-# Notes:  Set the from header used when sending notification emails.
-#         This option is needed by notification type 'email'.
-# Values: [ FROM_HEADER ]
-#
-notification_email_from = Notification <notification@domain.tld>
-
-# Option: notification_email_usbject
-# Notes:  Set the subject used when sending notification emails.
-#         This option is needed by notification type 'email'.
-# Values: [ SUBJECT ]
-#
-notification_email_subject = Spam Quarantine Notification
-
-# Option: notification_email_template
-# Notes:  Set the template used when sending notification emails.
-#         A relative path to this config file can be used.
-#         This option is needed by notification type 'email'.
-# Values: [ TEMPLATE_PATH ]
-#
-notification_email_template = templates/notification.template
-
-# Option: notification_email_strip_images
-# Notes:  Optionally enable this option to strip img tags from emails.
-# Values: [ TRUE | ON | YES | FALSE | OFF | NO ]
-#
-notification_email_strip_images = False
-
-# Option: notification_email_replacement_img
-# Notes:  Optionally set the path to a replacement image for img tags within emails.
-#         A relative path to this config file can be used.
-# Values: [ IMAGE_PATH ]
-#
-notification_email_replacement_img = templates/removed.png
-
-# Option: notification_email_embedded_imgs
-# Notes:  Set a list of paths to images to embed in e-mails (comma-separated).
-#         Relative paths to this config file can be used.
-#         This option is needed by notification type 'email'.
-# Values: [ IMAGE_PATH ]
-#
-notification_email_embedded_imgs = templates/logo.png
-
-# Option: notification_email_parser_lib
-# Notes:  Optionally set the parser library used to parse
-#         the text part of emails.
-# Values: [ lxml | html.parser ]
-#
-notification_email_parser_lib = lxml
-
-# Option: whitelist_type
-# Notes:  Set the whitelist type.
-# Values: [ db | none ]
-#
-whitelist_type = db
-
-# Option: whitelist_db_connection
-# Notes:  Set the connection string to connect to the database.
-#         The configured user must have read/write access to
-#         the whitelist_db_table configured below.
-#         This option is needed by whitelist type 'db'.
-# Values: [ DB_CONNECTION_STRING | none ]
-#
-whitelist_db_connection = mysql://user:password@localhost/database
-
-# Option: whitelist_db_table
-# Notes:  Set the database table name.
-#         This option is needed by whitelist type 'db'.
-# Values: [ DATABASE_TABLE]
-#
-whitelist_db_table = whitelist

misc/openrc/pyquarantine-milter.confd

@@ -1,17 +0,0 @@
-# /etc/conf.d/pyquarantine-milter: config file for /etc/init.d/pyquarantine-milter
-
-# Set the socket used to communicate with the MTA.
-# Examples:
-#   unix:/path/to/socket             a named pipe
-#   inet:8899                        listen on ANY interface
-#   inet:8899@localhost              listen on a specific interface
-#   inet6:8899                       listen on ANY interface
-#   inet6:8899@[2001:db8:1234::1]    listen on a specific interface
-SOCKET="inet:8899@127.0.0.1"
-
-# Start the daemon as the user. You can optionally append a group name here also.
-# USER="daemon"
-# USER="daemon:nobody"
-
-# Optional parameters for pyquarantine-milter
-# MILTER_OPTS=""

pyquarantine/__init__.py

@@ -1,574 +1,340 @@
-# PyQuarantine-Milter is free software: you can redistribute it and/or modify
+# pyquarantine is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
-# PyQuarantine-Milter is distributed in the hope that it will be useful,
+# pyquarantine is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
-# along with PyQuarantineMilter.  If not, see <http://www.gnu.org/licenses/>.
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
 #
 
 __all__ = [
-    "QuarantineMilter",
+    "action",
-    "generate_milter_config",
+    "base",
-    "reload_config",
     "cli",
+    "conditions",
+    "config",
     "mailer",
-    "notifications",
+    "modify",
-    "quarantines",
+    "notify",
+    "rule",
     "run",
-    "version",
+    "storage",
-    "whitelists"]
+    "whitelist",
+    "QuarantineMilter"]
 
-name = "pyquarantine"
+__version__ = "2.0.6"
+
+from pyquarantine import _runtime_patches
 
 import Milter
-import configparser
 import logging
-import os
-import re
-import sys
 
 from Milter.utils import parse_addr
 from collections import defaultdict
+from copy import copy
+from email import message_from_binary_file
+from email.header import Header, decode_header, make_header
+from email.headerregistry import AddressHeader, _default_header_map
+from email.policy import SMTP
 from io import BytesIO
-from itertools import groupby
+from netaddr import IPNetwork, AddrFormatError
-from netaddr import IPAddress, IPNetwork
+
-from pyquarantine import quarantines
+from pyquarantine.base import CustomLogger, MilterMessage
-from pyquarantine import notifications
+from pyquarantine.base import replace_illegal_chars
-from pyquarantine import whitelists
+from pyquarantine.rule import Rule
 
 
 class QuarantineMilter(Milter.Base):
-    """QuarantineMilter based on Milter.Base to implement milter communication
+    """QuarantineMilter based on Milter.Base to implement
+       milter communication"""
 
-    The class variable config needs to be filled with the result of the generate_milter_config function.
+    _rules = []
+    _loglevel = logging.INFO
+    _addr_fields = [f for f, v in _default_header_map.items()
+                    if issubclass(v, AddressHeader)]
 
-    """
+    @staticmethod
-    config = None
+    def set_config(cfg, debug):
-    global_config = None
+        QuarantineMilter._loglevel = cfg.get_loglevel(debug)
 
-    # list of default config files
+        try:
-    _config_files = [
+            local_addrs = []
-        "/etc/pyquarantine/pyquarantine.conf",
+            for addr in cfg["local_addrs"]:
-        os.path.expanduser('~/pyquarantine.conf'),
+                local_addrs.append(IPNetwork(addr))
-        "pyquarantine.conf"]
+        except AddrFormatError as e:
-    # list of possible actions
+            raise RuntimeError(e)
-    _actions = {
+
-        "ACCEPT": Milter.ACCEPT,
+        logger = logging.getLogger(__name__)
-        "REJECT": Milter.REJECT,
+        logger.setLevel(QuarantineMilter._loglevel)
-        "DISCARD": Milter.DISCARD}
+        for idx, rule_cfg in enumerate(cfg["rules"]):
+            rule = Rule(rule_cfg, local_addrs, debug)
+            logger.debug(rule)
+            QuarantineMilter._rules.append(rule)
 
     def __init__(self):
         self.logger = logging.getLogger(__name__)
-        # save config, it must not change during runtime
+        self.logger.setLevel(QuarantineMilter._loglevel)
-        self.global_config = QuarantineMilter.global_config
-        self.config = QuarantineMilter.config
 
-    def _get_preferred_quarantine(self):
+    def addheader(self, field, value, idx=-1):
-        matching_quarantines = [
+        value = replace_illegal_chars(Header(s=value).encode())
-            q for q in self.recipients_quarantines.values() if q]
+        self.logger.debug(f"addheader: {field}: {value}")
-        if self.global_config["preferred_quarantine_action"] == "first":
+        super().addheader(field, value, idx)
-            quarantine = sorted(
+
-                matching_quarantines,
+    def chgheader(self, field, value, idx=1):
-                key=lambda x: x["index"])[0]
+        value = replace_illegal_chars(Header(s=value).encode())
+        if value:
+            self.logger.debug(f"chgheader: {field}[{idx}]: {value}")
         else:
-            quarantine = sorted(
+            self.logger.debug(f"delheader: {field}[{idx}]")
-                matching_quarantines,
+        super().chgheader(field, idx, value)
-                key=lambda x: x["index"],
-                reverse=True)[0]
-        return quarantine
 
-    @staticmethod
+    def msg_as_bytes(self):
-    def get_configfiles():
+        try:
-        return QuarantineMilter._config_files
+            data = self.msg.as_bytes()
+        except Exception as e:
+            self.logger.warning(f"unable to serialize message as bytes: {e}")
+            try:
+                self.logger.warning("try to serialize as str and encode")
+                data = self.msg.as_string().encode(errors="replace")
+            except Exception as e:
+                self.logger.error(
+                    f"unable to serialize message, giving up: {e}")
+                raise e
 
-    @staticmethod
+        return data
-    def get_actions():
-        return QuarantineMilter._actions
 
-    @staticmethod
+    def update_headers(self, old_headers):
-    def set_configfiles(config_files):
+        if self.msg.is_multipart() and not self.msg["MIME-Version"]:
-        QuarantineMilter._config_files = config_files
+            self.msg.add_header("MIME-Version", "1.0")
+
+        # serialize the message object so it updates its internal strucure
+        self.msg_as_bytes()
+
+        headers = set(self.msg.items())
+        to_remove = list(set(old_headers) - headers)
+        to_add = list(headers - set(old_headers))
+
+        idx = defaultdict(int)
+        for field, value in old_headers:
+            field_lower = field.lower()
+            if (field, value) in to_remove:
+                self.chgheader(field, "", idx=idx[field_lower] + 1)
+                continue
+            idx[field_lower] += 1
+
+        for field, value in to_add:
+            self.addheader(field, value)
+
+    def replacebody(self):
+        self._body_changed = True
+
+    def _replacebody(self):
+        if not self._body_changed:
+            return
+        data = self.msg_as_bytes()
+        body_pos = data.find(b"\r\n\r\n") + 4
+        self.logger.debug("replace body")
+        super().replacebody(data[body_pos:])
+        del data
+
+    def delrcpt(self, rcpts):
+        "Remove recipient. May be called from eom callback only."
+        if not isinstance(rcpts, list):
+            rcpts = [rcpts]
+        for rcpt in rcpts:
+            self.logger.debug(f"delrcpt: {rcpt}")
+            self.msginfo["rcpts"].remove(rcpt)
+            super().delrcpt(rcpt)
 
     def connect(self, IPname, family, hostaddr):
+        try:
+            if hostaddr is None:
+                self.logger.error(f"received invalid host address {hostaddr}, "
+                                  f"unable to proceed")
+                return Milter.TEMPFAIL
+
+            self.IP = hostaddr[0]
+            self.port = hostaddr[1]
             self.logger.debug(
-            "accepted milter connection from {} port {}".format(
+                f"accepted milter connection from {self.IP} "
-                *hostaddr))
+                f"port {self.port}")
-        ip = IPAddress(hostaddr[0])
+
-        for quarantine in self.config.copy():
+            # pre-filter rules and actions by the host condition
-            for ignore in quarantine["ignore_hosts_list"]:
+            # also check if the mail body is needed by any upcoming action.
-                if ip in ignore:
+            self.rules = []
+            self._headersonly = True
+            for rule in QuarantineMilter._rules:
+                if rule.conditions is None or \
+                        rule.conditions.match_host(self.IP):
+                    actions = []
+                    for action in rule.actions:
+                        if action.conditions is None or \
+                                action.conditions.match_host(self.IP):
+                            actions.append(action)
+                            if not action.headersonly():
+                                self._headersonly = False
+
+                    if actions:
+                        # copy needed rules to preserve configured actions
+                        rule = copy(rule)
+                        rule.actions = actions
+                        self.rules.append(rule)
+
+            if not self.rules:
                 self.logger.debug(
-                        "host {} is ignored by quarantine {}".format(
+                    "host is ignored by all rules, skip further processing")
-                            hostaddr[0], quarantine["name"]))
-                    self.config.remove(quarantine)
-                    break
-            if not self.config:
-                self.logger.debug(
-                    "host {} is ignored by all quarantines, "
-                    "skip further processing",
-                    hostaddr[0])
                 return Milter.ACCEPT
+
+        except Exception as e:
+            self.logger.exception(
+                f"an exception occured in connect method: {e}")
+            return Milter.TEMPFAIL
+
         return Milter.CONTINUE
 
-    @Milter.noreply
+    def hello(self, heloname):
+        try:
+            self.heloname = heloname
+            self.logger.debug(f"received HELO name: {heloname}")
+        except Exception as e:
+            self.logger.exception(
+                f"an exception occured in hello method: {e}")
+            return Milter.TEMPFAIL
+
+        return Milter.CONTINUE
+
+    @Milter.decode("replace")
     def envfrom(self, mailfrom, *str):
+        try:
             self.mailfrom = "@".join(parse_addr(mailfrom)).lower()
-        self.recipients = set()
+            self.rcpts = set()
+        except Exception as e:
+            self.logger.exception(
+                f"an exception occured in envfrom method: {e}")
+            return Milter.TEMPFAIL
+
         return Milter.CONTINUE
 
-    @Milter.noreply
+    @Milter.decode("replace")
     def envrcpt(self, to, *str):
-        self.recipients.add("@".join(parse_addr(to)).lower())
+        try:
+            self.rcpts.add("@".join(parse_addr(to)).lower())
+        except Exception as e:
+            self.logger.exception(
+                f"an exception occured in envrcpt method: {e}")
+            return Milter.TEMPFAIL
+
         return Milter.CONTINUE
 
-    @Milter.noreply
     def data(self):
-        self.queueid = self.getsymval('i')
+        try:
-        self.logger.debug(
+            self.qid = self.getsymval('i')
-            "{}: received queue-id from MTA".format(self.queueid))
+            self.logger = CustomLogger(
-        self.recipients = list(self.recipients)
+                self.logger, {"qid": self.qid, "name": "milter"})
-        self.headers = []
+            self.logger.debug("received queue-id from MTA")
+            self.fp = BytesIO()
+        except Exception as e:
+            self.logger.exception(
+                f"an exception occured in data method: {e}")
+            return Milter.TEMPFAIL
+
         return Milter.CONTINUE
 
-    @Milter.noreply
+    @Milter.decode("replace")
-    def header(self, name, value):
+    def header(self, field, value):
-        self.headers.append((name, value))
+        try:
+            # remove CR and LF from address fields, otherwise pythons
+            # email library throws an exception
+            if field.lower() in QuarantineMilter._addr_fields:
+                try:
+                    v = str(make_header(decode_header(value)))
+                except Exception as e:
+                    self.logger.error(
+                        f"unable to decode field '{field}': {e}")
+                else:
+                    if any(c in v for c in ["\r", "\n"]):
+                        v = v.replace("\r", "").replace("\n", "")
+                        value = Header(s=v).encode()
+
+            # remove surrogates
+            field = field.encode("ascii", errors="replace")
+            value = value.encode("ascii", errors="replace")
+
+            self.fp.write(field.encode() + b": " + value.encode() + b"\r\n")
+        except Exception as e:
+            self.logger.exception(
+                f"an exception occured in header method: {e}")
+            return Milter.TEMPFAIL
+
         return Milter.CONTINUE
 
     def eoh(self):
         try:
-            self.whitelist_cache = whitelists.WhitelistCache()
+            self.fp.write(b"\r\n")
-
+        except Exception as e:
-            # initialize dicts to set quaranines per recipient and keep matches
+            self.logger.exception(
-            self.recipients_quarantines = {}
+                f"an exception occured in eoh method: {e}")
-            self.quarantines_matches = {}
-
-            # iterate email headers
-            recipients_to_check = self.recipients.copy()
-            for name, value in self.headers:
-                header = "{}: {}".format(name, value)
-                self.logger.debug(
-                    "{}: checking header against configured quarantines: {}".format(
-                        self.queueid, header))
-                # iterate quarantines
-                for quarantine in self.config:
-                    if len(self.recipients_quarantines) == len(
-                            self.recipients):
-                        # every recipient matched a quarantine already
-                        if quarantine["index"] >= max(
-                                [q["index"] for q in self.recipients_quarantines.values()]):
-                            # all recipients matched a quarantine with at least
-                            # the same precedence already, skip checks against
-                            # quarantines with lower precedence
-                            self.logger.debug(
-                                "{}: {}: skip further checks of this header".format(
-                                    self.queueid, quarantine["name"]))
-                            break
-
-                    # check email header against quarantine regex
-                    self.logger.debug(
-                        "{}: {}: checking header against regex '{}'".format(
-                            self.queueid, quarantine["name"], quarantine["regex"]))
-                    match = quarantine["regex_compiled"].search(header)
-                    if match:
-                        self.logger.debug(
-                            "{}: {}: header matched regex".format(
-                                self.queueid, quarantine["name"]))
-                        # check for whitelisted recipients
-                        whitelist = quarantine["whitelist_obj"]
-                        if whitelist is not None:
-                            try:
-                                whitelisted_recipients = self.whitelist_cache.get_whitelisted_recipients(
-                                    whitelist, self.mailfrom, recipients_to_check)
-                            except RuntimeError as e:
-                                self.logger.error(
-                                    "{}: {}: unable to query whitelist: {}".format(
-                                        self.queueid, quarantine["name"], e))
             return Milter.TEMPFAIL
-                        else:
-                            whitelisted_recipients = {}
-
-                        # iterate recipients
-                        for recipient in recipients_to_check.copy():
-                            if recipient in whitelisted_recipients:
-                                # recipient is whitelisted in this quarantine
-                                self.logger.debug(
-                                    "{}: {}: recipient '{}' is whitelisted".format(
-                                        self.queueid, quarantine["name"], recipient))
-                                continue
-
-                            if recipient not in self.recipients_quarantines.keys() or \
-                                    self.recipients_quarantines[recipient]["index"] > quarantine["index"]:
-                                self.logger.debug(
-                                    "{}: {}: set quarantine for recipient '{}'".format(
-                                        self.queueid, quarantine["name"], recipient))
-                                # save match for later use as template
-                                # variables
-                                self.quarantines_matches[quarantine["name"]] = match
-                                self.recipients_quarantines[recipient] = quarantine
-                                if quarantine["index"] == 0:
-                                    # we do not need to check recipients which
-                                    # matched the quarantine with the highest
-                                    # precedence already
-                                    recipients_to_check.remove(recipient)
-                            else:
-                                self.logger.debug(
-                                    "{}: {}: a quarantine with same or higher precedence "
-                                    "matched already for recipient '{}'".format(
-                                        self.queueid, quarantine["name"], recipient))
-
-                if not recipients_to_check:
-                    self.logger.debug(
-                        "{}: all recipients matched the first quarantine, "
-                        "skipping all remaining header checks".format(
-                            self.queueid))
-                    break
-
-            # check if no quarantine has matched for all recipients
-            if not self.recipients_quarantines:
-                # accept email
-                self.logger.info(
-                    "{}: passed clean for all recipients".format(
-                        self.queueid))
-                return Milter.ACCEPT
-
-            # check if the email body is needed
-            keep_body = False
-            for recipient, quarantine in self.recipients_quarantines.items():
-                if quarantine["quarantine_obj"] or quarantine["notification_obj"]:
-                    keep_body = True
-                    break
-
-            if keep_body:
-                self.logger.debug(
-                    "{}: initializing memory buffer to save email data".format(
-                        self.queueid))
-                # initialize memory buffer to save email data
-                self.fp = BytesIO()
-                # write email headers to memory buffer
-                for name, value in self.headers:
-                    self.fp.write("{}: {}\n".format(name, value).encode())
-                self.fp.write("\n".encode())
-            else:
-                # quarantine and notification are disabled on all matching
-                # quarantines, return configured action
-                quarantine = self._get_preferred_quarantine()
-                self.logger.info(
-                    "{}: {} matching quarantine is '{}', performing milter action {}".format(
-                        self.queueid,
-                        self.global_config["preferred_quarantine_action"],
-                        quarantine["name"],
-                        quarantine["action"].upper()))
-                if quarantine["action"] == "reject":
-                    self.setreply("554", "5.7.0", quarantine["reject_reason"])
-                return quarantine["milter_action"]
 
         return Milter.CONTINUE
 
-        except Exception as e:
-            self.logger.exception(
-                "an exception occured in eoh function: {}".format(e))
-            return Milter.TEMPFAIL
-
     def body(self, chunk):
         try:
-            # save received body chunk
+            if not self._headersonly:
                 self.fp.write(chunk)
         except Exception as e:
             self.logger.exception(
-                "an exception occured in body function: {}".format(e))
+                f"an exception occured in body method: {e}")
             return Milter.TEMPFAIL
+
         return Milter.CONTINUE
 
     def eom(self):
         try:
-            # processing recipients grouped by quarantines
+            # msg and msginfo contain the runtime data that
-            quarantines = []
+            # is read/modified by actions
-            for quarantine, recipients in groupby(
+            self.fp.seek(0)
-                    sorted(self.recipients_quarantines,
+            self.msg = message_from_binary_file(
-                           key=lambda x: self.recipients_quarantines[x]["index"]),
+                self.fp, _class=MilterMessage, policy=SMTP.clone(
-                    lambda x: self.recipients_quarantines[x]):
+                    refold_source='none'))
-                quarantines.append((quarantine, list(recipients)))
+            self.msginfo = {
+                "mailfrom": self.mailfrom,
+                "rcpts": [*self.rcpts],
+                "vars": {}}
 
-            # iterate quarantines sorted by index
+            self._body_changed = False
-            for quarantine, recipients in sorted(
+            milter_action = None
-                    quarantines, key=lambda x: x[0]["index"]):
+            for rule in self.rules:
-                quarantine_id = ""
+                milter_action = rule.execute(self)
-                headers = defaultdict(str)
+                self.logger.debug(
-                for name, value in self.headers:
+                    f"current template variables: {self.msginfo['vars']}")
-                    headers[name.lower()] = value
+                if milter_action is not None:
-                subgroups = self.quarantines_matches[quarantine["name"]].groups(
+                    break
-                    default="")
+                elif not self.msginfo["rcpts"]:
-                named_subgroups = self.quarantines_matches[quarantine["name"]].groupdict(
+                    milter_action = ("DISCARD", None)
-                    default="")
+                    break
 
-                # check if a quarantine is configured
+            if milter_action is None:
-                if quarantine["quarantine_obj"] is not None:
+                self._replacebody()
-                    # add email to quarantine
+            else:
-                    self.logger.info("{}: adding to quarantine '{}' for: {}".format(
+                action, reason = milter_action
-                        self.queueid, quarantine["name"], ", ".join(recipients)))
+                if action == "ACCEPT":
-                    try:
+                    self._replacebody()
-                        quarantine_id = quarantine["quarantine_obj"].add(
-                            self.queueid, self.mailfrom, recipients, headers, self.fp,
-                            subgroups, named_subgroups)
-                    except RuntimeError as e:
-                        self.logger.error(
-                            "{}: unable to add to quarantine '{}': {}".format(
-                                self.queueid, quarantine["name"], e))
-                        return Milter.TEMPFAIL
-
-                # check if a notification is configured
-                if quarantine["notification_obj"] is not None:
-                    # notify
-                    self.logger.info(
-                        "{}: sending notification for quarantine '{}' to: {}".format(
-                            self.queueid, quarantine["name"], ", ".join(recipients)))
-                    try:
-                        quarantine["notification_obj"].notify(
-                            self.queueid, quarantine_id,
-                            self.mailfrom, recipients, headers, self.fp,
-                            subgroups, named_subgroups)
-                    except RuntimeError as e:
-                        self.logger.error(
-                            "{}: unable to send notification for quarantine '{}': {}".format(
-                                self.queueid, quarantine["name"], e))
-                        return Milter.TEMPFAIL
-
-                # remove processed recipient
-                for recipient in recipients:
-                    self.delrcpt(recipient)
-                    self.recipients.remove(recipient)
-
-            self.fp.close()
-
-            # email passed clean for at least one recipient, accepting email
-            if self.recipients:
-                self.logger.info(
-                    "{}: passed clean for: {}".format(
-                        self.queueid, ", ".join(
-                            self.recipients)))
                     return Milter.ACCEPT
-
+                elif action == "REJECT":
-            # return configured action
+                    self.setreply("554", "5.7.0", reason)
-            quarantine = self._get_preferred_quarantine()
+                    return Milter.REJECT
-            self.logger.info(
+                elif action == "DISCARD":
-                "{}: {} matching quarantine is '{}', performing milter action {}".format(
+                    return Milter.DISCARD
-                    self.queueid,
-                    self.global_config["preferred_quarantine_action"],
-                    quarantine["name"],
-                    quarantine["action"].upper()))
-            if quarantine["action"] == "reject":
-                self.setreply("554", "5.7.0", quarantine["reject_reason"])
-            return quarantine["milter_action"]
 
         except Exception as e:
             self.logger.exception(
-                "an exception occured in eom function: {}".format(e))
+                f"an exception occured in eom method: {e}")
             return Milter.TEMPFAIL
 
-
+        return Milter.ACCEPT
-def generate_milter_config(configtest=False, config_files=[]):
-    "Generate the configuration for QuarantineMilter class."
-    logger = logging.getLogger(__name__)
-
-    # read config file
-    parser = configparser.ConfigParser()
-    if not config_files:
-        config_files = parser.read(QuarantineMilter.get_configfiles())
-    else:
-        config_files = parser.read(config_files)
-    if not config_files:
-        raise RuntimeError("config file not found")
-
-    QuarantineMilter.set_configfiles(config_files)
-    os.chdir(os.path.dirname(config_files[0]))
-
-    # check if mandatory config options in global section are present
-    if "global" not in parser.sections():
-        raise RuntimeError(
-            "mandatory section 'global' not present in config file")
-    for option in ["quarantines", "preferred_quarantine_action"]:
-        if not parser.has_option("global", option):
-            raise RuntimeError(
-                "mandatory option '{}' not present in config section 'global'".format(option))
-
-    # read global config section
-    global_config = dict(parser.items("global"))
-    global_config["preferred_quarantine_action"] = global_config["preferred_quarantine_action"].lower()
-    if global_config["preferred_quarantine_action"] not in ["first", "last"]:
-        raise RuntimeError(
-            "option preferred_quarantine_action has illegal value")
-
-    # read active quarantine names
-    quarantine_names = [
-        q.strip() for q in global_config["quarantines"].split(",")]
-    if len(quarantine_names) != len(set(quarantine_names)):
-        raise RuntimeError(
-            "at least one quarantine is specified multiple times in quarantines option")
-    if "global" in quarantine_names:
-        quarantine_names.remove("global")
-        logger.warning(
-            "removed illegal quarantine name 'global' from list of active quarantines")
-    if not quarantine_names:
-        raise RuntimeError("no quarantines configured")
-
-    milter_config = []
-
-    logger.debug("preparing milter configuration ...")
-    # iterate quarantine names
-    for index, quarantine_name in enumerate(quarantine_names):
-
-        # check if config section for current quarantine exists
-        if quarantine_name not in parser.sections():
-            raise RuntimeError(
-                "config section '{}' does not exist".format(quarantine_name))
-        config = dict(parser.items(quarantine_name))
-
-        # check if mandatory config options are present in config
-        for option in ["regex", "quarantine_type", "notification_type",
-                       "action", "whitelist_type", "smtp_host", "smtp_port"]:
-            if option not in config.keys() and \
-                    option in global_config.keys():
-                config[option] = global_config[option]
-            if option not in config.keys():
-                raise RuntimeError(
-                    "mandatory option '{}' not present in config section '{}' or 'global'".format(
-                        option, quarantine_name))
-
-        # check if optional config options are present in config
-        defaults = {
-            "reject_reason": "Message rejected",
-            "ignore_hosts": ""
-        }
-        for option in defaults.keys():
-            if option not in config.keys() and \
-                    option in global_config.keys():
-                config[option] = global_config[option]
-            if option not in config.keys():
-                config[option] = defaults[option]
-
-        # set quarantine name
-        config["name"] = quarantine_name
-
-        # set the index
-        config["index"] = index
-
-        # pre-compile regex
-        logger.debug(
-            "{}: compiling regex '{}'".format(
-                quarantine_name,
-                config["regex"]))
-        config["regex_compiled"] = re.compile(
-            config["regex"], re.MULTILINE + re.DOTALL + re.IGNORECASE)
-
-        # create quarantine instance
-        quarantine_type = config["quarantine_type"].lower()
-        if quarantine_type in quarantines.TYPES.keys():
-            logger.debug(
-                "{}: initializing quarantine type '{}'".format(
-                    quarantine_name,
-                    quarantine_type.upper()))
-            quarantine = quarantines.TYPES[quarantine_type](
-                global_config, config, configtest)
-        elif quarantine_type == "none":
-            logger.debug("{}: quarantine is NONE".format(quarantine_name))
-            quarantine = None
-        else:
-            raise RuntimeError(
-                "{}: unknown quarantine type '{}'".format(
-                    quarantine_name, quarantine_type))
-
-        config["quarantine_obj"] = quarantine
-
-        # create whitelist instance
-        whitelist_type = config["whitelist_type"].lower()
-        if whitelist_type in whitelists.TYPES.keys():
-            logger.debug(
-                "{}: initializing whitelist type '{}'".format(
-                    quarantine_name,
-                    whitelist_type.upper()))
-            whitelist = whitelists.TYPES[whitelist_type](
-                global_config, config, configtest)
-        elif whitelist_type == "none":
-            logger.debug("{}: whitelist is NONE".format(quarantine_name))
-            whitelist = None
-        else:
-            raise RuntimeError(
-                "{}: unknown whitelist type '{}'".format(
-                    quarantine_name, whitelist_type))
-
-        config["whitelist_obj"] = whitelist
-
-        # create notification instance
-        notification_type = config["notification_type"].lower()
-        if notification_type in notifications.TYPES.keys():
-            logger.debug(
-                "{}: initializing notification type '{}'".format(
-                    quarantine_name,
-                    notification_type.upper()))
-            notification = notifications.TYPES[notification_type](
-                global_config, config, configtest)
-        elif notification_type == "none":
-            logger.debug("{}: notification is NONE".format(quarantine_name))
-            notification = None
-        else:
-            raise RuntimeError(
-                "{}: unknown notification type '{}'".format(
-                    quarantine_name, notification_type))
-
-        config["notification_obj"] = notification
-
-        # determining milter action for this quarantine
-        action = config["action"].upper()
-        if action in QuarantineMilter.get_actions().keys():
-            logger.debug("{}: action is {}".format(quarantine_name, action))
-            config["milter_action"] = QuarantineMilter.get_actions()[action]
-        else:
-            raise RuntimeError(
-                "{}: unknown action '{}'".format(
-                    quarantine_name, action))
-
-        # create host/network whitelist
-        config["ignore_hosts_list"] = []
-        ignored = set([p.strip()
-                       for p in config["ignore_hosts"].split(",") if p])
-        for ignore in ignored:
-            if not ignore:
-                continue
-            # parse network notation
-            try:
-                net = IPNetwork(ignore)
-            except AddrFormatError as e:
-                raise RuntimeError("error parsing ignore_hosts: {}".format(e))
-            else:
-                config["ignore_hosts_list"].append(net)
-        if config["ignore_hosts_list"]:
-            logger.debug(
-                "{}: ignore hosts: {}".format(
-                    quarantine_name,
-                    ", ".join(ignored)))
-
-        milter_config.append(config)
-
-    return global_config, milter_config
-
-
-def reload_config():
-    "Reload the configuration of QuarantineMilter class."
-    logger = logging.getLogger(__name__)
-
-    try:
-        global_config, config = generate_milter_config()
-    except RuntimeError as e:
-        logger.info(e)
-        logger.info("daemon is still running with previous configuration")
-    else:
-        logger.info("reloading configuration")
-        QuarantineMilter.global_config = global_config
-        QuarantineMilter.config = config

pyquarantine/_install.py

@@ -0,0 +1,177 @@
+#!/usr/bin/env python3
+
+# pyinotifyd is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# pyinotifyd is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with pyinotifyd.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import filecmp
+import logging
+import os
+import shutil
+import sys
+
+
+SYSTEMD_PATHS = ["/lib/systemd/system", "/usr/lib/systemd/system"]
+OPENRC = "/sbin/openrc"
+
+
+def _systemd_files(pkg_dir, name):
+    for path in SYSTEMD_PATHS:
+        if os.path.isdir(path):
+            break
+
+    return [
+        (f"{pkg_dir}/misc/systemd/{name}-milter.service",
+            f"{path}/{name}-milter.service", True)]
+
+
+def _openrc_files(pkg_dir, name):
+    return [
+        (f"{pkg_dir}/misc/openrc/{name}-milter.initd", f"/etc/init.d/{name}-milter", True),
+        (f"{pkg_dir}/misc/openrc/{name}-milter.confd", f"/etc/conf.d/{name}-milter", False)]
+
+
+def _config_files(pkg_dir, name):
+    return [
+        (f"{pkg_dir}/misc/{name}.conf.default", f"/etc/{name}/{name}.conf.default", False),
+        (f"{pkg_dir}/misc/templates/removed.png", f"/etc/{name}/templates/removed.png", False),
+        (f"{pkg_dir}/misc/templates/disclaimer_html.template", f"/etc/{name}/templates/disclaimer_html.template", False),
+        (f"{pkg_dir}/misc/templates/disclaimer_text.template", f"/etc/{name}/templates/disclaimer_text.template", False),
+        (f"{pkg_dir}/misc/templates/notification.template", f"/etc/{name}/templates/notification.template", False)]
+
+
+def _install_files(files):
+    for src, dst, force in files:
+        if os.path.exists(dst):
+            if os.path.isdir(dst):
+                logging.error(
+                    " => unable to copy file, destination path is a directory")
+                continue
+            elif not force:
+                logging.info(f" => file {dst} already exists")
+                continue
+
+        try:
+            logging.info(f" => install file {dst}")
+            shutil.copy2(src, dst)
+        except Exception as e:
+            logging.error(f" => unable to install file {dst}: {e}")
+
+
+def _uninstall_files(files):
+    for src, dst, force in files:
+        if not os.path.isfile(dst):
+            continue
+
+        if not force and not filecmp.cmp(src, dst, shallow=True):
+            logging.warning(
+                f" => keep modified file {dst}, "
+                f"you have to remove it manually")
+            continue
+
+        try:
+            logging.info(f" => uninstall file {dst}")
+            os.remove(dst)
+        except Exception as e:
+            logging.error(f" => unable to uninstall file {dst}: {e}")
+
+
+def _create_dir(path):
+    if os.path.isdir(path):
+        logging.info(f" => directory {path} already exists")
+    else:
+        try:
+            logging.info(f" => create directory {path}")
+            os.mkdir(path)
+        except Exception as e:
+            logging.error(f" => unable to create directory {path}: {e}")
+            return False
+
+    return True
+
+
+def _delete_dir(path):
+    if os.path.isdir(path):
+        if not os.listdir(path):
+            try:
+                logging.info(f" => delete directory {path}")
+                os.rmdir(path)
+            except Exception as e:
+                logging.error(f" => unable to delete directory {path}: {e}")
+        else:
+            logging.warning(f" => keep non-empty directory {path}")
+
+
+def _check_root():
+    if os.getuid() != 0:
+        logging.error("you need to have root privileges, please try again")
+        return False
+
+    return True
+
+
+def _check_systemd():
+    for path in SYSTEMD_PATHS:
+        systemd = os.path.isdir(path)
+        if systemd:
+            break
+
+    if systemd:
+        logging.info("systemd detected")
+
+    return systemd
+
+
+def _check_openrc():
+    openrc = os.path.isfile(OPENRC) and os.access(OPENRC, os.X_OK)
+    if openrc:
+        logging.info("openrc detected")
+
+    return openrc
+
+
+def install(name):
+    if not _check_root():
+        sys.exit(2)
+
+    pkg_dir = os.path.dirname(__file__)
+
+    if _check_systemd():
+        _install_files(_systemd_files(pkg_dir, name))
+
+    if _check_openrc():
+        _install_files(_openrc_files(pkg_dir, name))
+
+    for d in [f"/etc/{name}", f"/etc/{name}/templates"]:
+        if not _create_dir(d):
+            logging.error(" => unable to create config dir, giving up ...")
+            sys.exit(3)
+    _install_files(_config_files(pkg_dir, name))
+
+    logging.info(f"{name} successfully installed")
+
+
+def uninstall(name):
+    if not _check_root():
+        sys.exit(2)
+
+    pkg_dir = os.path.dirname(__file__)
+
+    _uninstall_files(_systemd_files(pkg_dir, name))
+    _uninstall_files(_openrc_files(pkg_dir, name))
+    _uninstall_files(_config_files(pkg_dir, name))
+
+    _delete_dir(f"/etc/{name}/templates")
+    _delete_dir(f"/etc/{name}")
+
+    logging.info(f"{name} successfully uninstalled")

pyquarantine/_runtime_patches.py

@@ -0,0 +1,231 @@
+# pyquarantine is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# pyquarantine is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+from sys import version_info
+import encodings
+
+
+#####################################
+#    patch pythons email library    #
+#####################################
+#
+# https://bugs.python.org/issue27257
+# https://bugs.python.org/issue30988
+#
+# fix: https://github.com/python/cpython/pull/15600
+
+import email._header_value_parser
+from email._header_value_parser import TokenList, NameAddr
+from email._header_value_parser import get_display_name, get_angle_addr
+from email._header_value_parser import get_cfws, errors
+from email._header_value_parser import CFWS_LEADER, PHRASE_ENDS
+
+
+class DisplayName(email._header_value_parser.DisplayName):
+    @property
+    def display_name(self):
+        res = TokenList(self)
+        if len(res) == 0:
+            return res.value
+        if res[0].token_type == 'cfws':
+            res.pop(0)
+        else:
+            if isinstance(res[0], TokenList) and \
+                    res[0][0].token_type == 'cfws':
+                res[0] = TokenList(res[0][1:])
+        if res[-1].token_type == 'cfws':
+            res.pop()
+        else:
+            if isinstance(res[-1], TokenList) and \
+                    res[-1][-1].token_type == 'cfws':
+                res[-1] = TokenList(res[-1][:-1])
+        return res.value
+
+
+def get_name_addr(value):
+    """ name-addr = [display-name] angle-addr
+
+    """
+    name_addr = NameAddr()
+    # Both the optional display name and the angle-addr can start with cfws.
+    leader = None
+    if value[0] in CFWS_LEADER:
+        leader, value = get_cfws(value)
+        if not value:
+            raise errors.HeaderParseError(
+                "expected name-addr but found '{}'".format(leader))
+    if value[0] != '<':
+        if value[0] in PHRASE_ENDS:
+            raise errors.HeaderParseError(
+                "expected name-addr but found '{}'".format(value))
+        token, value = get_display_name(value)
+        if not value:
+            raise errors.HeaderParseError(
+                "expected name-addr but found '{}'".format(token))
+        if leader is not None:
+            if isinstance(token[0], TokenList):
+                token[0][:0] = [leader]
+            else:
+                token[:0] = [leader]
+            leader = None
+        name_addr.append(token)
+    token, value = get_angle_addr(value)
+    if leader is not None:
+        token[:0] = [leader]
+    name_addr.append(token)
+    return name_addr, value
+
+
+setattr(email._header_value_parser, "DisplayName", DisplayName)
+setattr(email._header_value_parser, "get_name_addr", get_name_addr)
+
+
+# https://bugs.python.org/issue42484
+#
+# fix: https://github.com/python/cpython/pull/24669
+
+from email._header_value_parser import DOT, ObsLocalPart, ValueTerminal, get_word
+
+
+def get_obs_local_part(value):
+    """ obs-local-part = word *("." word)
+    """
+    obs_local_part = ObsLocalPart()
+    last_non_ws_was_dot = False
+    while value and (value[0]=='\\' or value[0] not in PHRASE_ENDS):
+        if value[0] == '.':
+            if last_non_ws_was_dot:
+                obs_local_part.defects.append(errors.InvalidHeaderDefect(
+                    "invalid repeated '.'"))
+            obs_local_part.append(DOT)
+            last_non_ws_was_dot = True
+            value = value[1:]
+            continue
+        elif value[0]=='\\':
+            obs_local_part.append(ValueTerminal(value[0],
+                                                'misplaced-special'))
+            value = value[1:]
+            obs_local_part.defects.append(errors.InvalidHeaderDefect(
+                "'\\' character outside of quoted-string/ccontent"))
+            last_non_ws_was_dot = False
+            continue
+        if obs_local_part and obs_local_part[-1].token_type != 'dot':
+            obs_local_part.defects.append(errors.InvalidHeaderDefect(
+                "missing '.' between words"))
+        try:
+            token, value = get_word(value)
+            last_non_ws_was_dot = False
+        except errors.HeaderParseError:
+            if value[0] not in CFWS_LEADER:
+                raise
+            token, value = get_cfws(value)
+        obs_local_part.append(token)
+    if not obs_local_part:
+        return obs_local_part, value
+    if (obs_local_part[0].token_type == 'dot' or
+            obs_local_part[0].token_type=='cfws' and
+            obs_local_part[1].token_type=='dot'):
+        obs_local_part.defects.append(errors.InvalidHeaderDefect(
+            "Invalid leading '.' in local part"))
+    if (obs_local_part[-1].token_type == 'dot' or
+            obs_local_part[-1].token_type=='cfws' and
+            obs_local_part[-2].token_type=='dot'):
+        obs_local_part.defects.append(errors.InvalidHeaderDefect(
+            "Invalid trailing '.' in local part"))
+    if obs_local_part.defects:
+        obs_local_part.token_type = 'invalid-obs-local-part'
+    return obs_local_part, value
+
+
+setattr(email._header_value_parser, "get_obs_local_part", get_obs_local_part)
+
+
+#######################################
+#  add charset alias for windows-874  #
+#######################################
+#
+# https://bugs.python.org/issue17254
+#
+# fix: https://github.com/python/cpython/pull/10237
+
+aliases = encodings.aliases.aliases
+
+for alias in ["windows-874", "windows_874"]:
+    if alias not in aliases:
+        aliases[alias] = "cp874"
+
+setattr(encodings.aliases, "aliases", aliases)
+
+
+if version_info.major == 3 and version_info.minor < 10:
+    # https://bugs.python.org/issue30681
+    #
+    # fix: https://github.com/python/cpython/pull/22090
+
+    import email.errors
+    from email.errors import HeaderDefect
+
+
+    class InvalidDateDefect(HeaderDefect):
+        """Header has unparseable or invalid date"""
+
+
+    setattr(email.errors, "InvalidDateDefect", InvalidDateDefect)
+
+
+    import email.utils
+    from email.utils import _parsedate_tz
+    import datetime
+
+
+    def parsedate_to_datetime(data):
+        parsed_date_tz = _parsedate_tz(data)
+        if parsed_date_tz is None:
+            raise ValueError('Invalid date value or format "%s"' % str(data))
+        *dtuple, tz = parsed_date_tz
+        if tz is None:
+            return datetime.datetime(*dtuple[:6])
+        return datetime.datetime(*dtuple[:6],
+                tzinfo=datetime.timezone(datetime.timedelta(seconds=tz)))
+
+
+    setattr(email.utils, "parsedate_to_datetime", parsedate_to_datetime)
+
+
+    import email.headerregistry
+    from email import utils, _header_value_parser as parser
+
+    @classmethod
+    def parse(cls, value, kwds):
+        if not value:
+            kwds['defects'].append(errors.HeaderMissingRequiredValue())
+            kwds['datetime'] = None
+            kwds['decoded'] = ''
+            kwds['parse_tree'] = parser.TokenList()
+            return
+        if isinstance(value, str):
+            kwds['decoded'] = value
+            try:
+                value = utils.parsedate_to_datetime(value)
+            except ValueError:
+                kwds['defects'].append(errors.InvalidDateDefect('Invalid date value or format'))
+                kwds['datetime'] = None
+                kwds['parse_tree'] = parser.TokenList()
+                return
+        kwds['datetime'] = value
+        kwds['decoded'] = utils.format_datetime(kwds['datetime'])
+        kwds['parse_tree'] = cls.value_parser(kwds['decoded'])
+
+
+    setattr(email.headerregistry.DateHeader, "parse", parse)

pyquarantine/action.py

@@ -0,0 +1,64 @@
+# pyquarantine is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# pyquarantine is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+__all__ = ["Action"]
+
+from pyquarantine import modify, notify, storage
+from pyquarantine.conditions import Conditions
+
+
+class Action:
+    """Action to implement a pre-configured action to perform on e-mails."""
+    ACTION_TYPES = {
+        "add_header": modify.Modify,
+        "mod_header": modify.Modify,
+        "del_header": modify.Modify,
+        "add_disclaimer": modify.Modify,
+        "rewrite_links": modify.Modify,
+        "store": storage.Store,
+        "notify": notify.Notify,
+        "quarantine": storage.Quarantine}
+
+    def __init__(self, cfg, local_addrs, debug):
+        self.cfg = cfg
+
+        self.conditions = cfg["conditions"] if "conditions" in cfg else None
+        if self.conditions is not None:
+            self.conditions["name"] = f"{cfg['name']}: conditions"
+            self.conditions["loglevel"] = cfg["loglevel"]
+            self.conditions = Conditions(self.conditions, local_addrs, debug)
+
+        action_type = cfg["type"]
+        self.action = self.ACTION_TYPES[action_type](
+            cfg, local_addrs, debug)
+
+    def __str__(self):
+        cfg = []
+        for key in ["name", "loglevel", "pretend", "type"]:
+            value = self.cfg[key]
+            cfg.append(f"{key}={value}")
+        if self.conditions is not None:
+            cfg.append(f"conditions={self.conditions}")
+        cfg.append(f"action={self.action}")
+        return "Action(" + ", ".join(cfg) + ")"
+
+    def headersonly(self):
+        """Return the needs of this action."""
+        return self.action._headersonly
+
+    def execute(self, milter):
+        """Execute configured action."""
+        if self.conditions is None or \
+                self.conditions.match(milter):
+            return self.action.execute(milter)

pyquarantine/base.py

@@ -0,0 +1,199 @@
+# pyquarantine is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# pyquarantine is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+__all__ = [
+    "CustomLogger",
+    "MilterMessage",
+    "replace_illegal_chars"]
+
+import logging
+
+from email.message import MIMEPart
+
+
+class CustomLogger(logging.LoggerAdapter):
+    def process(self, msg, kwargs):
+        if "name" in self.extra:
+            msg = f"{self.extra['name']}: {msg}"
+
+        if "qid" in self.extra:
+            msg = f"{self.extra['qid']}: {msg}"
+
+        if self.logger.getEffectiveLevel() != logging.DEBUG:
+            msg = msg.replace("\n", "").replace("\r", "")
+
+        return msg, kwargs
+
+
+class MilterMessage(MIMEPart):
+    def replace_header(self, _name, _value, idx=None):
+        _name = _name.lower()
+        counter = 0
+        for i, (k, v) in zip(range(len(self._headers)), self._headers):
+            if k.lower() == _name:
+                counter += 1
+                if not idx or counter == idx:
+                    self._headers[i] = self.policy.header_store_parse(
+                        k, _value)
+                    break
+
+        else:
+            raise KeyError(_name)
+
+    def remove_header(self, name, idx=None):
+        name = name.lower()
+        newheaders = []
+        counter = 0
+        for k, v in self._headers:
+            if k.lower() == name:
+                counter += 1
+                if counter != idx:
+                    newheaders.append((k, v))
+            else:
+                newheaders.append((k, v))
+
+        self._headers = newheaders
+
+    def _find_body_parent(self, part, preferencelist, parent=None):
+        if part.is_attachment():
+            return
+        maintype, subtype = part.get_content_type().split("/")
+        if maintype == "text":
+            if subtype in preferencelist:
+                yield(preferencelist.index(subtype), parent)
+            return
+        if maintype != "multipart" or not self.is_multipart():
+            return
+        if subtype != "related":
+            for subpart in part.iter_parts():
+                yield from self._find_body_parent(
+                    subpart, preferencelist, part)
+            return
+        if 'related' in preferencelist:
+            yield(preferencelist.index('related'), parent)
+        candidate = None
+        start = part.get_param('start')
+        if start:
+            for subpart in part.iter_parts():
+                if subpart['content-id'] == start:
+                    candidate = subpart
+                    break
+        if candidate is None:
+            subparts = part.get_payload()
+            candidate = subparts[0] if subparts else None
+        if candidate is not None:
+            yield from self._find_body_parent(candidate, preferencelist, part)
+
+    def get_body_parent(self, preferencelist=("related", "html", "plain")):
+        best_prio = len(preferencelist)
+        body_parent = None
+        for prio, parent in self._find_body_parent(self, preferencelist):
+            if prio < best_prio:
+                best_prio = prio
+                body_parent = parent
+                if prio == 0:
+                    break
+        return body_parent
+
+    def get_body_content(self, pref):
+        part = None
+        content = None
+        if not self.is_multipart() and \
+                self.get_content_type() == f"text/{pref}":
+            part = self
+        else:
+            part = self.get_body(preferencelist=(pref))
+
+        if part is not None:
+            content = part.get_content()
+
+        return (part, content)
+
+    def set_body(self, text_content=None, html_content=None):
+        parent = self.get_body_parent() or self
+        if "Content-Type" not in parent:
+            # set Content-Type header if not present, otherwise
+            # make_alternative and make_mixed skip the payload
+            parent["Content-Type"] = parent.get_content_type()
+
+        maintype, subtype = parent.get_content_type().split("/")
+        if not parent.is_multipart() or maintype != "multipart":
+            if maintype == "text" and subtype in ("html", "plain"):
+                parent.make_alternative()
+                maintype, subtype = ("multipart", "alternative")
+            else:
+                parent.make_mixed()
+                maintype, subtype = ("multipart", "mixed")
+
+        text_body = parent.get_body(preferencelist=("plain"))
+        html_body = parent.get_body(preferencelist=("html"))
+
+        if text_content is not None:
+            if text_body:
+                text_body.set_content(text_content)
+            else:
+                if not html_body or subtype == "alternative":
+                    inject_body_part(parent, text_content)
+                else:
+                    html_body.add_alternative(text_content)
+                text_body = parent.get_body(preferencelist=("plain"))
+
+        if html_content is not None:
+            if html_body:
+                html_body.set_content(html_content, subtype="html")
+            else:
+                if not text_body or subtype == "alternative":
+                    inject_body_part(parent, html_content, subtype="html")
+                else:
+                    text_body.add_alternative(html_content, subtype="html")
+
+
+def inject_body_part(part, content, subtype="plain"):
+    parts = []
+    text_body = None
+    text_content = None
+    if subtype == "html":
+        text_body, text_content = part.get_body_content("plain")
+
+    for p in part.iter_parts():
+        if text_body and p == text_body:
+            continue
+        parts.append(p)
+
+    boundary = part.get_boundary()
+    p_subtype = part.get_content_subtype()
+    part.clear_content()
+    if text_content != None:
+        part.set_content(text_content)
+        part.add_alternative(content, subtype=subtype)
+    else:
+        part.set_content(content, subtype=subtype)
+
+    if part.get_content_subtype() != p_subtype:
+        if p_subtype == "alternative":
+            part.make_alternative()
+        elif p_subtype == "related":
+            part.make_related()
+        else:
+            part.make_mixed()
+
+    if boundary:
+        part.set_boundary(boundary)
+    for p in parts:
+        part.attach(p)
+
+
+def replace_illegal_chars(string):
+    """Remove illegal characters from header values."""
+    return "".join(string.replace("\x00", "").splitlines())

pyquarantine/cli.py

@@ -15,31 +15,39 @@
 #
 
 import argparse
+import json
 import logging
 import logging.handlers
 import sys
 import time
 
-import pyquarantine
+from pyquarantine.config import get_milter_config, ActionConfig
+from pyquarantine.storage import Quarantine
+from pyquarantine import __version__ as version
 
-from pyquarantine.version import __version__ as version
 
-def _get_quarantine_obj(config, quarantine):
+def _get_quarantine(quarantines, name, debug):
     try:
-        quarantine_obj = next((q["quarantine_obj"]
+        quarantine = next((q for q in quarantines if q["name"] == name))
-                               for q in config if q["name"] == quarantine))
     except StopIteration:
-        raise RuntimeError("invalid quarantine '{}'".format(quarantine))
+        raise RuntimeError(f"invalid quarantine '{name}'")
-    return quarantine_obj
+    return Quarantine(ActionConfig(quarantine), [], debug)
 
 
-def _get_whitelist_obj(config, quarantine):
+def _get_notification(quarantines, name, debug):
-    try:
+    notification = _get_quarantine(quarantines, name, debug).notification
-        whitelist_obj = next((q["whitelist_obj"]
+    if not notification:
-                              for q in config if q["name"] == quarantine))
+        raise RuntimeError(
-    except StopIteration:
+                "notification type is set to NONE")
-        raise RuntimeError("invalid quarantine '{}'".format(quarantine))
+    return notification
-    return whitelist_obj
+
+
+def _get_whitelist(quarantines, name, debug):
+    whitelist = _get_quarantine(quarantines, name, debug).whitelist
+    if not whitelist:
+        raise RuntimeError(
+                "whitelist type is set to NONE")
+    return whitelist
 
 
 def print_table(columns, rows):
@@ -56,10 +64,10 @@ def print_table(columns, rows):
         # get the length of the longest value
         lengths.append(
             len(str(max(rows, key=lambda x: len(str(x[key])))[key])))
-        # use the the longer one
+        # use the longer one
         length = max(lengths)
         column_lengths.append(length)
-        column_formats.append("{{:<{}}}".format(length))
+        column_formats.append(f"{{:<{length}}}")
 
     # define row format
     row_format = " | ".join(column_formats)
@@ -83,51 +91,72 @@ def print_table(columns, rows):
         print(row_format.format(*row))
 
 
-def list_quarantines(config, args):
+def list_quarantines(quarantines, args):
     if args.batch:
-        print("\n".join([quarantine["name"] for quarantine in config]))
+        print("\n".join([q["name"] for q in quarantines]))
     else:
+        qlist = []
+        for q in quarantines:
+            cfg = q["options"]
+            storage_type = cfg["store"]["type"]
+
+            if "notify" in cfg:
+                notification_type = cfg["notify"]["type"]
+            else:
+                notification_type = "NONE"
+
+            if "whitelist" in cfg:
+                whitelist_type = cfg["whitelist"]["type"]
+            else:
+                whitelist_type = "NONE"
+
+            if "milter_action" in cfg:
+                milter_action = cfg["milter_action"]
+            else:
+                milter_action = "NONE"
+
+            qlist.append({
+                "name": q["name"],
+                "storage": storage_type,
+                "notification": notification_type,
+                "whitelist": whitelist_type,
+                "action": milter_action})
+
         print_table(
-            [("Name", "name"), ("Quarantine", "quarantine_type"),
+            [("Name", "name"),
-             ("Notification", "notification_type"), ("Action", "action")],
+             ("Storage", "storage"),
-            config
+             ("Notification", "notification"),
+             ("Whitelist", "whitelist"),
+             ("Action", "action")],
+            qlist
         )
 
 
-def list_quarantine_emails(config, args):
+def list_quarantine_emails(quarantines, args):
-    logger = logging.getLogger(__name__)
+    storage = _get_quarantine(quarantines, args.quarantine, args.debug).storage
-
-    # get quarantine object
-    quarantine = _get_quarantine_obj(config, args.quarantine)
-    if quarantine is None:
-        raise RuntimeError(
-            "quarantine type is set to None, unable to list emails")
 
     # find emails and transform some metadata values to strings
     rows = []
-    emails = quarantine.find(
+    emails = storage.find(
-        mailfrom=args.mailfrom,
+        args.mailfrom, args.recipients, args.older_than)
-        recipients=args.recipients,
+    for storage_id, metadata in emails.items():
-        older_than=args.older_than)
+        row = emails[storage_id]
-    for quarantine_id, metadata in emails.items():
+        row["storage_id"] = storage_id
-        row = emails[quarantine_id]
+        row["timestamp"] = time.strftime(
-        row["quarantine_id"] = quarantine_id
-        row["date"] = time.strftime(
             '%Y-%m-%d %H:%M:%S',
             time.localtime(
-                metadata["date"]))
+                metadata["timestamp"]))
         row["mailfrom"] = metadata["mailfrom"]
         row["recipient"] = metadata["recipients"].pop(0)
-        if "subject" in emails[quarantine_id]["headers"].keys():
+        if "subject" not in emails[storage_id]:
-            row["subject"] = emails[quarantine_id]["headers"]["subject"][:60]
+            emails[storage_id]["subject"] = ""
-        else
+        row["subject"] = emails[storage_id]["subject"][:60].strip()
-            row["subject"] = ""
         rows.append(row)
 
         if metadata["recipients"]:
             row = {
-                "quarantine_id": "",
+                "storage_id": "",
-                "date": "",
+                "timestamp": "",
                 "mailfrom": "",
                 "recipient": metadata["recipients"].pop(0),
                 "subject": ""
@@ -140,23 +169,19 @@ def list_quarantine_emails(config, args):
         return
 
     if not emails:
-        logger.info("quarantine '{}' is empty".format(args.quarantine))
+        print(f"quarantine '{args.quarantine}' is empty")
+        return
+
     print_table(
-        [("Quarantine-ID", "quarantine_id"), ("Date", "date"),
+        [("Quarantine-ID", "storage_id"), ("When", "timestamp"),
          ("From", "mailfrom"), ("Recipient(s)", "recipient"),
          ("Subject", "subject")],
         rows
     )
 
 
-def list_whitelist(config, args):
+def list_whitelist(quarantines, args):
-    logger = logging.getLogger(__name__)
+    whitelist = _get_whitelist(quarantines, args.quarantine, args.debug)
-
-    # get whitelist object
-    whitelist = _get_whitelist_obj(config, args.quarantine)
-    if whitelist is None:
-        raise RuntimeError(
-            "whitelist type is set to None, unable to list entries")
 
     # find whitelist entries
     entries = whitelist.find(
@@ -164,17 +189,15 @@ def list_whitelist(config, args):
         recipients=args.recipients,
         older_than=args.older_than)
     if not entries:
-        logger.info(
+        print(f"whitelist of quarantine '{args.quarantine}' is empty")
-            "whitelist of quarantine '{}' is empty".format(
-                args.quarantine))
         return
 
     # transform some values to strings
-    for entry_id, entry in entries.items():
+    for eid, entry in entries.items():
-        entries[entry_id]["permanent_str"] = str(entry["permanent"])
+        entries[eid]["permanent_str"] = str(entry["permanent"])
-        entries[entry_id]["created_str"] = entry["created"].strftime(
+        entries[eid]["created_str"] = entry["created"].strftime(
             '%Y-%m-%d %H:%M:%S')
-        entries[entry_id]["last_used_str"] = entry["last_used"].strftime(
+        entries[eid]["last_used_str"] = entry["last_used"].strftime(
             '%Y-%m-%d %H:%M:%S')
 
     print_table(
@@ -187,17 +210,12 @@ def list_whitelist(config, args):
     )
 
 
-def add_whitelist_entry(config, args):
+def add_whitelist_entry(quarantines, args):
     logger = logging.getLogger(__name__)
-
+    whitelist = _get_whitelist(quarantines, args.quarantine, args.debug)
-    # get whitelist object
-    whitelist = _get_whitelist_obj(config, args.quarantine)
-    if whitelist is None:
-        raise RuntimeError(
-            "whitelist type is set to None, unable to add entries")
 
     # check existing entries
-    entries = whitelist.check(args.mailfrom, args.recipient)
+    entries = whitelist.check(args.mailfrom, args.recipient, logger)
     if entries:
         # check if the exact entry exists already
         for entry in entries.values():
@@ -208,11 +226,11 @@ def add_whitelist_entry(config, args):
 
         if not args.force:
             # the entry is already covered by others
-            for entry_id, entry in entries.items():
+            for eid, entry in entries.items():
-                entries[entry_id]["permanent_str"] = str(entry["permanent"])
+                entries[eid]["permanent_str"] = str(entry["permanent"])
-                entries[entry_id]["created_str"] = entry["created"].strftime(
+                entries[eid]["created_str"] = entry["created"].strftime(
                     '%Y-%m-%d %H:%M:%S')
-                entries[entry_id]["last_used_str"] = entry["last_used"].strftime(
+                entries[eid]["last_used_str"] = entry["last_used"].strftime(
                     '%Y-%m-%d %H:%M:%S')
             print_table(
                 [
@@ -229,54 +247,47 @@ def add_whitelist_entry(config, args):
 
     # add entry to whitelist
     whitelist.add(args.mailfrom, args.recipient, args.comment, args.permanent)
-    logger.info("whitelist entry added successfully")
+    print("whitelist entry added successfully")
 
 
-def delete_whitelist_entry(config, args):
+def delete_whitelist_entry(quarantines, args):
-    logger = logging.getLogger(__name__)
+    whitelist = _get_whitelist(quarantines, args.quarantine, args.debug)
-
-    whitelist = _get_whitelist_obj(config, args.quarantine)
-    if whitelist is None:
-        raise RuntimeError(
-            "whitelist type is set to None, unable to delete entries")
-
     whitelist.delete(args.whitelist_id)
-    logger.info("whitelist entry deleted successfully")
+    print("whitelist entry deleted successfully")
 
 
-def notify_email(config, args):
+def notify(quarantines, args):
-    logger = logging.getLogger(__name__)
+    quarantine = _get_quarantine(quarantines, args.quarantine, args.debug)
-
-    quarantine = _get_quarantine_obj(config, args.quarantine)
-    if quarantine is None:
-        raise RuntimeError(
-            "quarantine type is set to None, unable to send notification")
     quarantine.notify(args.quarantine_id, args.recipient)
-    logger.info("sent notification successfully")
+    print("notification sent successfully")
 
 
-def release_email(config, args):
+def release(quarantines, args):
     logger = logging.getLogger(__name__)
-
+    quarantine = _get_quarantine(quarantines, args.quarantine, args.debug)
-    quarantine = _get_quarantine_obj(config, args.quarantine)
+    rcpts = quarantine.release(args.quarantine_id, args.recipient)
-    if quarantine is None:
+    rcpts = ", ".join(rcpts)
-        raise RuntimeError(
+    logger.info(
-            "quarantine type is set to None, unable to release email")
+        f"{args.quarantine}: released message with id {args.quarantine_id} "
-
+        f"for {rcpts}")
-    quarantine.release(args.quarantine_id, args.recipient)
-    logger.info("quarantined email released successfully")
 
 
-def delete_email(config, args):
+def delete(quarantines, args):
-    logger = logging.getLogger(__name__)
+    storage = _get_quarantine(quarantines, args.quarantine, args.debug).storage
+    storage.delete(args.quarantine_id, args.recipient)
+    print("quarantined message deleted successfully")
 
-    quarantine = _get_quarantine_obj(config, args.quarantine)
-    if quarantine is None:
-        raise RuntimeError(
-            "quarantine type is set to None, unable to delete email")
 
-    quarantine.delete(args.quarantine_id, args.recipient)
+def get(quarantines, args):
-    logger.info("quarantined email deleted successfully")
+    storage = _get_quarantine(quarantines, args.quarantine, args.debug).storage
+    data = storage.get_mail_bytes(args.quarantine_id)
+    sys.stdout.buffer.write(data)
+
+
+def metadata(quarantines, args):
+    storage = _get_quarantine(quarantines, args.quarantine, args.debug).storage
+    metadata = storage.get_metadata(args.quarantine_id)
+    print(json.dumps(metadata))
 
 
 class StdErrFilter(logging.Filter):
@@ -290,6 +301,9 @@ class StdOutFilter(logging.Filter):
 
 
 def main():
+    python_version = ".".join([str(v) for v in sys.version_info[0:3]])
+    python_version = f"{python_version}-{sys.version_info[3]}"
+
     "PyQuarantine command-line interface."
     # parse command line
     def formatter_class(prog): return argparse.HelpFormatter(
@@ -298,10 +312,8 @@ def main():
         description="PyQuarantine CLI",
         formatter_class=formatter_class)
     parser.add_argument(
-        "-c", "--config",
+        "-c", "--config", help="Config file to read.",
-        help="Config files to read.",
+        default="/etc/pyquarantine/pyquarantine.conf")
-        nargs="+", metavar="CFG",
-        default=pyquarantine.QuarantineMilter.get_configfiles())
     parser.add_argument(
         "-d", "--debug",
         help="Log debugging messages.",
@@ -310,7 +322,7 @@ def main():
         "-v", "--version",
         help="Print version.",
         action="version",
-        version="%(prog)s ({})".format(version))
+        version=f"%(prog)s {version} (python {python_version})")
     parser.set_defaults(syslog=False)
     subparsers = parser.add_subparsers(
         dest="command",
@@ -329,122 +341,145 @@ def main():
     list_parser.set_defaults(func=list_quarantines)
 
     # quarantine command group
-    quarantine_parser = subparsers.add_parser(
+    quar_parser = subparsers.add_parser(
         "quarantine",
         description="Manage quarantines.",
         help="Manage quarantines.",
         formatter_class=formatter_class)
-    quarantine_parser.add_argument(
+    quar_parser.add_argument(
         "quarantine",
         metavar="QUARANTINE",
         help="Quarantine name.")
-    quarantine_subparsers = quarantine_parser.add_subparsers(
+    quar_subparsers = quar_parser.add_subparsers(
         dest="command",
         title="Quarantine commands")
-    quarantine_subparsers.required = True
+    quar_subparsers.required = True
     # quarantine list command
-    quarantine_list_parser = quarantine_subparsers.add_parser(
+    quar_list_parser = quar_subparsers.add_parser(
         "list",
         description="List emails in quarantines.",
         help="List emails in quarantine.",
         formatter_class=formatter_class)
-    quarantine_list_parser.add_argument(
+    quar_list_parser.add_argument(
         "-f", "--from",
         dest="mailfrom",
         help="Filter emails by from address.",
         default=None,
         nargs="+")
-    quarantine_list_parser.add_argument(
+    quar_list_parser.add_argument(
         "-t", "--to",
         dest="recipients",
         help="Filter emails by recipient address.",
         default=None,
         nargs="+")
-    quarantine_list_parser.add_argument(
+    quar_list_parser.add_argument(
         "-o", "--older-than",
         dest="older_than",
         help="Filter emails by age (days).",
         default=None,
         type=float)
-    quarantine_list_parser.add_argument(
+    quar_list_parser.add_argument(
         "-b", "--batch",
-        help="Print results using only email quarantine IDs, each on a new line.",
+        help="Print results using only email quarantine IDs, "
+             "each on a new line.",
         action="store_true")
-    quarantine_list_parser.set_defaults(func=list_quarantine_emails)
+    quar_list_parser.set_defaults(func=list_quarantine_emails)
     # quarantine notify command
-    quarantine_notify_parser = quarantine_subparsers.add_parser(
+    quar_notify_parser = quar_subparsers.add_parser(
         "notify",
         description="Notify recipient about email in quarantine.",
         help="Notify recipient about email in quarantine.",
         formatter_class=formatter_class)
-    quarantine_notify_parser.add_argument(
+    quar_notify_parser.add_argument(
         "quarantine_id",
         metavar="ID",
         help="Quarantine ID.")
-    quarantine_notify_parser_group = quarantine_notify_parser.add_mutually_exclusive_group(
+    quar_notify_parser_grp = quar_notify_parser.add_mutually_exclusive_group(
         required=True)
-    quarantine_notify_parser_group.add_argument(
+    quar_notify_parser_grp.add_argument(
         "-t", "--to",
         dest="recipient",
         help="Release email for one recipient address.")
-    quarantine_notify_parser_group.add_argument(
+    quar_notify_parser_grp.add_argument(
         "-a", "--all",
         help="Release email for all recipients.",
         action="store_true")
-    quarantine_notify_parser.set_defaults(func=notify_email)
+    quar_notify_parser.set_defaults(func=notify)
     # quarantine release command
-    quarantine_release_parser = quarantine_subparsers.add_parser(
+    quar_release_parser = quar_subparsers.add_parser(
         "release",
         description="Release email from quarantine.",
         help="Release email from quarantine.",
         formatter_class=formatter_class)
-    quarantine_release_parser.add_argument(
+    quar_release_parser.add_argument(
         "quarantine_id",
         metavar="ID",
         help="Quarantine ID.")
-    quarantine_release_parser.add_argument(
+    quar_release_parser.add_argument(
         "-n",
         "--disable-syslog",
         dest="syslog",
         help="Disable syslog messages.",
         action="store_false")
-    quarantine_release_parser_group = quarantine_release_parser.add_mutually_exclusive_group(
+    quar_release_parser_grp = quar_release_parser.add_mutually_exclusive_group(
         required=True)
-    quarantine_release_parser_group.add_argument(
+    quar_release_parser_grp.add_argument(
         "-t", "--to",
         dest="recipient",
         help="Release email for one recipient address.")
-    quarantine_release_parser_group.add_argument(
+    quar_release_parser_grp.add_argument(
         "-a", "--all",
         help="Release email for all recipients.",
         action="store_true")
-    quarantine_release_parser.set_defaults(func=release_email)
+    quar_release_parser.set_defaults(func=release)
     # quarantine delete command
-    quarantine_delete_parser = quarantine_subparsers.add_parser(
+    quar_delete_parser = quar_subparsers.add_parser(
         "delete",
         description="Delete email from quarantine.",
         help="Delete email from quarantine.",
         formatter_class=formatter_class)
-    quarantine_delete_parser.add_argument(
+    quar_delete_parser.add_argument(
         "quarantine_id",
         metavar="ID",
         help="Quarantine ID.")
-    quarantine_delete_parser.add_argument(
+    quar_delete_parser.add_argument(
         "-n", "--disable-syslog",
         dest="syslog",
         help="Disable syslog messages.",
         action="store_false")
-    quarantine_delete_parser_group = quarantine_delete_parser.add_mutually_exclusive_group(
+    quar_delete_parser_grp = quar_delete_parser.add_mutually_exclusive_group(
         required=True)
-    quarantine_delete_parser_group.add_argument(
+    quar_delete_parser_grp.add_argument(
         "-t", "--to",
         dest="recipient",
         help="Delete email for one recipient address.")
-    quarantine_delete_parser_group.add_argument(
+    quar_delete_parser_grp.add_argument(
         "-a", "--all",
         help="Delete email for all recipients.",
         action="store_true")
-    quarantine_delete_parser.set_defaults(func=delete_email)
+    quar_delete_parser.set_defaults(func=delete)
+    # quarantine get command
+    quar_get_parser = quar_subparsers.add_parser(
+        "get",
+        description="Get email from quarantine.",
+        help="Get email from quarantine",
+        formatter_class=formatter_class)
+    quar_get_parser.add_argument(
+        "quarantine_id",
+        metavar="ID",
+        help="Quarantine ID.")
+    quar_get_parser.set_defaults(func=get)
+    # quarantine metadata command
+    quar_metadata_parser = quar_subparsers.add_parser(
+        "metadata",
+        description="Get metadata of email from quarantine.",
+        help="Get metadata of email from quarantine",
+        formatter_class=formatter_class)
+    quar_metadata_parser.add_argument(
+        "quarantine_id",
+        metavar="ID",
+        help="Quarantine ID.")
+    quar_metadata_parser.set_defaults(func=metadata)
 
     # whitelist command group
     whitelist_parser = subparsers.add_parser(
@@ -511,7 +546,8 @@ def main():
         action="store_true")
     whitelist_add_parser.add_argument(
         "--force",
-        help="Force adding an entry, even if already covered by another entry.",
+        help="Force adding an entry, "
+             "even if already covered by another entry.",
         action="store_true")
     whitelist_add_parser.set_defaults(func=add_whitelist_entry)
     # whitelist delete command
@@ -553,14 +589,26 @@ def main():
     root_logger.addHandler(stderrhandler)
     logger = logging.getLogger(__name__)
 
-    # try to generate milter configs
     try:
-        global_config, config = pyquarantine.generate_milter_config(
+        logger.debug("read milter configuration")
-            config_files=args.config, configtest=True)
+        cfg = get_milter_config(args.config, raw=True)
-    except RuntimeError as e:
+        if "rules" not in cfg or not cfg["rules"]:
-        logger.error(e)
+            raise RuntimeError("no rules configured")
+
+        for rule in cfg["rules"]:
+            if "actions" not in rule or not rule["actions"]:
+                raise RuntimeError(
+                    f"{rule['name']}: no actions configured")
+    except (RuntimeError, AssertionError) as e:
+        logger.error(f"config error: {e}")
         sys.exit(255)
 
+    quarantines = []
+    for rule in cfg["rules"]:
+        for action in rule["actions"]:
+            if action["type"] == "quarantine":
+                quarantines.append(action)
+
     if args.syslog:
         # setup syslog
         sysloghandler = logging.handlers.SysLogHandler(
@@ -577,7 +625,7 @@ def main():
 
     # call the commands function
     try:
-        args.func(config, args)
+        args.func(quarantines, args)
     except RuntimeError as e:
         logger.error(e)
         sys.exit(1)

pyquarantine/conditions.py

@@ -0,0 +1,203 @@
+# pyquarantine is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# pyquarantine is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+__all__ = ["Conditions"]
+
+import logging
+import re
+
+from netaddr import IPAddress, IPNetwork, AddrFormatError
+from pyquarantine import CustomLogger
+from pyquarantine.whitelist import DatabaseWhitelist
+
+
+class Conditions:
+    """Conditions to implement conditions for rules and actions."""
+
+    def __init__(self, cfg, local_addrs, debug):
+        self.cfg = cfg
+        self.local_addrs = local_addrs
+
+        self.logger = logging.getLogger(cfg["name"])
+        self.logger.setLevel(cfg.get_loglevel(debug))
+
+        for arg in ("local", "hosts", "envfrom", "envto", "headers", "metavar",
+                    "var"):
+            if arg not in cfg:
+                setattr(self, arg, None)
+                continue
+
+            if arg == "hosts":
+                try:
+                    self.hosts = []
+                    for host in cfg["hosts"]:
+                        self.hosts.append(IPNetwork(host))
+                except AddrFormatError as e:
+                    raise RuntimeError(e)
+            elif arg in ("envfrom", "envto"):
+                try:
+                    setattr(self, arg, re.compile(
+                        cfg[arg], re.IGNORECASE))
+                except re.error as e:
+                    raise RuntimeError(e)
+            elif arg == "headers":
+                try:
+                    self.headers = []
+                    for header in cfg["headers"]:
+                        self.headers.append(re.compile(
+                            header, re.IGNORECASE + re.DOTALL + re.MULTILINE))
+                except re.error as e:
+                    raise RuntimeError(e)
+            else:
+                setattr(self, arg, cfg[arg])
+
+        self.whitelist = cfg["whitelist"] if "whitelist" in cfg else None
+        if self.whitelist is not None:
+            self.whitelist["name"] = f"{cfg['name']}: whitelist"
+            self.whitelist["loglevel"] = cfg["loglevel"]
+            if self.whitelist["type"] == "db":
+                self.whitelist = DatabaseWhitelist(self.whitelist, debug)
+            else:
+                raise RuntimeError("invalid whitelist type")
+
+    def __str__(self):
+        cfg = []
+        for arg in ("local", "hosts", "envfrom", "envto", "headers",
+                    "var", "metavar"):
+            if arg in self.cfg:
+                cfg.append(f"{arg}={self.cfg[arg]}")
+        if self.whitelist is not None:
+            cfg.append(f"whitelist={self.whitelist}")
+        return "Conditions(" + ", ".join(cfg) + ")"
+
+    def get_whitelist(self):
+        return self.whitelist
+
+    def match_host(self, host):
+        logger = CustomLogger(
+            self.logger, {"name": self.cfg["name"]})
+        ip = IPAddress(host)
+
+        if self.local is not None:
+            is_local = False
+            for addr in self.local_addrs:
+                if ip in addr:
+                    is_local = True
+                    break
+
+            if is_local != self.local:
+                logger.debug(
+                    f"ignore host {host}, "
+                    f"local does not match")
+                return False
+
+            logger.debug(
+                f"local matches for host {host}")
+
+        if self.hosts is not None:
+            found = False
+            for addr in self.hosts:
+                if ip in addr:
+                    found = True
+                    break
+
+            if not found:
+                logger.debug(
+                    f"ignore host {host}, "
+                    f"hosts does not match")
+                return False
+
+            logger.debug(
+                f"hosts matches for host {host}")
+
+        return True
+
+    def get_wl_rcpts(self, mailfrom, rcpts, logger):
+        if not self.whitelist:
+            return {}
+
+        wl_rcpts = []
+        for rcpt in rcpts:
+            if self.whitelist.check(mailfrom, rcpt, logger):
+                wl_rcpts.append(rcpt)
+
+        return wl_rcpts
+
+    def update_msginfo_from_match(self, milter, match):
+        if self.metavar is None:
+            return
+
+        named_subgroups = match.groupdict(default=None)
+        for group, value in named_subgroups.items():
+            if value is None:
+                continue
+            name = f"{self.metavar}_{group}"
+            milter.msginfo["vars"][name] = value
+
+    def match(self, milter):
+        logger = CustomLogger(
+            self.logger, {"qid": milter.qid, "name": self.cfg["name"]})
+
+        if self.envfrom is not None:
+            envfrom = milter.msginfo["mailfrom"]
+            if match := self.envfrom.match(envfrom):
+                logger.debug(
+                    f"envfrom matches for "
+                    f"envelope-from address {envfrom}")
+                self.update_msginfo_from_match(milter, match)
+            else:
+                logger.debug(
+                    f"ignore envelope-from address {envfrom}, "
+                    f"envfrom does not match")
+                return False
+
+        if self.envto is not None:
+            envto = milter.msginfo["rcpts"]
+            if not isinstance(envto, list):
+                envto = [envto]
+
+            for to in envto:
+                match = self.envto.match(to)
+                if not match:
+                    logger.debug(
+                        f"ignore envelope-to address {envto}, "
+                        f"envto does not match")
+                    return False
+
+            logger.debug(
+                f"envto matches for "
+                f"envelope-to address {envto}")
+            self.update_msginfo_from_match(milter, match)
+
+        if self.headers is not None:
+            headers = map(lambda h: f"{h[0]}: {h[1]}", milter.msg.items())
+            for hdr in self.headers:
+                matches = filter(None, map(lambda h: hdr.search(h), headers))
+                if match := next(matches, None):
+                    logger.debug(
+                        f"headers matches for "
+                        f"header: {match.string}")
+                    self.update_msginfo_from_match(milter, match)
+                    continue
+
+                logger.debug(
+                    "ignore message, "
+                    "headers does not match")
+                return False
+
+        if self.var is not None:
+            if self.var not in milter.msginfo["vars"]:
+                return False
+
+        return True

pyquarantine/config.py

@@ -0,0 +1,376 @@
+# pyquarantine is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# pyquarantine is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+__all__ = [
+    "BaseConfig",
+    "ConditionsConfig",
+    "AddHeaderConfig",
+    "ModHeaderConfig",
+    "DelHeaderConfig",
+    "AddDisclaimerConfig",
+    "RewriteLinksConfig",
+    "StoreConfig",
+    "NotifyConfig",
+    "WhitelistConfig",
+    "QuarantineConfig",
+    "ActionConfig",
+    "RuleConfig",
+    "QuarantineMilterConfig",
+    "get_milter_config"]
+
+import json
+import jsonschema
+import logging
+import re
+
+
+class BaseConfig:
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": [],
+        "additionalProperties": True,
+        "properties": {
+            "loglevel": {"type": "string", "default": "info"}}}
+
+    def __init__(self, config):
+        required = self.JSON_SCHEMA["required"]
+        properties = self.JSON_SCHEMA["properties"]
+        for p in properties.keys():
+            if p in required:
+                continue
+            elif p not in config and "default" in properties[p]:
+                config[p] = properties[p]["default"]
+        try:
+            jsonschema.validate(config, self.JSON_SCHEMA)
+        except jsonschema.exceptions.ValidationError as e:
+            raise RuntimeError(e)
+
+        self._config = config
+
+    def __getitem__(self, key):
+        return self._config[key]
+
+    def __setitem__(self, key, value):
+        self._config[key] = value
+
+    def __delitem__(self, key):
+        del self._config[key]
+
+    def __contains__(self, key):
+        return key in self._config
+
+    def keys(self):
+        return self._config.keys()
+
+    def items(self):
+        return self._config.items()
+
+    def get_loglevel(self, debug):
+        if debug:
+            level = logging.DEBUG
+        else:
+            level = getattr(logging, self["loglevel"].upper(), None)
+            assert isinstance(level, int), \
+                "loglevel: invalid value"
+        return level
+
+    def get_config(self):
+        return self._config
+
+
+class WhitelistConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["type"],
+        "additionalProperties": True,
+        "properties": {
+            "type": {"enum": ["db"]}},
+        "if": {"properties": {"type": {"const": "db"}}},
+        "then": {
+            "required": ["connection", "table"],
+            "additionalProperties": False,
+            "properties": {
+                "type": {"type": "string"},
+                "connection": {"type": "string"},
+                "table": {"type": "string"}}}}
+
+
+class ConditionsConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": [],
+        "additionalProperties": False,
+        "properties": {
+            "metavar": {"type": "string"},
+            "local": {"type": "boolean"},
+            "hosts": {"type": "array",
+                      "items": {"type": "string"}},
+            "envfrom": {"type": "string"},
+            "envto": {"type": "string"},
+            "headers": {"type": "array",
+                        "items": {"type": "string"}},
+            "var": {"type": "string"},
+            "whitelist": {"type": "object"}}}
+
+    def __init__(self, config, rec=True):
+        super().__init__(config)
+        if not rec:
+            return
+        if "whitelist" in self:
+            self["whitelist"] = WhitelistConfig(self["whitelist"])
+
+
+class AddHeaderConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["field", "value"],
+        "additionalProperties": False,
+        "properties": {
+            "field": {"type": "string"},
+            "value": {"type": "string"}}}
+
+
+class ModHeaderConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["field", "value"],
+        "additionalProperties": False,
+        "properties": {
+            "field": {"type": "string"},
+            "value": {"type": "string"},
+            "search": {"type": "string"}}}
+
+
+class DelHeaderConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["field"],
+        "additionalProperties": False,
+        "properties": {
+            "field": {"type": "string"},
+            "value": {"type": "string"}}}
+
+
+class AddDisclaimerConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["action", "html_template", "text_template"],
+        "additionalProperties": False,
+        "properties": {
+            "action": {"type": "string"},
+            "html_template": {"type": "string"},
+            "text_template": {"type": "string"},
+            "error_policy": {"type": "string", "default": "wrap"},
+            "add_html_body": {"type": "boolean", "default": False}}}
+
+
+class RewriteLinksConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["repl"],
+        "additionalProperties": False,
+        "properties": {
+            "repl": {"type": "string"}}}
+
+
+class StoreConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["type"],
+        "additionalProperties": True,
+        "properties": {
+            "type": {"enum": ["file"]}},
+        "if": {"properties": {"type": {"const": "file"}}},
+        "then": {
+            "required": ["directory"],
+            "additionalProperties": False,
+            "properties": {
+                "type": {"type": "string"},
+                "directory": {"type": "string"},
+                "mode": {"type": "string"},
+                "metavar": {"type": "string"},
+                "metadata": {"type": "boolean", "default": False},
+                "original": {"type": "boolean", "default": False}}}}
+
+
+class NotifyConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["type"],
+        "additionalProperties": True,
+        "properties": {
+            "type": {"enum": ["email"]}},
+        "if": {"properties": {"type": {"const": "email"}}},
+        "then": {
+            "required": ["smtp_host", "smtp_port", "envelope_from",
+                         "from_header", "subject", "template"],
+            "additionalProperties": False,
+            "properties": {
+                "type": {"type": "string"},
+                "smtp_host": {"type": "string"},
+                "smtp_port": {"type": "number"},
+                "envelope_from": {"type": "string"},
+                "from_header": {"type": "string"},
+                "subject": {"type": "string"},
+                "template": {"type": "string"},
+                "repl_img": {"type": "string"},
+                "strip_imgs": {"type": "boolean", "default": False},
+                "embed_imgs": {
+                    "type": "array",
+                    "items": {"type": "string"},
+                    "default": []}}}}
+
+
+class QuarantineConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["store", "smtp_host", "smtp_port"],
+        "additionalProperties": False,
+        "properties": {
+            "name": {"type": "string"},
+            "notify": {"type": "object"},
+            "milter_action": {"type": "string"},
+            "reject_reason": {"type": "string"},
+            "whitelist": {"type": "object"},
+            "store": {"type": "object"},
+            "smtp_host": {"type": "string"},
+            "smtp_port": {"type": "number"}}}
+
+    def __init__(self, config, rec=True):
+        super().__init__(config)
+        if not rec:
+            return
+        if "metadata" not in self["store"]:
+            self["store"]["metadata"] = True
+        self["store"] = StoreConfig(self["store"])
+        if "notify" in self:
+            self["notify"] = NotifyConfig(self["notify"])
+        if "whitelist" in self:
+            self["whitelist"] = ConditionsConfig(
+                {"whitelist": self["whitelist"]}, rec)
+
+
+class ActionConfig(BaseConfig):
+    ACTION_TYPES = {
+        "add_header": AddHeaderConfig,
+        "mod_header": ModHeaderConfig,
+        "del_header": DelHeaderConfig,
+        "add_disclaimer": AddDisclaimerConfig,
+        "rewrite_links": RewriteLinksConfig,
+        "store": StoreConfig,
+        "notify": NotifyConfig,
+        "quarantine": QuarantineConfig}
+
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["name", "type", "options"],
+        "additionalProperties": False,
+        "properties": {
+            "name": {"type": "string"},
+            "loglevel": {"type": "string", "default": "info"},
+            "pretend": {"type": "boolean", "default": False},
+            "conditions": {"type": "object"},
+            "type": {"enum": list(ACTION_TYPES.keys())},
+            "options": {"type": "object"}}}
+
+    def __init__(self, config, rec=True):
+        super().__init__(config)
+        if not rec:
+            return
+        if "conditions" in self:
+            self["conditions"] = ConditionsConfig(self["conditions"])
+        self["action"] = self.ACTION_TYPES[self["type"]](self["options"])
+
+
+class RuleConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["name", "actions"],
+        "additionalProperties": False,
+        "properties": {
+            "name": {"type": "string"},
+            "loglevel": {"type": "string", "default": "info"},
+            "pretend": {"type": "boolean", "default": False},
+            "conditions": {"type": "object"},
+            "actions": {"type": "array"}}}
+
+    def __init__(self, config, rec=True):
+        super().__init__(config)
+        if not rec:
+            return
+        if "conditions" in self:
+            self["conditions"] = ConditionsConfig(self["conditions"])
+
+        actions = []
+        for idx, action in enumerate(self["actions"]):
+            if "loglevel" not in action:
+                action["loglevel"] = config["loglevel"]
+            if "pretend" not in action:
+                action["pretend"] = config["pretend"]
+            actions.append(ActionConfig(action, rec))
+        self["actions"] = actions
+
+
+class QuarantineMilterConfig(BaseConfig):
+    JSON_SCHEMA = {
+        "type": "object",
+        "required": ["rules"],
+        "additionalProperties": False,
+        "properties": {
+            "socket": {"type": "string"},
+            "local_addrs": {"type": "array",
+                            "items": {"type": "string"},
+                            "default": [
+                                "fe80::/64",
+                                "::1/128",
+                                "127.0.0.0/8",
+                                "10.0.0.0/8",
+                                "172.16.0.0/12",
+                                "192.168.0.0/16"]},
+            "loglevel": {"type": "string", "default": "info"},
+            "pretend": {"type": "boolean", "default": False},
+            "rules": {"type": "array"}}}
+
+    def __init__(self, config, rec=True):
+        super().__init__(config)
+        if not rec:
+            return
+        rules = []
+        for idx, rule in enumerate(self["rules"]):
+            if "loglevel" not in rule:
+                rule["loglevel"] = config["loglevel"]
+            if "pretend" not in rule:
+                rule["pretend"] = config["pretend"]
+            rules.append(RuleConfig(rule, rec))
+        self["rules"] = rules
+
+
+def get_milter_config(cfgfile, raw=False):
+    try:
+        with open(cfgfile, "r") as fh:
+            # remove lines with leading # (comments), they
+            # are not allowed in json
+            cfg = re.sub(r"(?m)^\s*#.*\n?", "", fh.read())
+    except IOError as e:
+        raise RuntimeError(f"unable to open/read config file: {e}")
+
+    try:
+        cfg = json.loads(cfg)
+    except json.JSONDecodeError as e:
+        cfg_text = [f"{n+1}: {l}" for n, l in enumerate(cfg.splitlines())]
+        msg = "\n".join(cfg_text)
+        raise RuntimeError(f"{e}\n{msg}")
+    if raw:
+        return cfg
+    return QuarantineMilterConfig(cfg)

pyquarantine/mailer.py

@@ -1,20 +1,19 @@
-# PyQuarantine-Milter is free software: you can redistribute it and/or modify
+# pyquarantine is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
-# PyQuarantine-Milter is distributed in the hope that it will be useful,
+# pyquarantine is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
-# along with PyQuarantineMilter.  If not, see <http://www.gnu.org/licenses/>.
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
 #
 
 import logging
 import smtplib
-import sys
 
 from multiprocessing import Process, Queue
 
@@ -45,23 +44,22 @@ def mailprocess():
             if not m:
                 break
 
-            smtp_host, smtp_port, queueid, mailfrom, recipient, mail, emailtype = m
+            smtp_host, smtp_port, qid, mailfrom, recipient, mail, emailtype = m
             try:
                 smtp_send(smtp_host, smtp_port, mailfrom, recipient, mail)
             except Exception as e:
                 logger.error(
-                    "{}: error while sending {} to '{}': {}".format(
+                    f"{qid}: error while sending {emailtype} "
-                        queueid, emailtype, recipient, e))
+                    f"to '{recipient}': {e}")
             else:
                 logger.info(
-                    "{}: successfully sent {} to: {}".format(
+                    f"{qid}: successfully sent {emailtype} to: {recipient}")
-                        queueid, emailtype, recipient))
     except KeyboardInterrupt:
         pass
     logger.debug("mailer process terminated")
 
 
-def sendmail(smtp_host, smtp_port, queueid, mailfrom, recipients, mail,
+def sendmail(smtp_host, smtp_port, qid, mailfrom, recipients, mail,
              emailtype="email"):
     "Send an email."
     global logger
@@ -81,8 +79,8 @@ def sendmail(smtp_host, smtp_port, queueid, mailfrom, recipients, mail,
     for recipient in recipients:
         try:
             queue.put(
-                (smtp_host, smtp_port, queueid, mailfrom, recipient, mail,
+                (smtp_host, smtp_port, qid, mailfrom, recipient, mail,
                  emailtype),
                 timeout=30)
-        except Queue.Full as e:
+        except Queue.Full:
             raise RuntimeError("email queue is full")

pyquarantine/misc/openrc/pyquarantine-milter.confd

@@ -0,0 +1,8 @@
+# /etc/conf.d/pyquarantine-milter: config file for /etc/init.d/pyquarantine-milter
+
+# Start the daemon as the user. You can optionally append a group name here also.
+# USER="daemon"
+# USER="daemon:nobody"
+
+# Optional parameters for pyquarantine-milter
+# MILTER_OPTS=""

pyquarantine/misc/openrc/pyquarantine-milter.initd

@@ -1,16 +1,15 @@
 #!/sbin/openrc-run
 
 user=${USER:-daemon}
-socket="${SOCKET:-}"
 milter_opts="${MILTER_OPTS:-}"
 
 pidfile="/run/${RC_SVCNAME}.pid"
 command="/usr/bin/pyquarantine-milter"
-command_args="-s ${socket} ${milter_opts}"
+command_args="${milter_opts}"
 command_background=true
-start_stop_daemon_args="--user ${user}"
+command_user="${user}"
-
 extra_commands="configtest"
+start_stop_daemon_args="--wait 500"
 
 depend() {
 	need net
@@ -18,9 +17,6 @@ depend() {
 }
 
 checkconfig() {
-	if [ -z "${socket}" ]; then
-		eerror "No socket specified in config!"
-	fi
 	OUTPUT=$( ${command} ${command_args} -t 2>&1 )
 	ret=$?
 	if [ $ret -ne 0 ]; then
@@ -44,7 +40,7 @@ start_pre() {
 }
 
 stop_pre() {
-	if [ "${RC_CMD}" != "restart" ]; then
+	if [ "${RC_CMD}" == "restart" ]; then
 		checkconfig || return $?
 	fi
 }

pyquarantine/misc/pyquarantine.conf.default

@@ -0,0 +1,33 @@
+# This is an example /etc/pyquarantine/pyquarantine.conf file.
+# Copy it into place and edit before use.
+#
+# The file is in JSON format but comments are allowed.
+#
+{
+    # Option: socket (optional)
+    # Notes:  The socket used to communicate with the MTA.
+    #
+    #         Examples:
+    #         unix:/path/to/socket             a named pipe
+    #         inet:8899                        listen on ANY interface
+    #         inet:8899@localhost              listen on a specific interface
+    #         inet6:8899                       listen on ANY interface
+    #         inet6:8899@[2001:db8:1234::1]    listen on a specific interface
+    #
+    "socket": "inet:8898@127.0.0.1",
+
+    # Option: local_addrs (optional)
+    # Notes:  A list of local hosts and networks.
+    #
+    #"local_addrs": ["fe80::/64", "::1/128", "127.0.0.0/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"],
+
+    # Option: loglevel (optional)
+    # Notes:  The default loglevel.
+    #         
+    #"loglevel": "info",
+
+    # Option: rules
+    # Notes:  List of rules.
+    #
+    "rules": []
+}

pyquarantine/misc/systemd/pyquarantine-milter.service

@@ -0,0 +1,14 @@
+[Unit]
+Description=pyquarantine-milter
+After=network.target
+
+[Service]
+Type=simple
+EnvironmentFile=/etc/conf.d/pyquarantine-milter
+ExecStart=/usr/bin/pyquarantine-milter $MILTER_OPTS
+User=mail
+Group=mail
+TimeoutStopSec=300
+
+[Install]
+WantedBy=multi-user.target

pyquarantine/misc/templates/disclaimer_html.template

@@ -0,0 +1,9 @@
+<table style="border: 1px solid; background-color: #F8E898; border-color: #885800; font-family: Arial; font-size: 11pt;">
+ <tr>
+  <td>
+   <span style="font-weight: bold; color: #905800;">CAUTION:</span> This email originated from outside the organization.
+   Do not follow guidance, click links or open attachments unless you recognize the sender and know the content is safe.
+  </td>
+ </tr>
+</table>
+<br/><br/>

pyquarantine/misc/templates/disclaimer_text.template

@@ -0,0 +1,4 @@
+
+CAUTION: This email originated from outside the organization. Do not follow guidance, click links or open attachments unless you recognize the sender and know the content is safe.
+
+

pyquarantine/misc/templates/notification.template

@@ -4,26 +4,26 @@
   <table>
    <tr>
     <td><b>Envelope-From:</b></td>
-    <td>{EMAIL_ENVELOPE_FROM}</td>
+    <td>{ENVELOPE_FROM}</td>
    </tr>
    <tr>
     <td><b>From:</b></td>
-    <td>{EMAIL_FROM}</td>
+    <td>{FROM}</td>
+   </tr>
+   <tr>
+    <td><b>Envelope-To:</b></td>
+    <td>{ENVELOPE_TO}</td>
    </tr>
    <tr>
     <td><b>To:</b></td>
-    <td>{EMAIL_TO}</td>
+    <td>{TO}</td>
    </tr>
    <tr>
     <td><b>Subject:</b></td>
-    <td>{EMAIL_SUBJECT}</td>
+    <td>{SUBJECT}</td>
-   </tr>
-   <tr>
-    <td><b>Quarantine ID:</b></td>
-    <td>{EMAIL_QUARANTINE_ID}</td>
    </tr>
   </table><br/>
   <h2>Preview of the original e-mail</h2>
-  {EMAIL_HTML_TEXT}
+  {HTML_TEXT}
  </body>
 </html>

pyquarantine/modify.py

@@ -0,0 +1,401 @@
+# pyquarantine is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# pyquarantine is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+__all__ = [
+    "AddHeader",
+    "ModHeader",
+    "DelHeader",
+    "AddDisclaimer",
+    "RewriteLinks",
+    "Modify"]
+
+import logging
+import re
+
+from base64 import b64encode
+from bs4 import BeautifulSoup
+from collections import defaultdict
+from copy import copy
+from email.message import MIMEPart
+from email.policy import SMTP
+from html import escape
+from urllib.parse import quote
+
+from pyquarantine import replace_illegal_chars
+from pyquarantine.base import CustomLogger
+
+
+class AddHeader:
+    """Add a mail header field."""
+    _headersonly = True
+
+    def __init__(self, field, value, pretend=False):
+        self.field = field
+        self.value = value
+        self.pretend = pretend
+
+    def execute(self, milter, logger):
+        header = f"{self.field}: {self.value}"
+        if logger.getEffectiveLevel() == logging.DEBUG:
+            logger.debug(f"add_header: {header}")
+        else:
+            logger.info(f"add_header: {header[0:70]}")
+
+        milter.msg.add_header(self.field, self.value)
+        if not self.pretend:
+            milter.addheader(self.field, self.value)
+
+
+class ModHeader:
+    """Change the value of a mail header field."""
+    _headersonly = True
+
+    def __init__(self, field, value, search=None, pretend=False):
+        try:
+            self.field = re.compile(field, re.IGNORECASE)
+            self.search = search
+            if self.search is not None:
+                self.search = re.compile(
+                    self.search, re.MULTILINE + re.DOTALL + re.IGNORECASE)
+
+        except re.error as e:
+            raise RuntimeError(e)
+
+        self.value = value
+        self.pretend = pretend
+
+    def execute(self, milter, logger):
+        idx = defaultdict(int)
+
+        for i, (field, value) in enumerate(milter.msg.items()):
+            field_lower = field.lower()
+            idx[field_lower] += 1
+
+            if not self.field.match(field):
+                continue
+
+            new_value = value
+            if self.search is not None:
+                new_value = self.search.sub(self.value, value).strip()
+            else:
+                new_value = self.value
+
+            if not new_value:
+                logger.warning(
+                    "mod_header: resulting value is empty, "
+                    "skip modification")
+                continue
+
+            if new_value == value:
+                continue
+
+            header = f"{field}: {value}"
+            new_header = f"{field}: {new_value}"
+
+            if logger.getEffectiveLevel() == logging.DEBUG:
+                logger.debug(f"mod_header: {header}: {new_header}")
+            else:
+                logger.info(
+                    f"mod_header: {header[0:70]}: {new_header[0:70]}")
+
+            milter.msg.replace_header(
+                field, replace_illegal_chars(new_value), idx=idx[field_lower])
+
+            if not self.pretend:
+                milter.chgheader(field, new_value, idx=idx[field_lower])
+
+
+class DelHeader:
+    """Delete a mail header field."""
+    _headersonly = True
+
+    def __init__(self, field, value=None, pretend=False):
+        try:
+            self.field = re.compile(field, re.IGNORECASE)
+            self.value = value
+            if self.value is not None:
+                self.value = re.compile(
+                    value, re.MULTILINE + re.DOTALL + re.IGNORECASE)
+        except re.error as e:
+            raise RuntimeError(e)
+
+        self.pretend = pretend
+
+    def execute(self, milter, logger):
+        idx = defaultdict(int)
+
+        for field, value in milter.msg.items():
+            field_lower = field.lower()
+            idx[field_lower] += 1
+
+            if not self.field.match(field):
+                continue
+
+            if self.value is not None and not self.value.search(value):
+                continue
+
+            header = f"{field}: {value}"
+            if logger.getEffectiveLevel() == logging.DEBUG:
+                logger.debug(f"del_header: {header}")
+            else:
+                logger.info(f"del_header: {header[0:70]}")
+            milter.msg.remove_header(field, idx=idx[field_lower])
+
+            if not self.pretend:
+                milter.chgheader(field, "", idx=idx[field_lower])
+
+            idx[field_lower] -= 1
+
+
+def _get_body_content(msg, pref):
+    part = None
+    content = None
+    if not msg.is_multipart() and msg.get_content_type() == f"text/{pref}":
+        part = msg
+    else:
+        part = msg.get_body(preferencelist=(pref))
+
+    if part is not None:
+        content = part.get_content()
+
+    return (part, content)
+
+
+def _has_content_before_body_tag(soup):
+    s = copy(soup)
+    for element in s.find_all("head") + s.find_all("body"):
+        element.extract()
+
+    if len(s.text.strip()) > 0:
+        return True
+
+    return False
+
+
+def _wrap_message(milter):
+    attachment = MIMEPart(policy=SMTP)
+    attachment.set_content(milter.msg_as_bytes(),
+                           maintype="plain", subtype="text",
+                           disposition="attachment",
+                           filename=f"{milter.qid}.eml",
+                           params={"name": f"{milter.qid}.eml"})
+
+    milter.msg.clear_content()
+    milter.msg.set_content(
+        "Please see the original email attached.")
+    milter.msg.add_alternative(
+        "<html><body>Please see the original email attached.</body></html>",
+        subtype="html")
+    milter.msg.make_mixed()
+    milter.msg.attach(attachment)
+
+
+class AddDisclaimer:
+    """Append or prepend a disclaimer to the mail body."""
+    _headersonly = False
+
+    def __init__(self, text_template, html_template, action, error_policy,
+                 add_html_body, pretend=False):
+        self.text_template_path = text_template
+        self.html_template_path = html_template
+        try:
+            with open(text_template, "r") as f:
+                self.text_template = f.read()
+
+            with open(html_template, "r") as f:
+                self.html_template = f.read()
+
+        except IOError as e:
+            raise RuntimeError(e)
+        self.action = action.lower()
+        assert self.action in ["prepend", "append"], \
+            f"invalid action '{action}'"
+        self.error_policy = error_policy.lower()
+        assert self.error_policy in ["ignore", "reject", "wrap"], \
+            f"invalid error_policy '{error_policy}'"
+        self.add_html_body = add_html_body
+        self.pretend = pretend
+
+    def patch_message_body(self, milter, logger):
+        text_body, text_content = milter.msg.get_body_content("plain")
+        html_body, html_content = milter.msg.get_body_content("html")
+
+        if text_content is None and html_content is None:
+            logger.info("message contains no body, inject it")
+            if self.add_html_body:
+                milter.msg.set_body("", "")
+                html_body, html_content = milter.msg.get_body_content("html")
+            else:
+                milter.msg.set_body("")
+            text_body, text_content = milter.msg.get_body_content("plain")
+
+        if html_content is None and self.add_html_body:
+            logger.info("inject html body based on plain body")
+            header = '<meta http-equiv="Content-Type" content="text/html; ' \
+                     'charset=utf-8">'
+            html_text = re.sub(r"^(.*)$", r"\1<br/>",
+                               escape(text_content, quote=False),
+                               flags=re.MULTILINE)
+            milter.msg.set_body(None, f"{header}{html_text}")
+            text_body, text_content = milter.msg.get_body_content("plain")
+            html_body, html_content = milter.msg.get_body_content("html")
+
+        variables = defaultdict(str, milter.msginfo["vars"])
+        variables["ENVELOPE_FROM"] = escape(
+            milter.msginfo["mailfrom"], quote=False)
+        variables["ENVELOPE_FROM_URL"] = escape(
+            quote(milter.msginfo["mailfrom"]), quote=False)
+
+        if text_content is not None:
+            logger.info(f"{self.action} text disclaimer")
+            text_template = self.text_template.format_map(variables)
+
+            if self.action == "prepend":
+                content = f"{text_template}{text_content}"
+            else:
+                content = f"{text_content}{text_template}"
+
+            text_body.set_content(
+                content.encode(errors="replace"),
+                maintype="text",
+                subtype="plain")
+            text_body.set_param("charset", "UTF-8", header="Content-Type")
+            del text_body["MIME-Version"]
+
+        if html_content is not None:
+            logger.info(f"{self.action} html disclaimer")
+
+            soup = BeautifulSoup(html_content, "html.parser")
+            body = soup.find('body')
+            if not body:
+                body = soup
+            elif _has_content_before_body_tag(soup):
+                body = soup
+
+            html_template = self.html_template.format_map(variables)
+            html_template = BeautifulSoup(html_template, "html.parser")
+            html_template = html_template.find("body") or html_template
+            if self.action == "prepend":
+                body.insert(0, html_template)
+            else:
+                body.append(html_template)
+
+            html_body.set_content(
+                str(soup).encode(errors="replace"),
+                maintype="text",
+                subtype="html")
+            html_body.set_param("charset", "UTF-8", header="Content-Type")
+            del html_body["MIME-Version"]
+
+    def execute(self, milter, logger):
+        old_headers = milter.msg.items()
+
+        try:
+            self.patch_message_body(milter, logger)
+        except Exception as e:
+            logger.warning(e)
+            if self.error_policy == "ignore":
+                logger.info(
+                    "unable to add disclaimer to message body, "
+                    "ignore error according to policy")
+                return
+            elif self.error_policy == "reject":
+                logger.info(
+                    "unable to add disclaimer to message body, "
+                    "reject message according to policy")
+                return [
+                    ("reject", "Message rejected due to error")]
+
+            logger.info("wrap original message in a new message envelope")
+            try:
+                _wrap_message(milter)
+                self.patch_message_body(milter, logger)
+            except Exception as e:
+                logger.error(e)
+                raise Exception(
+                    "unable to wrap message in a new message envelope, "
+                    "give up ...")
+
+        if not self.pretend:
+            milter.update_headers(old_headers)
+            milter.replacebody()
+
+
+class RewriteLinks:
+    """Rewrite link targets in the mail html body."""
+    _headersonly = False
+
+    def __init__(self, repl, pretend=False):
+        self.repl = repl
+        self.pretend = pretend
+
+    def execute(self, milter, logger):
+        html_body, html_content = _get_body_content(milter.msg, "html")
+        if html_content is not None:
+            soup = BeautifulSoup(html_content, "html.parser")
+
+            rewritten = 0
+            for link in soup.find_all("a", href=True):
+                if not link["href"]:
+                    continue
+
+                if "{URL_B64}" in self.repl:
+                    url_b64 = b64encode(link["href"].encode()).decode()
+                    target = self.repl.replace("{URL_B64}", url_b64)
+                else:
+                    target = self.repl
+
+                link["href"] = target
+                rewritten += 1
+
+            if rewritten:
+                logger.info(f"rewrote {rewritten} link(s) in html body")
+
+                html_body.set_content(
+                    str(soup).encode(), maintype="text", subtype="html")
+                html_body.set_param("charset", "UTF-8", header="Content-Type")
+                del html_body["MIME-Version"]
+
+                if not self.pretend:
+                    milter.replacebody()
+
+
+class Modify:
+    MODIFICATION_TYPES = {
+        "add_header": AddHeader,
+        "mod_header": ModHeader,
+        "del_header": DelHeader,
+        "add_disclaimer": AddDisclaimer,
+        "rewrite_links": RewriteLinks}
+
+    def __init__(self, cfg, local_addrs, debug):
+        self.cfg = cfg
+        self.logger = logging.getLogger(cfg["name"])
+        self.logger.setLevel(cfg.get_loglevel(debug))
+        cfg["options"]["pretend"] = cfg["pretend"]
+        self._modification = self.MODIFICATION_TYPES[cfg["type"]](
+            **cfg["options"])
+        self._headersonly = self._modification._headersonly
+
+    def __str__(self):
+        cfg = []
+        for key, value in self.cfg["options"].items():
+            cfg.append(f"{key}={value}")
+        class_name = type(self._modification).__name__
+        return f"{class_name}(" + ", ".join(cfg) + ")"
+
+    def execute(self, milter):
+        logger = CustomLogger(
+            self.logger, {"name": self.cfg["name"], "qid": milter.qid})
+        self._modification.execute(milter, logger)

pyquarantine/notifications.py

@@ -1,419 +0,0 @@
-# PyQuarantine-Milter is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# PyQuarantine-Milter is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with PyQuarantineMilter.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-import email
-import logging
-import re
-
-from bs4 import BeautifulSoup
-from cgi import escape
-from collections import defaultdict
-from email.mime.multipart import MIMEMultipart
-from email.mime.text import MIMEText
-from email.mime.image import MIMEImage
-from os.path import basename
-
-from pyquarantine import mailer
-
-
-class BaseNotification(object):
-    "Notification base class"
-
-    def __init__(self, global_config, config, configtest=False):
-        self.quarantine_name = config["name"]
-        self.global_config = global_config
-        self.config = config
-        self.logger = logging.getLogger(__name__)
-
-    def notify(self, queueid, quarantine_id, mailfrom, recipients, headers,
-               fp, subgroups=None, named_subgroups=None, synchronous=False):
-        fp.seek(0)
-        pass
-
-
-class EMailNotification(BaseNotification):
-    "Notification class to send notifications via mail."
-    _html_text = "text/html"
-    _plain_text = "text/plain"
-    _bad_tags = [
-        "applet",
-        "embed",
-        "frame",
-        "frameset",
-        "head",
-        "iframe",
-        "script"
-    ]
-    _good_tags = [
-        "a",
-        "b",
-        "br",
-        "center",
-        "div",
-        "font",
-        "h1",
-        "h2",
-        "h3",
-        "h4",
-        "h5",
-        "h6",
-        "i",
-        "img",
-        "li",
-        "p",
-        "pre",
-        "span",
-        "table",
-        "td",
-        "th",
-        "tr",
-        "tt",
-        "u",
-        "ul"
-    ]
-    _good_attributes = [
-        "align",
-        "alt",
-        "bgcolor",
-        "border",
-        "cellpadding",
-        "cellspacing",
-        "class",
-        "color",
-        "colspan",
-        "dir",
-        "face",
-        "headers",
-        "height",
-        "id",
-        "name",
-        "rowspan",
-        "size",
-        "src",
-        "style",
-        "title",
-        "type",
-        "valign",
-        "value",
-        "width"
-    ]
-
-    def __init__(self, global_config, config, configtest=False):
-        super(EMailNotification, self).__init__(
-            global_config, config, configtest)
-
-        # check if mandatory options are present in config
-        for option in [
-            "notification_email_smtp_host",
-            "notification_email_smtp_port",
-            "notification_email_envelope_from",
-            "notification_email_from",
-            "notification_email_subject",
-            "notification_email_template",
-                "notification_email_embedded_imgs"]:
-            if option not in self.config.keys() and option in self.global_config.keys():
-                self.config[option] = self.global_config[option]
-            if option not in self.config.keys():
-                raise RuntimeError(
-                    "mandatory option '{}' not present in config section '{}' or 'global'".format(
-                        option, self.quarantine_name))
-
-        # check if optional config options are present in config
-        defaults = {
-            "notification_email_replacement_img": "",
-            "notification_email_strip_images": "false",
-            "notification_email_parser_lib": "lxml"
-        }
-        for option in defaults.keys():
-            if option not in config.keys() and \
-                    option in global_config.keys():
-                config[option] = global_config[option]
-            if option not in config.keys():
-                config[option] = defaults[option]
-
-        self.smtp_host = self.config["notification_email_smtp_host"]
-        self.smtp_port = self.config["notification_email_smtp_port"]
-        self.mailfrom = self.config["notification_email_envelope_from"]
-        self.from_header = self.config["notification_email_from"]
-        self.subject = self.config["notification_email_subject"]
-
-        testvars = defaultdict(str, test="TEST")
-
-        # test-parse from header
-        try:
-            self.from_header.format_map(testvars)
-        except ValueError as e:
-            raise RuntimeError(
-                "error parsing notification_email_from: {}".format(e))
-
-        # test-parse subject
-        try:
-            self.subject.format_map(testvars)
-        except ValueError as e:
-            raise RuntimeError(
-                "error parsing notification_email_subject: {}".format(e))
-
-        # read and parse email notification template
-        try:
-            self.template = open(
-                self.config["notification_email_template"], "r").read()
-            self.template.format_map(testvars)
-        except IOError as e:
-            raise RuntimeError("error reading template: {}".format(e))
-        except ValueError as e:
-            raise RuntimeError("error parsing template: {}".format(e))
-
-        strip_images = self.config["notification_email_strip_images"].strip().upper()
-        if strip_images in ["TRUE", "ON", "YES"]:
-            self.strip_images = True
-        elif strip_images in ["FALSE", "OFF", "NO"]:
-            self.strip_images = False
-        else:
-            raise RuntimeError("error parsing notification_email_strip_images: unknown value")
-
-        self.parser_lib = self.config["notification_email_parser_lib"].strip()
-        if self.parser_lib not in ["lxml", "html.parser"]:
-            raise RuntimeError("error parsing notification_email_parser_lib: unknown value")
-
-        # read email replacement image if specified
-        replacement_img = self.config["notification_email_replacement_img"].strip()
-        if not self.strip_images and replacement_img:
-            try:
-                self.replacement_img = MIMEImage(
-                    open(replacement_img, "rb").read())
-            except IOError as e:
-                raise RuntimeError(
-                    "error reading replacement image: {}".format(e))
-            else:
-                self.replacement_img.add_header(
-                    "Content-ID", "<removed_for_security_reasons>")
-        else:
-            self.replacement_img = None
-
-        # read images to embed if specified
-        embedded_img_paths = [
-            p.strip() for p in self.config["notification_email_embedded_imgs"].split(",") if p]
-        self.embedded_imgs = []
-        for img_path in embedded_img_paths:
-            # read image
-            try:
-                img = MIMEImage(open(img_path, "rb").read())
-            except IOError as e:
-                raise RuntimeError("error reading image: {}".format(e))
-            else:
-                img.add_header("Content-ID", "<{}>".format(basename(img_path)))
-                self.embedded_imgs.append(img)
-
-    def get_decoded_email_body(self, queueid, msg, preferred=_html_text):
-        "Find and decode email body."
-        # try to find the body part
-        self.logger.debug("{}: trying to find email body".format(queueid))
-        body = None
-        for part in msg.walk():
-            content_type = part.get_content_type()
-            if content_type in [EMailNotification._plain_text,
-                                EMailNotification._html_text]:
-                body = part
-                if content_type == preferred:
-                    break
-
-        if body is not None:
-            # get the character set, fallback to utf-8 if not defined in header
-            charset = body.get_content_charset()
-            if charset is None:
-                charset = "utf-8"
-
-            # decode content
-            content = body.get_payload(decode=True).decode(
-                encoding=charset, errors="replace")
-
-            content_type = body.get_content_type()
-            if content_type == EMailNotification._plain_text:
-                # convert text/plain to text/html
-                self.logger.debug(
-                    "{}: content type is {}, converting to {}".format(
-                        queueid, content_type, EMailNotification._html_text))
-                content = re.sub(r"^(.*)$", r"\1<br/>",
-                                 escape(content), flags=re.MULTILINE)
-            else:
-                self.logger.debug(
-                    "{}: content type is {}".format(
-                        queueid, content_type))
-        else:
-            self.logger.error(
-                "{}: unable to find email body".format(queueid))
-            content = "ERROR: unable to find email body"
-
-        return content
-
-    def sanitize(self, queueid, soup):
-        "Sanitize mail html text."
-        self.logger.debug("{}: sanitizing email text".format(queueid))
-
-        # completly remove bad elements
-        for element in soup(EMailNotification._bad_tags):
-            self.logger.debug(
-                "{}: removing dangerous tag '{}' and its content".format(
-                    queueid, element.name))
-            element.extract()
-
-        # remove not whitelisted elements, but keep their content
-        for element in soup.find_all(True):
-            if element.name not in EMailNotification._good_tags:
-                self.logger.debug(
-                    "{}: removing tag '{}', keep its content".format(
-                        queueid, element.name))
-                element.replaceWithChildren()
-
-        # remove not whitelisted attributes
-        for element in soup.find_all(True):
-            for attribute in list(element.attrs.keys()):
-                if attribute not in EMailNotification._good_attributes:
-                    if element.name == "a" and attribute == "href":
-                        self.logger.debug(
-                            "{}: setting attribute href to '#' on tag '{}'".format(
-                                queueid, element.name))
-                        element["href"] = "#"
-                    else:
-                        self.logger.debug(
-                            "{}: removing attribute '{}' from tag '{}'".format(
-                                queueid, attribute, element.name))
-                        del(element.attrs[attribute])
-        return soup
-
-    def notify(self, queueid, quarantine_id, mailfrom, recipients, headers, fp,
-               subgroups=None, named_subgroups=None, synchronous=False):
-        "Notify recipients via email."
-        super(
-            EMailNotification,
-            self).notify(
-            queueid,
-            quarantine_id,
-            mailfrom,
-            recipients,
-            headers,
-            fp,
-            subgroups,
-            named_subgroups,
-            synchronous)
-
-        # extract body from email
-        content = self.get_decoded_email_body(
-            queueid, email.message_from_binary_file(fp))
-
-        # create BeautifulSoup object
-        self.logger.debug(
-            "{}: trying to create BeatufilSoup object with parser lib {}, "
-            "text length is {} bytes".format(
-                queueid, self.parser_lib, len(content)))
-        soup = BeautifulSoup(content, self.parser_lib)
-        self.logger.debug(
-            "{}: sucessfully created BeautifulSoup object".format(queueid))
-
-        # replace picture sources
-        image_replaced = False
-        if self.strip_images:
-            self.logger.debug(
-                "{}: looking for images to strip".format(queueid))
-            for element in soup("img"):
-                if "src" in element.attrs.keys():
-                    self.logger.debug(
-                        "{}: strip image: {}".format(
-                            queueid, element["src"]))
-                element.extract()
-        elif self.replacement_img:
-            self.logger.debug(
-                "{}: looking for images to replace".format(queueid))
-            for element in soup("img"):
-                if "src" in element.attrs.keys():
-                    self.logger.debug(
-                        "{}: replacing image: {}".format(
-                            queueid, element["src"]))
-                element["src"] = "cid:removed_for_security_reasons"
-                image_replaced = True
-
-        # sanitizing email text of original email
-        sanitized_text = self.sanitize(queueid, soup)
-        del soup
-
-        # sending email notifications
-        for recipient in recipients:
-            self.logger.debug(
-                "{}: generating notification email for '{}'".format(
-                    queueid, recipient))
-            self.logger.debug("{}: parsing email template".format(queueid))
-            if "from" not in headers.keys():
-                headers["from"] = ""
-            if "to" not in headers.keys():
-                headers["to"] = ""
-            if "subject" not in headers.keys():
-                headers["subject"] = ""
-            # generate dict containing all template variables
-            variables = defaultdict(str,
-                                    EMAIL_HTML_TEXT=sanitized_text,
-                                    EMAIL_FROM=escape(headers["from"]),
-                                    EMAIL_ENVELOPE_FROM=escape(mailfrom),
-                                    EMAIL_TO=escape(recipient),
-                                    EMAIL_SUBJECT=escape(headers["subject"]),
-                                    EMAIL_QUARANTINE_ID=quarantine_id)
-
-            if subgroups:
-                number = 0
-                for subgroup in subgroups:
-                    variables["SUBGROUP_{}".format(number)] = escape(subgroup)
-            if named_subgroups:
-                for key, value in named_subgroups.items():
-                    named_subgroups[key] = escape(value)
-                variables.update(named_subgroups)
-
-            # parse template
-            htmltext = self.template.format_map(variables)
-
-            msg = MIMEMultipart('related')
-            msg["From"] = self.from_header.format_map(
-                defaultdict(str, EMAIL_FROM=headers["from"]))
-            msg["To"] = headers["to"]
-            msg["Subject"] = self.subject.format_map(
-                defaultdict(str, EMAIL_SUBJECT=headers["subject"]))
-            msg["Date"] = email.utils.formatdate()
-            msg.attach(MIMEText(htmltext, "html", 'UTF-8'))
-
-            if image_replaced:
-                self.logger.debug(
-                    "{}: attaching notification_replacement_img".format(queueid))
-                msg.attach(self.replacement_img)
-
-            for img in self.embedded_imgs:
-                self.logger.debug("{}: attaching imgage".format(queueid))
-                msg.attach(img)
-
-            self.logger.debug(
-                "{}: sending notification email to: {}".format(
-                    queueid, recipient))
-            if synchronous:
-                try:
-                    mailer.smtp_send(self.smtp_host, self.smtp_port,
-                                     self.mailfrom, recipient, msg.as_string())
-                except Exception as e:
-                    raise RuntimeError(
-                        "error while sending email to '{}': {}".format(
-                            recipient, e))
-            else:
-                mailer.sendmail(self.smtp_host, self.smtp_port, queueid,
-                                self.mailfrom, recipient, msg.as_string(),
-                                "notification email")
-
-
-# list of notification types and their related notification classes
-TYPES = {"email": EMailNotification}

pyquarantine/notify.py

@@ -0,0 +1,356 @@
+# pyquarantine is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# pyquarantine is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+__all__ = [
+    "BaseNotification",
+    "EMailNotification",
+    "Notify"]
+
+import email
+import logging
+import re
+
+from bs4 import BeautifulSoup
+from collections import defaultdict
+from email.mime.multipart import MIMEMultipart
+from email.mime.text import MIMEText
+from email.mime.image import MIMEImage
+from html import escape
+from os.path import basename
+from urllib.parse import quote
+
+from pyquarantine.base import CustomLogger
+from pyquarantine import mailer
+
+
+class BaseNotification:
+    "Notification base class"
+    _headersonly = True
+
+    def __init__(self, pretend=False):
+        self.pretend = pretend
+
+    def execute(self, milter, logger):
+        return
+
+
+class EMailNotification(BaseNotification):
+    "Notification class to send notifications via mail."
+    _headersonly = False
+    _bad_tags = [
+        "applet",
+        "embed",
+        "frame",
+        "frameset",
+        "head",
+        "iframe",
+        "script",
+        "style"
+    ]
+    _good_tags = [
+        "a",
+        "b",
+        "br",
+        "center",
+        "div",
+        "font",
+        "h1",
+        "h2",
+        "h3",
+        "h4",
+        "h5",
+        "h6",
+        "i",
+        "img",
+        "li",
+        "p",
+        "pre",
+        "span",
+        "table",
+        "td",
+        "th",
+        "tr",
+        "tt",
+        "u",
+        "ul"
+    ]
+    _good_attributes = [
+        "align",
+        "alt",
+        "bgcolor",
+        "border",
+        "cellpadding",
+        "cellspacing",
+        "class",
+        "color",
+        "colspan",
+        "dir",
+        "face",
+        "headers",
+        "height",
+        "id",
+        "name",
+        "rowspan",
+        "size",
+        "src",
+        "style",
+        "title",
+        "type",
+        "valign",
+        "value",
+        "width"
+    ]
+
+    def __init__(self, smtp_host, smtp_port, envelope_from, from_header,
+                 subject, template, embed_imgs=[], repl_img=None,
+                 strip_imgs=False, parser_lib="lxml", pretend=False):
+        super().__init__(pretend)
+        self.smtp_host = smtp_host
+        self.smtp_port = smtp_port
+        self.mailfrom = envelope_from
+        self.from_header = from_header
+        self.subject = subject
+        try:
+            with open(template, "r") as fh:
+                self.template = fh.read()
+            self.embed_imgs = []
+            for img_path in embed_imgs:
+                with open(img_path, "rb") as fh:
+                    img = MIMEImage(fh.read())
+                filename = basename(img_path)
+                img.add_header("Content-ID", f"<{filename}>")
+                self.embed_imgs.append(img)
+            self.replacement_img = repl_img
+            self.strip_images = strip_imgs
+
+            if not strip_imgs and repl_img:
+                with open(repl_img, "rb") as fh:
+                    self.replacement_img = MIMEImage(fh.read())
+                self.replacement_img.add_header(
+                    "Content-ID", "<removed_for_security_reasons>")
+
+        except IOError as e:
+            raise RuntimeError(e)
+
+        self.parser_lib = parser_lib
+
+    def get_msg_body_soup(self, msg, logger):
+        "Extract and decode message body, return it as BeautifulSoup object."
+        # try to find the body part
+        logger.debug("trying to find message body")
+        try:
+            body = msg.get_body(preferencelist=("html", "plain"))
+        except Exception as e:
+            logger.error(
+                f"an error occured in email.message.EmailMessage.get_body: "
+                f"{e}")
+            body = None
+
+        if body:
+            charset = body.get_content_charset() or "utf-8"
+            content = body.get_payload(decode=True)
+            try:
+                content = content.decode(encoding=charset, errors="replace")
+            except LookupError:
+                logger.info(
+                    f"unknown encoding '{charset}', falling back to UTF-8")
+                content = content.decode("utf-8", errors="replace")
+            content_type = body.get_content_type()
+            if content_type == "text/plain":
+                # convert text/plain to text/html
+                logger.debug(
+                    f"content type is {content_type}, "
+                    f"converting to text/html")
+                content = re.sub(r"^(.*)$", r"\1<br/>",
+                                 escape(content, quote=False),
+                                 flags=re.MULTILINE)
+            else:
+                logger.debug(f"content type is {content_type}")
+        else:
+            logger.error("unable to find message body")
+            content = "ERROR: unable to find message body"
+
+        # create BeautifulSoup object
+        length = len(content)
+        logger.debug(
+            f"trying to create BeatufilSoup object with "
+            f"parser lib {self.parser_lib}, "
+            f"text length is {length} bytes")
+        soup = BeautifulSoup(content, self.parser_lib)
+        logger.debug("sucessfully created BeautifulSoup object")
+
+        return soup
+
+    def sanitize(self, soup, logger):
+        "Sanitize mail html text."
+        logger.debug("sanitize message text")
+
+        # completly remove bad elements
+        for element in soup(EMailNotification._bad_tags):
+            logger.debug(
+                f"removing dangerous tag '{element.name}' "
+                f"and its content")
+            element.extract()
+
+        # remove not whitelisted elements, but keep their content
+        for element in soup.find_all(True):
+            if element.name not in EMailNotification._good_tags:
+                logger.debug(
+                    f"removing tag '{element.name}', keep its content")
+                element.replaceWithChildren()
+
+        # remove not whitelisted attributes
+        for element in soup.find_all(True):
+            for attribute in list(element.attrs.keys()):
+                if attribute not in EMailNotification._good_attributes:
+                    if element.name == "a" and attribute == "href":
+                        logger.debug(
+                            f"setting attribute href to '#' "
+                            f"on tag '{element.name}'")
+                        element["href"] = "#"
+                    else:
+                        logger.debug(
+                            f"removing attribute '{attribute}' "
+                            f"from tag '{element.name}'")
+                        del(element.attrs[attribute])
+        return soup
+
+    def notify(self, msg, qid, mailfrom, recipients, logger,
+               template_vars={}, synchronous=False):
+        "Notify recipients via email."
+        # extract body from email
+        soup = self.get_msg_body_soup(msg, logger)
+
+        # replace picture sources
+        image_replaced = False
+        if self.strip_images:
+            logger.debug("looking for images to strip")
+            for element in soup("img"):
+                if "src" in element.attrs.keys():
+                    logger.debug(f"strip image: {element['src']}")
+                element.extract()
+        elif self.replacement_img:
+            logger.debug("looking for images to replace")
+            for element in soup("img"):
+                if "src" in element.attrs.keys():
+                    logger.debug(f"replacing image: {element['src']}")
+                element["src"] = "cid:removed_for_security_reasons"
+                image_replaced = True
+
+        # sanitize message text
+        sanitized_text = self.sanitize(soup, logger)
+        del soup
+
+        # send email notifications
+        for recipient in recipients:
+            logger.debug(
+                f"generating email notification for '{recipient}'")
+            logger.debug("parsing message template")
+
+            variables = defaultdict(str, template_vars)
+            variables["HTML_TEXT"] = sanitized_text
+            variables["ENVELOPE_FROM"] = escape(mailfrom, quote=False)
+            variables["ENVELOPE_FROM_URL"] = escape(
+                quote(mailfrom), quote=False)
+            variables["ENVELOPE_TO"] = escape(recipient, quote=False)
+            variables["ENVELOPE_TO_URL"] = escape(quote(recipient))
+
+            newmsg = MIMEMultipart('related')
+            if msg["from"] is not None:
+                newmsg["From"] = self.from_header.format_map(
+                    defaultdict(str, FROM=msg["from"]))
+                variables["FROM"] = escape(msg["from"], quote=False)
+            else:
+                newmsg["From"] = self.from_header.format_map(defaultdict(str))
+
+            if msg["to"] is not None:
+                newmsg["To"] = msg["to"]
+                variables["TO"] = escape(msg["to"], quote=False)
+            else:
+                newmsg["To"] = recipient
+
+            if msg["subject"] is not None:
+                newmsg["Subject"] = self.subject.format_map(
+                    defaultdict(str, SUBJECT=msg["subject"]))
+                variables["SUBJECT"] = escape(msg["subject"], quote=False)
+
+            newmsg["Date"] = email.utils.formatdate()
+
+            # parse template
+            htmltext = self.template.format_map(variables)
+            newmsg.attach(MIMEText(htmltext, "html", 'UTF-8'))
+
+            if image_replaced:
+                logger.debug("attaching notification_replacement_img")
+                newmsg.attach(self.replacement_img)
+
+            for img in self.embed_imgs:
+                logger.debug("attaching imgage")
+                newmsg.attach(img)
+
+            logger.debug(f"sending email notification to: {recipient}")
+            if synchronous:
+                try:
+                    mailer.smtp_send(self.smtp_host, self.smtp_port,
+                                     self.mailfrom, recipient,
+                                     newmsg.as_string())
+                except Exception as e:
+                    raise RuntimeError(
+                        f"error while sending email notification "
+                        f"to '{recipient}': {e}")
+            else:
+                mailer.sendmail(self.smtp_host, self.smtp_port, qid,
+                                self.mailfrom, recipient, newmsg.as_string(),
+                                "email notification")
+
+    def execute(self, milter, logger):
+        super().execute(milter, logger)
+
+        self.notify(msg=milter.msg, qid=milter.qid,
+                    mailfrom=milter.msginfo["mailfrom"],
+                    recipients=milter.msginfo["rcpts"],
+                    template_vars=milter.msginfo["vars"],
+                    logger=logger)
+
+
+class Notify:
+    NOTIFICATION_TYPES = {
+        "email": EMailNotification}
+
+    def __init__(self, cfg, local_addrs, debug):
+        self.cfg = cfg
+        self.logger = logging.getLogger(cfg["name"])
+        self.logger.setLevel(cfg.get_loglevel(debug))
+
+        nodification_type = cfg["options"]["type"]
+        del cfg["options"]["type"]
+        cfg["options"]["pretend"] = cfg["pretend"]
+        self._notification = self.NOTIFICATION_TYPES[nodification_type](
+            **cfg["options"])
+        self._headersonly = self._notification._headersonly
+
+    def __str__(self):
+        cfg = []
+        for key, value in self.cfg["options"].items():
+            cfg.append(f"{key}={value}")
+        class_name = type(self._notification).__name__
+        return f"{class_name}(" + ", ".join(cfg) + ")"
+
+    def get_notification(self):
+        return self._notification
+
+    def execute(self, milter):
+        logger = CustomLogger(
+            self.logger, {"name": self.cfg["name"], "qid": milter.qid})
+        self._notification.execute(milter, logger)

pyquarantine/quarantines.py

@@ -1,309 +0,0 @@
-# PyQuarantine-Milter is free software: you can redistribute it and/or modify
-# it under the terms of the GNU General Public License as published by
-# the Free Software Foundation, either version 3 of the License, or
-# (at your option) any later version.
-#
-# PyQuarantine-Milter is distributed in the hope that it will be useful,
-# but WITHOUT ANY WARRANTY; without even the implied warranty of
-# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
-# GNU General Public License for more details.
-#
-# You should have received a copy of the GNU General Public License
-# along with PyQuarantineMilter.  If not, see <http://www.gnu.org/licenses/>.
-#
-
-import json
-import logging
-import os
-
-from calendar import timegm
-from datetime import datetime
-from glob import glob
-from shutil import copyfileobj
-from time import gmtime
-
-from pyquarantine import mailer
-
-
-class BaseQuarantine(object):
-    "Quarantine base class"
-
-    def __init__(self, global_config, config, configtest=False):
-        self.name = config["name"]
-        self.global_config = global_config
-        self.config = config
-        self.logger = logging.getLogger(__name__)
-
-    def add(self, queueid, mailfrom, recipients, headers,
-            fp, subgroups=None, named_subgroups=None):
-        "Add email to quarantine."
-        fp.seek(0)
-        return ""
-
-    def find(self, mailfrom=None, recipients=None, older_than=None):
-        "Find emails in quarantine."
-        return
-
-    def get_metadata(self, quarantine_id):
-        "Return metadata of quarantined email."
-        return
-
-    def delete(self, quarantine_id, recipient=None):
-        "Delete email from quarantine."
-        return
-
-    def notify(self, quarantine_id, recipient=None):
-        "Notify recipient about email in quarantine."
-        if not self.config["notification_obj"]:
-            raise RuntimeError(
-                "notification type is set to None, unable to send notifications")
-        return
-
-    def release(self, quarantine_id, recipient=None):
-        "Release email from quarantine."
-        return
-
-
-class FileQuarantine(BaseQuarantine):
-    "Quarantine class to store mails on filesystem."
-
-    def __init__(self, global_config, config, configtest=False):
-        super(FileQuarantine, self).__init__(global_config, config, configtest)
-
-        # check if mandatory options are present in config
-        for option in ["quarantine_directory"]:
-            if option not in self.config.keys() and option in self.global_config.keys():
-                self.config[option] = self.global_config[option]
-            if option not in self.config.keys():
-                raise RuntimeError(
-                    "mandatory option '{}' not present in config section '{}' or 'global'".format(
-                        option, self.name))
-        self.directory = self.config["quarantine_directory"]
-
-        # check if quarantine directory exists and is writable
-        if not os.path.isdir(self.directory) or not os.access(
-                self.directory, os.W_OK):
-            raise RuntimeError(
-                "file quarantine directory '{}' does not exist or is not writable".format(
-                    self.directory))
-        self._metadata_suffix = ".metadata"
-
-    def _save_datafile(self, quarantine_id, fp):
-        datafile = os.path.join(self.directory, quarantine_id)
-        try:
-            with open(datafile, "wb") as f:
-                copyfileobj(fp, f)
-        except IOError as e:
-            raise RuntimeError("unable save data file: {}".format(e))
-
-    def _save_metafile(self, quarantine_id, metadata):
-        metafile = os.path.join(
-            self.directory, "{}{}".format(
-                quarantine_id, self._metadata_suffix))
-        try:
-            with open(metafile, "w") as f:
-                json.dump(metadata, f, indent=2)
-        except IOError as e:
-            raise RuntimeError("unable to save metadata file: {}".format(e))
-
-    def _remove(self, quarantine_id):
-        datafile = os.path.join(self.directory, quarantine_id)
-        metafile = "{}{}".format(datafile, self._metadata_suffix)
-
-        try:
-            os.remove(metafile)
-        except IOError as e:
-            raise RuntimeError("unable to remove metadata file: {}".format(e))
-
-        try:
-            os.remove(datafile)
-        except IOError as e:
-            raise RuntimeError("unable to remove data file: {}".format(e))
-
-    def add(self, queueid, mailfrom, recipients, headers,
-            fp, subgroups=None, named_subgroups=None):
-        "Add email to file quarantine and return quarantine-id."
-        super(
-            FileQuarantine,
-            self).add(
-            queueid,
-            mailfrom,
-            recipients,
-            headers,
-            fp,
-            subgroups,
-            named_subgroups)
-        quarantine_id = "{}_{}".format(
-            datetime.now().strftime("%Y%m%d%H%M%S"), queueid)
-
-        # save mail
-        self._save_datafile(quarantine_id, fp)
-
-        # save metadata
-        metadata = {
-            "mailfrom": mailfrom,
-            "recipients": recipients,
-            "headers": headers,
-            "date": timegm(gmtime()),
-            "queue_id": queueid,
-            "subgroups": subgroups,
-            "named_subgroups": named_subgroups
-        }
-        try:
-            self._save_metafile(quarantine_id, metadata)
-        except RuntimeError as e:
-            datafile = os.path.join(self.directory, quarantine_id)
-            os.remove(datafile)
-            raise e
-
-        return quarantine_id
-
-    def get_metadata(self, quarantine_id):
-        "Return metadata of quarantined email."
-        super(FileQuarantine, self).get_metadata(quarantine_id)
-
-        metafile = os.path.join(
-            self.directory, "{}{}".format(
-                quarantine_id, self._metadata_suffix))
-        if not os.path.isfile(metafile):
-            raise RuntimeError(
-                "invalid quarantine id '{}'".format(quarantine_id))
-
-        try:
-            with open(metafile, "r") as f:
-                metadata = json.load(f)
-        except IOError as e:
-            raise RuntimeError("unable to read metadata file: {}".format(e))
-        except json.JSONDecodeError as e:
-            raise RuntimeError(
-                "invalid meta file '{}': {}".format(
-                    metafile, e))
-
-        return metadata
-
-    def find(self, mailfrom=None, recipients=None, older_than=None):
-        "Find emails in quarantine."
-        super(FileQuarantine, self).find(mailfrom, recipients, older_than)
-        if isinstance(mailfrom, str):
-            mailfrom = [mailfrom]
-        if isinstance(recipients, str):
-            recipients = [recipients]
-
-        emails = {}
-        metafiles = glob(os.path.join(
-            self.directory, "*{}".format(self._metadata_suffix)))
-        for metafile in metafiles:
-            if not os.path.isfile(metafile):
-                continue
-
-            quarantine_id = os.path.basename(
-                metafile[:-len(self._metadata_suffix)])
-            metadata = self.get_metadata(quarantine_id)
-            if older_than is not None:
-                if timegm(gmtime()) - metadata["date"] < (older_than * 86400):
-                    continue
-
-            if mailfrom is not None:
-                if metadata["mailfrom"] not in mailfrom:
-                    continue
-
-            if recipients is not None:
-                if len(recipients) == 1 and \
-                        recipients[0] not in metadata["recipients"]:
-                    continue
-                elif len(set(recipients + metadata["recipients"])) == len(recipients + metadata["recipients"]):
-                    continue
-
-            emails[quarantine_id] = metadata
-
-        return emails
-
-    def delete(self, quarantine_id, recipient=None):
-        "Delete email in quarantine."
-        super(FileQuarantine, self).delete(quarantine_id, recipient)
-
-        try:
-            metadata = self.get_metadata(quarantine_id)
-        except RuntimeError as e:
-            raise RuntimeError("unable to delete email: {}".format(e))
-
-        if recipient is None:
-            self._remove(quarantine_id)
-        else:
-            if recipient not in metadata["recipients"]:
-                raise RuntimeError("invalid recipient '{}'".format(recipient))
-
-            metadata["recipients"].remove(recipient)
-            if not metadata["recipients"]:
-                self._remove(quarantine_id)
-            else:
-                self._save_metafile(quarantine_id, metadata)
-
-    def notify(self, quarantine_id, recipient=None):
-        "Notify recipient about email in quarantine."
-        super(FileQuarantine, self).notify(quarantine_id, recipient)
-
-        try:
-            metadata = self.get_metadata(quarantine_id)
-        except RuntimeError as e:
-            raise RuntimeError("unable to release email: {}".format(e))
-
-        if recipient is not None:
-            if recipient not in metadata["recipients"]:
-                raise RuntimeError("invalid recipient '{}'".format(recipient))
-            recipients = [recipient]
-        else:
-            recipients = metadata["recipients"]
-
-        datafile = os.path.join(self.directory, quarantine_id)
-        try:
-            with open(datafile, "rb") as fp:
-                self.config["notification_obj"].notify(
-                    metadata["queue_id"], quarantine_id, metadata["mailfrom"],
-                    recipients, metadata["headers"], fp,
-                    metadata["subgroups"], metadata["named_subgroups"],
-                    synchronous=True)
-        except IOError as e:
-            raise RuntimeError
-
-    def release(self, quarantine_id, recipient=None):
-        "Release email from quarantine."
-        super(FileQuarantine, self).release(quarantine_id, recipient)
-
-        try:
-            metadata = self.get_metadata(quarantine_id)
-        except RuntimeError as e:
-            raise RuntimeError("unable to release email: {}".format(e))
-
-        if recipient is not None:
-            if recipient not in metadata["recipients"]:
-                raise RuntimeError("invalid recipient '{}'".format(recipient))
-            recipients = [recipient]
-        else:
-            recipients = metadata["recipients"]
-
-        datafile = os.path.join(self.directory, quarantine_id)
-        try:
-            with open(datafile, "rb") as f:
-                mail = f.read()
-        except IOError as e:
-            raise RuntimeError("unable to read data file: {}".format(e))
-
-        for recipient in recipients:
-            try:
-                mailer.smtp_send(
-                    self.config["smtp_host"],
-                    self.config["smtp_port"],
-                    metadata["mailfrom"],
-                    recipient,
-                    mail)
-            except Exception as e:
-                raise RuntimeError(
-                    "error while sending email to '{}': {}".format(
-                        recipient, e))
-
-            self.delete(quarantine_id, recipient)
-
-
-# list of quarantine types and their related quarantine classes
-TYPES = {"file": FileQuarantine}

pyquarantine/rule.py

@@ -0,0 +1,59 @@
+# pyquarantine is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# pyquarantine is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+__all__ = ["Rule"]
+
+from pyquarantine.action import Action
+from pyquarantine.conditions import Conditions
+
+
+class Rule:
+    """
+    Rule to implement multiple actions on emails.
+    """
+    def __init__(self, cfg, local_addrs, debug):
+        self.cfg = cfg
+
+        self.conditions = cfg["conditions"] if "conditions" in cfg else None
+        if self.conditions is not None:
+            self.conditions["name"] = f"{cfg['name']}: condition"
+            self.conditions["loglevel"] = cfg["loglevel"]
+            self.conditions = Conditions(self.conditions, local_addrs, debug)
+
+        self.actions = []
+        for idx, action_cfg in enumerate(cfg["actions"]):
+            action_cfg["name"] = f"{cfg['name']}: {action_cfg['name']}"
+            self.actions.append(Action(action_cfg, local_addrs, debug))
+
+    def __str__(self):
+        cfg = []
+        for key in ["name", "loglevel", "pretend"]:
+            value = self.cfg[key]
+            cfg.append(f"{key}={value}")
+        if self.conditions is not None:
+            cfg.append(f"conditions={self.conditions}")
+        actions = []
+        for action in self.actions:
+            actions.append(str(action))
+        cfg.append("actions=[" + ", ".join(actions) + "]")
+        return "Rule(" + ", ".join(cfg) + ")"
+
+    def execute(self, milter):
+        """Execute all actions of this rule."""
+        if self.conditions is None or \
+                self.conditions.match(milter):
+            for action in self.actions:
+                milter_action = action.execute(milter)
+                if milter_action is not None:
+                    return milter_action

pyquarantine/run.py

@@ -1,129 +1,166 @@
-#!/usr/bin/env python
+# pyquarantine is free software: you can redistribute it and/or modify
-#
-# PyQuarantine-Milter is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
-# PyQuarantine-Milter is distributed in the hope that it will be useful,
+# pyquarantine is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
-# along with PyQuarantineMilter.  If not, see <http://www.gnu.org/licenses/>.
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+__all__ = ["main"]
+
 import Milter
 import argparse
 import logging
 import logging.handlers
 import sys
 
-import pyquarantine
+from pyquarantine._install import install, uninstall
+from pyquarantine import mailer
+from pyquarantine import QuarantineMilter
+from pyquarantine import __version__ as version
+from pyquarantine.config import get_milter_config
 
-from pyquarantine.version import __version__ as version
 
 def main():
-    "Run PyQuarantine-Milter."
+    python_version = ".".join([str(v) for v in sys.version_info[0:3]])
-    # parse command line
+    python_version = f"{python_version}-{sys.version_info[3]}"
+
+    "Run pyquarantine."
     parser = argparse.ArgumentParser(
-        description="PyQuarantine milter daemon",
+        description="pyquarantine-milter daemon",
         formatter_class=lambda prog: argparse.HelpFormatter(
             prog, max_help_position=45, width=140))
     parser.add_argument(
-        "-c", "--config",
+        "-c", "--config", help="Config file to read.",
-        help="List of config files to read.",
+        default="/etc/pyquarantine/pyquarantine.conf")
-        nargs="+",
-        default=pyquarantine.QuarantineMilter.get_configfiles())
     parser.add_argument(
-        "-s", "--socket",
+        "-s",
+        "--socket",
         help="Socket used to communicate with the MTA.",
-        required=True)
+        default="")
     parser.add_argument(
-        "-d", "--debug",
+        "-d",
+        "--debug",
         help="Log debugging messages.",
         action="store_true")
-    parser.add_argument(
+
-        "-t", "--test",
+    exclusive = parser.add_mutually_exclusive_group()
-        help="Check configuration.",
+    exclusive.add_argument(
-        action="store_true")
-    parser.add_argument(
         "-v", "--version",
         help="Print version.",
         action="version",
-        version="%(prog)s ({})".format(version))
+        version=f"%(prog)s {version} (python {python_version})")
+    exclusive.add_argument(
+        "-i",
+        "--install",
+        help="install service files and config",
+        action="store_true")
+    exclusive.add_argument(
+        "-u",
+        "--uninstall",
+        help="uninstall service files and unmodified config",
+        action="store_true")
+    exclusive.add_argument(
+        "-t",
+        "--test",
+        help="Check configuration.",
+        action="store_true")
+
     args = parser.parse_args()
 
-    # setup logging
-    loglevel = logging.INFO
-    logname = "pyquarantine-milter"
-    syslog_name = logname
-    if args.debug:
-        loglevel = logging.DEBUG
-        logname = "{}[%(name)s]".format(logname)
-        syslog_name = "{}: [%(name)s] %(levelname)s".format(syslog_name)
-
-    # set config files for milter class
-    pyquarantine.QuarantineMilter.set_configfiles(args.config)
     root_logger = logging.getLogger()
-    root_logger.setLevel(loglevel)
+    root_logger.setLevel(logging.DEBUG)
 
     # setup console log
     stdouthandler = logging.StreamHandler(sys.stdout)
-    stdouthandler.setLevel(logging.DEBUG)
+    formatter = logging.Formatter("%(levelname)s: %(message)s")
-    formatter = logging.Formatter("%(message)s".format(logname))
     stdouthandler.setFormatter(formatter)
     root_logger.addHandler(stdouthandler)
+
     logger = logging.getLogger(__name__)
-    if args.test:
+
+    if not args.debug:
+        logger.setLevel(logging.INFO)
+
+    name = "pyquarantine"
+    if args.install:
+        sys.exit(install(name))
+
+    if args.uninstall:
+        sys.exit(uninstall(name))
+
     try:
-            pyquarantine.generate_milter_config(args.test)
+        logger.debug("read milter configuration")
-            print("Configuration ok")
+        cfg = get_milter_config(args.config)
-        except RuntimeError as e:
+        logger.setLevel(cfg.get_loglevel(args.debug))
-            logger.error(e)
+
-            sys.exit(255)
+        if args.socket:
+            socket = args.socket
+        elif "socket" in cfg:
+            socket = cfg["socket"]
         else:
+            raise RuntimeError(
+                "listening socket is neither specified on the command line "
+                "nor in the configuration file")
+
+        if not cfg["rules"]:
+            raise RuntimeError("no rules configured")
+
+        for rule in cfg["rules"]:
+            if not rule["actions"]:
+                raise RuntimeError(
+                    f"{rule['name']}: no actions configured")
+        QuarantineMilter.set_config(cfg, args.debug)
+
+    except (RuntimeError, AssertionError) as e:
+        logger.error(f"config error: {e}")
+        sys.exit(255)
+
+    if args.test:
+        print("Configuration OK")
         sys.exit(0)
-    formatter = logging.Formatter(
+
-        "%(asctime)s {}: [%(levelname)s] %(message)s".format(logname),
+    # setup console log for runtime
-        datefmt="%Y-%m-%d %H:%M:%S")
+    formatter = logging.Formatter("%(asctime)s - %(levelname)s: %(message)s")
     stdouthandler.setFormatter(formatter)
+    stdouthandler.setLevel(logging.DEBUG)
 
     # setup syslog
     sysloghandler = logging.handlers.SysLogHandler(
         address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_MAIL)
-    sysloghandler.setLevel(loglevel)
+    sysloghandler.setFormatter(
-    formatter = logging.Formatter("{}: %(message)s".format(syslog_name))
+        logging.Formatter(f"{name}[%(process)d]: %(message)s"))
-    sysloghandler.setFormatter(formatter)
     root_logger.addHandler(sysloghandler)
 
-    logger.info("PyQuarantine-Milter starting")
+    logger.info("milter starting")
-    try:
-        # generate milter config
-        global_config, config = pyquarantine.generate_milter_config()
-    except RuntimeError as e:
-        logger.error(e)
-        sys.exit(255)
 
-    pyquarantine.QuarantineMilter.global_config = global_config
+    # register milter factory class
-    pyquarantine.QuarantineMilter.config = config
+    Milter.factory = QuarantineMilter
-
-    # register to have the Milter factory create instances of your class:
-    Milter.factory = pyquarantine.QuarantineMilter
     Milter.set_exception_policy(Milter.TEMPFAIL)
-    # Milter.set_flags(0)       # tell sendmail which features we use
 
-    # run milter
+    if args.debug:
+        Milter.setdbg(1)
+
+    # increase the recursion limit so that BeautifulSoup can
+    # parse larger html content
+    sys.setrecursionlimit(4000)
+
     rc = 0
     try:
-        Milter.runmilter("pyquarantine-milter", socketname=args.socket,
+        Milter.runmilter("pyquarantine", socketname=socket, timeout=600)
-                         timeout=300)
     except Milter.milter.error as e:
         logger.error(e)
         rc = 255
-    pyquarantine.mailer.queue.put(None)
+
-    logger.info("PyQuarantine-Milter terminated")
+    mailer.queue.put(None)
+    logger.info("milter stopped")
+
     sys.exit(rc)
 
 

pyquarantine/storage.py

@@ -0,0 +1,567 @@
+# pyquarantine is free software: you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation, either version 3 of the License, or
+# (at your option) any later version.
+#
+# pyquarantine is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+__all__ = [
+    "BaseMailStorage",
+    "FileMailStorage",
+    "Store",
+    "Quarantine"]
+
+import json
+import logging
+import os
+
+from calendar import timegm
+from datetime import datetime
+from email import message_from_bytes
+from email.policy import SMTP
+from glob import glob
+from time import gmtime
+
+from pyquarantine import mailer
+from pyquarantine.base import CustomLogger, MilterMessage
+from pyquarantine.conditions import Conditions
+from pyquarantine.config import ActionConfig
+from pyquarantine.notify import Notify
+
+
+class BaseMailStorage:
+    "Mail storage base class"
+    _headersonly = True
+
+    def __init__(self, original=False, metadata=False, metavar=None,
+                 pretend=False):
+        self.original = original
+        self.metadata = metadata
+        self.metavar = metavar
+        self.pretend = False
+
+    def add(self, data, qid, mailfrom, recipients, subject, variables):
+        "Add message to storage."
+        return ("", "")
+
+    def execute(self, milter, logger):
+        return
+
+    def find(self, mailfrom=None, recipients=None, older_than=None):
+        "Find messages in storage."
+        return
+
+    def get_metadata(self, storage_id):
+        "Return metadata of message in storage."
+        return
+
+    def delete(self, storage_id, recipients=None):
+        "Delete message from storage."
+        return
+
+    def get_mail(self, storage_id):
+        "Return message and metadata."
+        return
+
+
+class FileMailStorage(BaseMailStorage):
+    "Storage class to store mails on filesystem."
+    _headersonly = False
+
+    def __init__(self, directory, original=False, metadata=False, metavar=None,
+                 mode=None, pretend=False):
+        super().__init__(original, metadata, metavar, pretend)
+        # check if directory exists and is writable
+        if not os.path.isdir(directory) or \
+                not os.access(directory, os.R_OK):
+            raise RuntimeError(
+                f"directory '{directory}' does not exist or is "
+                f"not readable")
+        self.directory = directory
+        try:
+            self.mode = int(mode, 8) if mode is not None else None
+            if self.mode is not None and self.mode > 511:
+                raise ValueError
+        except ValueError:
+            raise RuntimeError(f"invalid mode '{mode}'")
+
+        self._metadata_suffix = ".metadata"
+
+    def __str__(self):
+        cfg = []
+        cfg.append(f"metadata={self.metadata}")
+        cfg.append(f"metavar={self.metavar}")
+        cfg.append(f"pretend={self.pretend}")
+        cfg.append(f"directory={self.directory}")
+        cfg.append(f"original={self.original}")
+        return "FileMailStorage(" + ", ".join(cfg) + ")"
+
+    def get_storageid(self, qid):
+        timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
+        return f"{timestamp}_{qid}"
+
+    def _get_file_paths(self, storage_id):
+        datafile = os.path.join(self.directory, storage_id)
+        metafile = f"{datafile}{self._metadata_suffix}"
+        return metafile, datafile
+
+    def _save_datafile(self, datafile, data):
+        try:
+            if self.mode is None:
+                with open(datafile, "wb") as f:
+                    f.write(data)
+            else:
+                umask = os.umask(0)
+                with open(
+                        os.open(datafile,
+                                os.O_CREAT | os.O_WRONLY | os.O_TRUNC,
+                                self.mode),
+                        "wb") as f:
+                    f.write(data)
+                os.umask(umask)
+
+        except IOError as e:
+            raise RuntimeError(f"unable save data file: {e}")
+
+    def _save_metafile(self, metafile, metadata):
+        try:
+            if self.mode is None:
+                with open(metafile, "w") as f:
+                    json.dump(metadata, f, indent=2)
+            else:
+                umask = os.umask(0)
+                with open(
+                        os.open(metafile,
+                                os.O_CREAT | os.O_WRONLY | os.O_TRUNC,
+                                self.mode),
+                        "w") as f:
+                    json.dump(metadata, f, indent=2)
+                os.umask(umask)
+
+        except IOError as e:
+            raise RuntimeError(f"unable to save metadata file: {e}")
+
+    def _remove(self, storage_id):
+        metafile, datafile = self._get_file_paths(storage_id)
+
+        try:
+            if self.metadata:
+                os.remove(metafile)
+
+            os.remove(datafile)
+        except IOError as e:
+            raise RuntimeError(f"unable to remove file: {e}")
+
+    def add(self, data, qid, mailfrom, recipients, subject, variables, logger):
+        "Add message to file storage and return storage id."
+        super().add(data, qid, mailfrom, recipients, subject, variables)
+
+        storage_id = self.get_storageid(qid)
+        metafile, datafile = self._get_file_paths(storage_id)
+
+        if self.metavar:
+            variables[f"{self.metavar}_ID"] = storage_id
+            variables[f"{self.metavar}_DATAFILE"] = datafile
+            if self.metadata:
+                variables[f"{self.metavar}_METAFILE"] = metafile
+
+        if self.pretend:
+            return
+
+        # save mail
+        logger.debug(f"save message to {datafile}")
+        self._save_datafile(datafile, data)
+        logger.info(f"stored message with id {storage_id}")
+
+        if not self.metadata:
+            return storage_id, None, datafile
+
+        # save metadata
+        metadata = {
+            "mailfrom": mailfrom,
+            "recipients": recipients,
+            "subject": subject,
+            "timestamp": timegm(gmtime()),
+            "queue_id": qid,
+            "vars": variables}
+
+        try:
+            logger.debug(f"save metadata to {metafile}")
+            self._save_metafile(metafile, metadata)
+        except RuntimeError as e:
+            os.remove(datafile)
+            raise e
+
+    def execute(self, milter, logger):
+        if self.original:
+            milter.fp.seek(0)
+            data = milter.fp.read
+            mailfrom = milter.mailfrom
+            recipients = list(milter.rcpts)
+            # getting the subject is the only operation that needs any
+            # parsing of the message, catch all exceptions here
+            try:
+                subject = milter.msg["subject"] or ""
+            except Exception:
+                subject = ""
+        else:
+            data = milter.msg_as_bytes
+            mailfrom = milter.msginfo["mailfrom"]
+            recipients = list(milter.msginfo["rcpts"])
+            subject = milter.msg["subject"] or ""
+
+        self.add(data(), milter.qid, mailfrom, recipients, subject,
+                 milter.msginfo["vars"], logger)
+
+    def get_metadata(self, storage_id):
+        "Return metadata of message in storage."
+        super().get_metadata(storage_id)
+
+        if not self.metadata:
+            return None
+
+        metafile, _ = self._get_file_paths(storage_id)
+        if not os.path.isfile(metafile):
+            raise RuntimeError(
+                f"invalid storage id '{storage_id}'")
+
+        try:
+            with open(metafile, "r") as f:
+                metadata = json.load(f)
+        except IOError as e:
+            raise RuntimeError(f"unable to read metadata file: {e}")
+        except json.JSONDecodeError as e:
+            raise RuntimeError(
+                f"invalid metafile '{metafile}': {e}")
+
+        # convert metafile structure, this can be removed in the future
+        converted = False
+        if "subject" not in metadata:
+            try:
+                metadata["subject"] = metadata["headers"]["subject"]
+            except KeyError:
+                metadata["subject"] = ""
+            converted = True
+        if "timestamp" not in metadata:
+            try:
+                metadata["timestamp"] = metadata["date"]
+            except KeyError:
+                metadata["timestamp"] = 0
+            converted = True
+        if "vars" not in metadata:
+            try:
+                metadata["vars"] = metadata["named_subgroups"]
+            except KeyError:
+                metadata["vars"] = {}
+            converted = True
+        if "headers" in metadata:
+            del metadata["headers"]
+            converted = True
+        if "date" in metadata:
+            del metadata["date"]
+            converted = True
+        if "named_subgroups" in metadata:
+            del metadata["named_subgroups"]
+            converted = True
+        if "subgroups" in metadata:
+            del metadata["subgroups"]
+            converted = True
+        if converted:
+            self._save_metafile(metafile, metadata)
+
+        return metadata
+
+    def find(self, mailfrom=None, recipients=None, older_than=None):
+        "Find messages in storage."
+        super().find(mailfrom, recipients, older_than)
+        if isinstance(mailfrom, str):
+            mailfrom = [mailfrom]
+        if isinstance(recipients, str):
+            recipients = [recipients]
+
+        if not self.metadata:
+            return {}
+
+        msgs = {}
+        metafiles = glob(os.path.join(
+            self.directory, f"*{self._metadata_suffix}"))
+        for metafile in metafiles:
+            if not os.path.isfile(metafile):
+                continue
+
+            storage_id = os.path.basename(
+                metafile[:-len(self._metadata_suffix)])
+            metadata = self.get_metadata(storage_id)
+            if older_than is not None:
+                age = timegm(gmtime()) - metadata["timestamp"]
+                if age < (older_than * 86400):
+                    continue
+
+            if mailfrom is not None:
+                if metadata["mailfrom"] not in mailfrom:
+                    continue
+
+            if recipients is not None:
+                if len(recipients) == 1 and \
+                        recipients[0] not in metadata["recipients"]:
+                    continue
+                elif len(set(recipients + metadata["recipients"])) == \
+                        len(recipients + metadata["recipients"]):
+                    continue
+
+            msgs[storage_id] = metadata
+
+        return msgs
+
+    def delete(self, storage_id, recipients=None):
+        "Delete message from storage."
+        super().delete(storage_id, recipients)
+        if not recipients or not self.metadata:
+            self._remove(storage_id)
+            return
+
+        try:
+            metadata = self.get_metadata(storage_id)
+        except RuntimeError as e:
+            raise RuntimeError(f"unable to delete message: {e}")
+
+        metafile, _ = self._get_file_paths(storage_id)
+
+        if type(recipients) == str:
+            recipients = [recipients]
+
+        for recipient in recipients:
+            if recipient not in metadata["recipients"]:
+                raise RuntimeError(f"invalid recipient '{recipient}'")
+            metadata["recipients"].remove(recipient)
+            if not metadata["recipients"]:
+                self._remove(storage_id)
+            else:
+                self._save_metafile(metafile, metadata)
+
+    def get_mail_bytes(self, storage_id):
+        _, datafile = self._get_file_paths(storage_id)
+        try:
+            with open(datafile, "rb") as fh:
+                data = fh.read()
+        except IOError as e:
+            raise RuntimeError(f"unable to open data file: {e}")
+        return data
+
+    def get_mail(self, storage_id):
+        super().get_mail(storage_id)
+
+        metadata = self.get_metadata(storage_id)
+        msg = message_from_bytes(
+            self.get_mail_bytes(storage_id),
+            _class=MilterMessage,
+            policy=SMTP.clone(refold_source='none'))
+        return (metadata, msg)
+
+
+class Store:
+    STORAGE_TYPES = {
+        "file": FileMailStorage}
+
+    def __init__(self, cfg, local_addrs, debug):
+        self.cfg = cfg
+        self.logger = logging.getLogger(cfg["name"])
+        self.logger.setLevel(cfg.get_loglevel(debug))
+
+        storage_type = cfg["options"]["type"]
+        del cfg["options"]["type"]
+        cfg["options"]["pretend"] = cfg["pretend"]
+        self._storage = self.STORAGE_TYPES[storage_type](
+            **cfg["options"])
+        self._headersonly = self._storage._headersonly
+
+    def __str__(self):
+        cfg = []
+        for key, value in self.cfg["options"].items():
+            cfg.append(f"{key}={value}")
+        class_name = type(self._storage).__name__
+        return f"{class_name}(" + ", ".join(cfg) + ")"
+
+    def get_storage(self):
+        return self._storage
+
+    def execute(self, milter):
+        logger = CustomLogger(
+            self.logger, {"name": self.cfg["name"], "qid": milter.qid})
+        self._storage.execute(milter, logger)
+
+
+class Quarantine:
+    "Quarantine class."
+    _headersonly = False
+
+    def __init__(self, cfg, local_addrs, debug):
+        self.cfg = cfg
+        self.logger = logging.getLogger(cfg["name"])
+        self.logger.setLevel(cfg.get_loglevel(debug))
+
+        storage_cfg = ActionConfig({
+            "name": cfg["name"],
+            "loglevel": cfg["loglevel"],
+            "pretend": cfg["pretend"],
+            "type": "store",
+            "options": cfg["options"]["store"].get_config()})
+        self._storage = Store(storage_cfg, local_addrs, debug)
+
+        self.smtp_host = cfg["options"]["smtp_host"]
+        self.smtp_port = cfg["options"]["smtp_port"]
+
+        self._notification = None
+        if "notify" in cfg["options"]:
+            notify_cfg = ActionConfig({
+                "name": cfg["name"],
+                "loglevel": cfg["loglevel"],
+                "pretend": cfg["pretend"],
+                "type": "notify",
+                "options": cfg["options"]["notify"].get_config()})
+            self._notification = Notify(notify_cfg, local_addrs, debug)
+
+        self._whitelist = None
+        if "whitelist" in cfg["options"]:
+            whitelist_cfg = cfg["options"]["whitelist"]
+            whitelist_cfg["name"] = cfg["name"]
+            whitelist_cfg["loglevel"] = cfg["loglevel"]
+            self._whitelist = Conditions(
+                whitelist_cfg,
+                local_addrs=[],
+                debug=debug)
+
+        self._milter_action = None
+        if "milter_action" in cfg["options"]:
+            self._milter_action = cfg["options"]["milter_action"].upper()
+            assert self._milter_action in ["ACCEPT", "REJECT", "DISCARD"], \
+                f"invalid milter_action '{cfg['args']['milter_action']}'"
+
+        self._reason = None
+        if self._milter_action == "REJECT":
+            if "reject_reason" in cfg["options"]:
+                self._reason = cfg["options"]["reject_reason"]
+            else:
+                self._reason = "Message rejected"
+
+    def __str__(self):
+        cfg = []
+        cfg.append(f"store={str(self._storage)}")
+        if self._notification is not None:
+            cfg.append(f"notify={str(self._notification)}")
+        if self._whitelist is not None:
+            cfg.append(f"whitelist={str(self._whitelist)}")
+        for key in ["milter_action", "reject_reason"]:
+            if key not in self.cfg["options"]:
+                continue
+            value = self.cfg["options"][key]
+            cfg.append(f"{key}={value}")
+        class_name = type(self).__name__
+        return f"{class_name}(" + ", ".join(cfg) + ")"
+
+    @property
+    def name(self):
+        return self.cfg["name"]
+
+    @property
+    def storage(self):
+        return self._storage.get_storage()
+
+    @property
+    def notification(self):
+        if self._notification is None:
+            return None
+        return self._notification.get_notification()
+
+    @property
+    def whitelist(self):
+        if self._whitelist is None:
+            return None
+        return self._whitelist.get_whitelist()
+
+    @property
+    def milter_action(self):
+        return self._milter_action
+
+    def notify(self, storage_id, recipient=None):
+        "Notify recipient about message in storage."
+        if not self._notification:
+            raise RuntimeError(
+                "notification not defined, "
+                "unable to send notification")
+        metadata, msg = self.storage.get_mail(storage_id)
+
+        if recipient is not None:
+            if recipient not in metadata["recipients"]:
+                raise RuntimeError(f"invalid recipient '{recipient}'")
+            recipients = [recipient]
+        else:
+            recipients = metadata["recipients"]
+
+        self.notification.notify(msg, metadata["queue_id"],
+                                 metadata["mailfrom"], recipients,
+                                 self.logger, metadata["vars"],
+                                 synchronous=True)
+
+    def release(self, storage_id, recipients=None):
+        metadata, msg = self.storage.get_mail(storage_id)
+        if recipients and type(recipients) == str:
+            recipients = [recipients]
+        else:
+            recipients = metadata["recipients"]
+
+        for recipient in recipients:
+            if recipient not in metadata["recipients"]:
+                raise RuntimeError(f"invalid recipient '{recipient}'")
+            try:
+                mailer.smtp_send(
+                    self.smtp_host,
+                    self.smtp_port,
+                    metadata["mailfrom"],
+                    recipient,
+                    msg.as_string())
+
+            except Exception as e:
+                raise RuntimeError(
+                    f"error while sending message to '{recipient}': {e}")
+            self.storage.delete(storage_id, recipient)
+
+        return recipients
+
+    def execute(self, milter):
+        logger = CustomLogger(
+            self.logger, {"name": self.cfg["name"], "qid": milter.qid})
+        rcpts = milter.msginfo["rcpts"]
+        wl_rcpts = []
+        if self._whitelist:
+            wl_rcpts = self._whitelist.get_wl_rcpts(
+                milter.msginfo["mailfrom"], rcpts, logger)
+            if wl_rcpts:
+                logger.info(f"whitelisted recipients: {wl_rcpts}")
+            rcpts = [rcpt for rcpt in rcpts if rcpt not in wl_rcpts]
+            if not rcpts:
+                # all recipients whitelisted
+                return
+            milter.msginfo["rcpts"] = rcpts.copy()
+
+        if self._milter_action in ["REJECT", "DISCARD"]:
+            logger.info(f"quarantine message for {rcpts}")
+
+        self._storage.execute(milter)
+
+        if self._notification is not None:
+            self._notification.execute(milter)
+
+        milter.msginfo["rcpts"].extend(wl_rcpts)
+
+        if self._milter_action is not None:
+            milter.delrcpt(rcpts)
+            if not milter.msginfo["rcpts"]:
+                return (self._milter_action, self._reason)

pyquarantine/version.py

@@ -1 +0,0 @@
-__version__ = "0.0.6"

pyquarantine/whitelist.py

@@ -1,37 +1,50 @@
-# PyQuarantine-Milter is free software: you can redistribute it and/or modify
+# pyquarantine is free software: you can redistribute it and/or modify
 # it under the terms of the GNU General Public License as published by
 # the Free Software Foundation, either version 3 of the License, or
 # (at your option) any later version.
 #
-# PyQuarantine-Milter is distributed in the hope that it will be useful,
+# pyquarantine is distributed in the hope that it will be useful,
 # but WITHOUT ANY WARRANTY; without even the implied warranty of
 # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 # GNU General Public License for more details.
 #
 # You should have received a copy of the GNU General Public License
-# along with PyQuarantineMilter.  If not, see <http://www.gnu.org/licenses/>.
+# along with pyquarantine.  If not, see <http://www.gnu.org/licenses/>.
 #
 
+__all__ = [
+    "DatabaseWhitelist",
+    "WhitelistBase"]
+
 import logging
 import peewee
 import re
-import sys
 
 from datetime import datetime
 from playhouse.db_url import connect
 
 
-class WhitelistBase(object):
+class WhitelistBase:
     "Whitelist base class"
+    def __init__(self, cfg, debug):
+        self.cfg = cfg
+        self.logger = logging.getLogger(cfg["name"])
+        self.logger.setLevel(cfg.get_loglevel(debug))
+
+        peewee_logger = logging.getLogger("peewee")
+        peewee_logger.setLevel(cfg.get_loglevel(debug))
 
-    def __init__(self, global_config, config, configtest=False):
-        self.global_config = global_config
-        self.config = config
-        self.configtest = configtest
-        self.name = config["name"]
-        self.logger = logging.getLogger(__name__)
         self.valid_entry_regex = re.compile(
-            r"^[a-zA-Z0-9_.+-]*?(@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+)?$")
+            r"^[a-zA-Z0-9_.=+-]*?(@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+)?$")
+        self.batv_regex = re.compile(
+            r"^prvs=[0-9]{3}[0-9A-Fa-f]{6,7}=(?P<LEFT_PART>.+?)@")
+
+    @property
+    def type(self):
+        return "Base"
+
+    def remove_batv(self, addr):
+        return self.batv_regex.sub(r"\g<LEFT_PART>@", addr, count=1)
 
     def check(self, mailfrom, recipient):
         "Check if mailfrom/recipient combination is whitelisted."
@@ -64,7 +77,7 @@ class WhitelistModel(peewee.Model):
     permanent = peewee.BooleanField(default=False)
 
 
-class Meta(object):
+class Meta:
     indexes = (
         # trailing comma is mandatory if only one index should be created
         (('mailfrom', 'recipient'), True),
@@ -73,44 +86,31 @@ class Meta(object):
 
 class DatabaseWhitelist(WhitelistBase):
     "Whitelist class to store whitelist in a database"
+    whitelist_type = "db"
     _db_connections = {}
     _db_tables = {}
 
-    def __init__(self, global_config, config, configtest=False):
+    def __init__(self, cfg, debug):
-        super(
+        super().__init__(cfg, debug)
-            DatabaseWhitelist,
-            self).__init__(
-            global_config,
-            config,
-            configtest)
 
-        # check if mandatory options are present in config
+        tablename = cfg["table"]
-        for option in ["whitelist_db_connection", "whitelist_db_table"]:
+        connection_string = cfg["connection"]
-            if option not in self.config.keys() and option in self.global_config.keys():
-                self.config[option] = self.global_config[option]
-            if option not in self.config.keys():
-                raise RuntimeError(
-                    "mandatory option '{}' not present in config section '{}' or 'global'".format(
-                        option, self.name))
-
-        tablename = self.config["whitelist_db_table"]
-        connection_string = self.config["whitelist_db_connection"]
 
         if connection_string in DatabaseWhitelist._db_connections.keys():
             db = DatabaseWhitelist._db_connections[connection_string]
         else:
             try:
                 # connect to database
-                self.logger.debug(
+                conn = re.sub(
-                    "connecting to database '{}'".format(
-                        re.sub(
                     r"(.*?://.*?):.*?(@.*)",
                     r"\1:<PASSWORD>\2",
-                            connection_string)))
+                    connection_string)
+                self.logger.debug(
+                    f"connecting to database '{conn}'")
                 db = connect(connection_string)
             except Exception as e:
                 raise RuntimeError(
-                    "unable to connect to database: {}".format(e))
+                    f"unable to connect to database: {e}")
 
             DatabaseWhitelist._db_connections[connection_string] = db
 
@@ -118,22 +118,28 @@ class DatabaseWhitelist(WhitelistBase):
         self.meta = Meta
         self.meta.database = db
         self.meta.table_name = tablename
-        self.model = type("WhitelistModel_{}".format(self.name), (WhitelistModel,), {
+        self.model = type(
-            "Meta": self.meta
+            f"WhitelistModel_{self.cfg['name']}",
-        })
+            (WhitelistModel,),
+            {"Meta": self.meta})
 
         if connection_string not in DatabaseWhitelist._db_tables.keys():
             DatabaseWhitelist._db_tables[connection_string] = []
 
         if tablename not in DatabaseWhitelist._db_tables[connection_string]:
             DatabaseWhitelist._db_tables[connection_string].append(tablename)
-            if not self.configtest:
             try:
                 db.create_tables([self.model])
             except Exception as e:
                 raise RuntimeError(
-                        "unable to initialize table '{}': {}".format(
+                    f"unable to initialize table '{tablename}': {e}")
-                            tablename, e))
+
+    def __str__(self):
+        cfg = []
+        for arg in ("connection", "table"):
+            if arg in self.cfg:
+                cfg.append(f"{arg}={self.cfg[arg]}")
+        return "DatabaseWhitelist(" + ", ".join(cfg) + ")"
 
     def _entry_to_dict(self, entry):
         result = {}
@@ -157,23 +163,27 @@ class DatabaseWhitelist(WhitelistBase):
                 value += 1
         return value
 
-    def check(self, mailfrom, recipient):
+    def check(self, mailfrom, recipient, logger):
         # check if mailfrom/recipient combination is whitelisted
-        super(DatabaseWhitelist, self).check(mailfrom, recipient)
+        super().check(mailfrom, recipient)
+        mailfrom = self.remove_batv(mailfrom)
+        recipient = self.remove_batv(recipient)
 
         # generate list of possible mailfroms
-        self.logger.debug(
+        logger.debug(
-            "query database for whitelist entries from <{}> to <{}>".format(
+            f"query database for whitelist entries from <{mailfrom}> "
-                mailfrom, recipient))
+            f"to <{recipient}>")
         mailfroms = [""]
         if "@" in mailfrom and not mailfrom.startswith("@"):
-            mailfroms.append("@{}".format(mailfrom.split("@")[1]))
+            domain = mailfrom.split("@")[1]
+            mailfroms.append(f"@{domain}")
         mailfroms.append(mailfrom)
 
         # generate list of possible recipients
         recipients = [""]
         if "@" in recipient and not recipient.startswith("@"):
-            recipients.append("@{}".format(recipient.split("@")[1]))
+            domain = recipient.split("@")[1]
+            recipients.append(f"@{domain}")
         recipients.append(recipient)
 
         # query the database
@@ -183,7 +193,7 @@ class DatabaseWhitelist(WhitelistBase):
                     self.model.mailfrom.in_(mailfroms),
                     self.model.recipient.in_(recipients)))
         except Exception as e:
-            raise RuntimeError("unable to query database: {}".format(e))
+            raise RuntimeError(f"unable to query database: {e}")
 
         if not entries:
             # no whitelist entry found
@@ -204,7 +214,7 @@ class DatabaseWhitelist(WhitelistBase):
 
     def find(self, mailfrom=None, recipients=None, older_than=None):
         "Find whitelist entries."
-        super(DatabaseWhitelist, self).find(mailfrom, recipients, older_than)
+        super().find(mailfrom, recipients, older_than)
 
         if isinstance(mailfrom, str):
             mailfrom = [mailfrom]
@@ -215,7 +225,8 @@ class DatabaseWhitelist(WhitelistBase):
         try:
             for entry in list(self.model.select()):
                 if older_than is not None:
-                    if (datetime.now() - entry.last_used).total_seconds() < (older_than * 86400):
+                    delta = (datetime.now() - entry.last_used).total_seconds()
+                    if delta < (older_than * 86400):
                         continue
 
                 if mailfrom is not None:
@@ -228,20 +239,21 @@ class DatabaseWhitelist(WhitelistBase):
 
                 entries.update(self._entry_to_dict(entry))
         except Exception as e:
-            raise RuntimeError("unable to query database: {}".format(e))
+            raise RuntimeError(f"unable to query database: {e}")
 
         return entries
 
     def add(self, mailfrom, recipient, comment, permanent):
         "Add entry to whitelist."
-        super(
+        super().add(
-            DatabaseWhitelist,
-            self).add(
             mailfrom,
             recipient,
             comment,
             permanent)
 
+        mailfrom = self.remove_batv(mailfrom)
+        recipient = self.remove_batv(recipient)
+
         try:
             self.model.create(
                 mailfrom=mailfrom,
@@ -249,46 +261,18 @@ class DatabaseWhitelist(WhitelistBase):
                 comment=comment,
                 permanent=permanent)
         except Exception as e:
-            raise RuntimeError("unable to add entry to database: {}".format(e))
+            raise RuntimeError(f"unable to add entry to database: {e}")
 
     def delete(self, whitelist_id):
         "Delete entry from whitelist."
-        super(DatabaseWhitelist, self).delete(whitelist_id)
+        super().delete(whitelist_id)
 
         try:
             query = self.model.delete().where(self.model.id == whitelist_id)
             deleted = query.execute()
         except Exception as e:
             raise RuntimeError(
-                "unable to delete entry from database: {}".format(e))
+                f"unable to delete entry from database: {e}")
 
         if deleted == 0:
             raise RuntimeError("invalid whitelist id")
-
-
-class WhitelistCache(object):
-    def __init__(self):
-        self.cache = {}
-
-    def load(self, whitelist, mailfrom, recipients):
-        for recipient in recipients:
-            self.check(whitelist, mailfrom, recipient)
-
-    def check(self, whitelist, mailfrom, recipient):
-        if whitelist not in self.cache.keys():
-            self.cache[whitelist] = {}
-        if recipient not in self.cache[whitelist].keys():
-            self.cache[whitelist][recipient] = None
-        if self.cache[whitelist][recipient] is None:
-            self.cache[whitelist][recipient] = whitelist.check(
-                mailfrom, recipient)
-        return self.cache[whitelist][recipient]
-
-    def get_whitelisted_recipients(self, whitelist, mailfrom, recipients):
-        self.load(whitelist, mailfrom, recipients)
-        return list(
-            filter(lambda x: self.cache[whitelist][x], self.cache[whitelist].keys()))
-
-
-# list of whitelist types and their related whitelist classes
-TYPES = {"db": DatabaseWhitelist}

setup.cfg

@@ -0,0 +1,2 @@
+[metadata]
+version = attr: pyquarantine.__version__

setup.py

@@ -4,17 +4,13 @@ def read_file(fname):
     with open(fname, 'r') as f:
         return f.read()
 
-version = {}
-exec(read_file("pyquarantine/version.py"), version)
-
 setup(name = "pyquarantine",
-    version = version["__version__"],
     author = "Thomas Oettli",
     author_email = "spacefreak@noop.ch",
     description = "A pymilter based sendmail/postfix pre-queue filter.",
     license = "GPL 3",
-    keywords = "quarantine milter",
+    keywords = "header quarantine milter",
-    url = "https://github.com/spacefreak86/pyquarantine-milter",
+    url = "https://github.com/spacefreak86/pyquarantine",
     packages = ["pyquarantine"],
     long_description = read_file("README.md"),
     long_description_content_type = "text/markdown",
@@ -22,19 +18,36 @@ setup(name = "pyquarantine",
         #   3 - Alpha
         #   4 - Beta
         #   5 - Production/Stable
-        "Development Status :: 3 - Alpha",
+        "Development Status :: 5 - Production/Stable",
         "License :: OSI Approved :: GNU General Public License v3 or later (GPLv3+)",
         "Operating System :: OS Independent",
         "Programming Language :: Python",
         "Programming Language :: Python :: 3",
         "Topic :: Communications :: Email :: Filters"
     ],
+    include_package_data = True,
     entry_points = {
         "console_scripts": [
             "pyquarantine-milter=pyquarantine.run:main",
-            "pyquarantine=pyquarantine.cli:main"
+            "pyquarantine=pyquarantine.cli:main",
         ]
     },
-    install_requires = ["pymilter", "netaddr", "beautifulsoup4[lxml]", "peewee"],
+    data_files = [
-    python_requires = ">=3"
+        (
+            "/etc/pyquarantine",
+            [
+                "pyquarantine/misc/pyquarantine.conf.default"
+            ]
+        ), (
+            "/etc/pyquarantine/templates",
+            [
+                "pyquarantine/misc/templates/disclaimer_html.template",
+                "pyquarantine/misc/templates/disclaimer_text.template",
+                "pyquarantine/misc/templates/notification.template",
+                "pyquarantine/misc/templates/removed.png"
+            ]
+        )
+    ],
+    install_requires = ["pymilter >= 1.5", "jsonschema", "netaddr", "beautifulsoup4[lxml]", "peewee"],
+    python_requires = ">=3.9"
 )

test-pyquarantine

@@ -1,9 +0,0 @@
-#!/usr/bin/env python
-
-import sys
-import pyquarantine.cli
-
-if __name__ == '__main__':
-    sys.exit(
-        pyquarantine.cli.main()
-    )

test-pyquarantine-milter

@@ -1,9 +0,0 @@
-#!/usr/bin/env python
-
-import sys
-import pyquarantine.run
-
-if __name__ == '__main__':
-    sys.exit(
-        pyquarantine.run.main()
-    )