diff --git a/README.md b/README.md index 9b51f69..2c5e99f 100644 --- a/README.md +++ b/README.md @@ -1,157 +1,140 @@ -# pyquarantine-milter -A pymilter based sendmail/postfix pre-queue filter with the ability to quarantine e-mails, sending notifications -to recipients and respond with a milter-action (ACCEPT, DISCARD or REJECT). +# pyquarantine +A pymilter based sendmail/postfix pre-queue filter with the ability to add, remove and modify e-mail headers. +The project is currently in beta status, but it is already used in a productive enterprise environment that processes about a million e-mails per month. -It is useful in many cases because of its felxible configuration and the ability to handle any number of quarantines. -The MTA can check e-mail headers using regular expressions to determine if and which quarantine to use. -Each quarantine can be configured with a quarantine type, notification type, whitelist and an action to respond with. +The basic idea is to define rules with conditions and actions which are processed when all conditions are true. -Addionally, pyquarantine-milter provides a sanitized, harmless version of the text parts of e-mails, which can be embedded in e-mail notifications. This makes it easier for users to decide, if a match is a false-positive or not. If a matching quarantine provides a quarantine ID of the original e-mail, it is also available as a template variable. This is useful if you want to add links to a webservice to notification e-mails, to give your users the ability to release e-mails or whitelist the from-address for example. The webservice then releases the e-mail from the quarantine. +## Dependencies +pyquarantine is depending on these python packages, but they are installed automatically if you are working with pip. +* [jsonschema](https://github.com/Julian/jsonschema) +* [pymilter](https://github.com/sdgathman/pymilter) +* [netaddr](https://github.com/drkjam/netaddr) +* [peewee](https://github.com/coleifer/peewee) +* [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/) -The project is currently in beta status, but it is already used in a productive enterprise environment which processes about a million e-mails per month. +## Installation +* Install pyquarantine with pip and copy the example config file. +```sh +pip install pyquarantine +cp /etc/pyquarantine/pyquarantine.conf.example /etc/pyquarantine/pyquarantine.conf +``` +* Modify /etc/pyquarantine/pyquarantine.conf according to your needs. -## Requirements -* pymilter -* netaddr -* peewee -* BeautifulSoup +## Configuration options +pyquarantine uses a config file in JSON format. The config file has to be JSON valid with the exception of allowed comment lines starting with **#**. The options are described below. +Rules and actions are processed in the given order. -## Configuration -The pyquarantine module uses an INI-style configuration file. The sections are described below. If you have to specify a path in the config, you can always use a relative path to the last loaded config file. - -### Section "global" -Any available configuration option can be set in the global section as default instead of in a quarantine section. - -The following configuration options are mandatory in the global section: -* **quarantines** - Comma-separated, ordered list of active quarantines. For each, there must be a section of the same name in the configuration. -* **preferred_quarantin_action** - Defines which quarantine action should be preferred if multiple quarantines are matching for multiple recipients. - If at least one recipient receives the original e-mail due to whitelisting, the action is always ACCEPT. - Possible values are: - * **first** - * **last** - -### Quarantine sections -The following configuration options are mandatory in each quarantine section: -* **regex** - Case insensitive regular expression to filter e-mail headers. -* **storage_type** - One of the storage types described below. -* **action** - One of the actions described below. -* **notification_type** - One of the notification types described below. -* **whitelist_type** - One of the whitelist types described below. -* **smtp_host** - SMTP host used to release original e-mails from the quarantine. -* **smtp_port** - SMTP port - -The following configuration options are optional in each quarantine section: -* **host_whitelist** - Comma-separated list of host and network addresses to be ignored by this quarantine. -* **reject_reason** - Reason to return to the client if action is set to reject. - - -### Storage types -* **NONE** - Original e-mails scrapped, sent to nirvana, black-holed or however you want to call it. - -* **FILE** - Original e-mails are stored on the filesystem with a unique filename. The filename is available as a - template variable used in notifiaction templates. - The following configuration options are mandatory for this quarantine type: - * **storage_directory** - The directory in which quarantined e-mails are stored. - - -### Notification types -* **NONE** - No quarantine notifications will be sent. - -* **EMAIL** - Quarantine e-mail notifications are sent to recipients. The SMTP host and port, E-mail template, from-address and the subject are configurable for each quarantine. The templates must contain the notification e-mail text in HTML form. - - The following template variables are available: - * **{EMAIL_ENVELOPE_FROM}** - E-mail sender address received by the milter. - * **{EMAIL_ENVELOPE_FROM_URL}** - Like EMAIL_ENVELOPE_FROM, but URL encoded - * **{EMAIL_FROM}** - Value of the FROM header of the original e-mail. - * **{EMAIL_ENVELOPE_TO}** - E-mail recipient address of this notification. - * **{EMAIL_ENVELOPE_TO_URL}** - Like EMAIL_ENVELOPE_TO, but URL encoded - * **{EMAIL_TO}** - Value of the TO header of the original e-mail. - * **{EMAIL_SUBJECT}** - Configured e-mail subject. - * **{EMAIL_QUARANTINE_ID}** - Quarantine-ID of the original e-mail if available, empty otherwise. - * **{EMAIL_HTML_TEXT}** - Sanitized version of the e-mail text part of the original e-mail. Only harmless HTML tags and attributes are included. Images are optionally stripped or replaced with the image set by notification_email_replacement_img option. - - Some template variables are only available if the regex of the matching quarantine contains subgroups or named subgroups (python syntax). This is useful to include information (e.g. virus names, spam points, ...) of the matching header within the notification. - The following dynamic template variables are available: - * **{SUBGROUP_n}** - Content of a subgroup, 'n' will be replaced by the index number of each subgroup, starting with 0. - * **{subgroup_name}** - Content of a named subgroup, 'subgroup_name' will be replaced by its name. - - The following configuration options are mandatory for this notification type: - * **notification_email_smtp_host** - SMTP host used to send notification e-mails. - * **notification_email_smtp_port** - SMTP port. - * **notification_email_envelope_from** - Notification e-mail envelope from-address. - * **notification_email_from** - Value of the notification e-mail from header. Optionally, you may use the EMAIL_FROM template variable described above. - * **notification_email_subject** - Notification e-mail subject. Optionally, you may use the EMAIL_SUBJECT template variable described above. - * **notification_email_template** - Path to the notification e-mail template. It is hold in memory during runtime. - * **notification_email_embedded_imgs** - Comma-separated list of images to embed into the notification e-mail. The Content-ID of each image will be set to the filename, so you can reference it from the e-mail template. All images are hold in memory during runtime. - Leave empty to disable. - - The following configuration options are optional for this notification type: - * **notification_email_strip_images** - Enable to strip images from e-mails. This option superseeds notification_email_replacement_img. - * **notification_email_replacement_img** - Path to an image to replace images in e-mails. It is hold in memory during runtime. - * **notification_email_parser_lib** - HTML parser library used to parse text part of emails. +### Global +Config options in **global** section: +* **socket** (optional) + The socket used to communicate with the MTA. If it is not specified in the config, it has to be set as command line option. +* **local_addrs** (optional) + A list of hosts and network addresses which are considered local. It is used to for the condition option [local](#Conditions). + Default: **fe80::/64, ::1/128, 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16** +* **loglevel** (optional) + Set the log level. This option may be overriden by any rule or action object. Possible values are: + * **error** + * **warning** + * **info** + * **debug** + Default: **info** +* **pretend** (optional) + Pretend actions, for test purposes. This option may be overriden by any rule or action object. +### Rules +Config options for **rule** objects: +* **name** (optional) + Name of the rule. + Default: **Rule #n** +* **actions** + A list of action objects which are processed in the given order. +* **conditions** (optional) + A list of conditions which all have to be true to process the actions. +* **loglevel** (optional) + As described above in the [Global](#Global) section. +* **pretend** (optional) + As described above in the [Global](#Global) section. ### Actions -Every quarantine responds with a milter-action if an e-mail header matches the configured regular expression. Please think carefully what you set here or your MTA will do something you do not want. -The following actions are available: -* **ACCEPT** - Tell the MTA to continue processing the e-mail. -* **DISCARD** - Tell the MTA to silently discard the e-mail. -* **REJECT** - Tell the MTA to reject the e-mail. +Config options for **action** objects: +* **name** (optional) + Name of the action. + Default: **Action #n** +* **type** + Action type. Possible values are: + * **add_header** + * **del_header** + * **mod_header** + * **add_disclaimer** + * **store** +* **conditions** (optional) + A list of conditions which all have to be true to process the action. +* **loglevel** (optional) + As described above in the [Global](#Global) section. +* **pretend** (optional) + As described above in the [Global](#Global) section. +Config options for **add_header** actions: + * **field** + Name of the header. + * **value** + Value of the header. -### Whitelist types -* **NONE** - No whitelist will be used. +Config options for **del_header** actions: + * **field** + Regular expression to match against header names. + * **value** (optional) + Regular expression to match against the headers value. -* **DB** - A database whitelist will be used. All database types supported by peewee are available. +Config options for **mod_header** actions: + * **field** + Regular expression to match against header names. + * **search** (optional) + Regular expression to match against header values. You may use subgroups or named subgroups (python syntax) to include parts of the original value in the new value. + * **value** + New value of the header. - The following configuration options are mandatory for this whitelist type: - * **whitelist_db_connection** - Database connection string (e.g. mysql://user:password@host:port). +Config options for **add_disclaimer** actions: + * **action** + Action to perform with the disclaimer. Possible values are: + * append + * prepend + * **html_template** + Path to a file which contains the html representation of the disclaimer. + * **text_template** + Path to a file which contains the text representation of the disclaimer. + * **error_policy** (optional) + Set the error policy in case the disclaimer cannot be added (e.g. if no body part is present in the e-mail). Possible values are: + * **wrap** + A new e-mail body is generated with the disclaimer as body and the original e-mail attached. + * **ignore** + Ignore the error and do nothing. + * **reject** + Reject the e-mail. + Default: **wrap** - * **whitelist_db_table** - Database table to use. +Config options for **store** actions: + * **storage_type** + Storage type. Possible values are: + * **file** + * **original** (optional) + Default: **false** + If set to true, store the message as received by the MTA instead of storing the current state of the message, that may was modified already by other actions. + +Config options for **file** storage: + * **directory** + Directory used to store e-mails. + +### Conditions +Config options for **conditions** objects: +* **local** (optional) + If set to true, the rule is only executed for e-mails originating from addresses defined in local_addrs and vice versa. +* **hosts** (optional) + A list of hosts and network addresses for which the rule should be executed. +* **envfrom** (optional) + A regular expression to match against the evenlope-from addresses for which the rule should be executed. +* **envto** (optional) + A regular expression to match against all evenlope-to addresses. All addresses must match to fulfill the condition. ## Developer information Everyone who wants to improve or extend this project is very welcome. diff --git a/pyquarantine/__init__.py b/pyquarantine/__init__.py index fbbb97d..c46681b 100644 --- a/pyquarantine/__init__.py +++ b/pyquarantine/__init__.py @@ -1,717 +1,324 @@ -# PyQuarantine-Milter is free software: you can redistribute it and/or modify +# pyquarantine is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # -# PyQuarantine-Milter is distributed in the hope that it will be useful, +# pyquarantine is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License -# along with PyQuarantineMilter. If not, see . +# along with pyquarantine. If not, see . # +__all__ = [ + "action", + "base", + "cli", + "conditions", + "config", + "mailer", + "modify", + "notify", + "rule", + "run", + "storage", + "whitelist", + "ModifyMilter"] + +__version__ = "2.0.0" + +from pyquarantine import _runtime_patches + import Milter -import configparser import logging -import os -import re -import encodings from Milter.utils import parse_addr from collections import defaultdict -from email.charset import Charset -from email.header import Header, decode_header +from copy import copy +from email import message_from_binary_file +from email.header import Header, decode_header, make_header +from email.headerregistry import AddressHeader, _default_header_map +from email.policy import SMTPUTF8 from io import BytesIO -from itertools import groupby -from netaddr import IPAddress, IPNetwork, AddrFormatError -from pyquarantine import mailer -from pyquarantine import notifications -from pyquarantine import storages -from pyquarantine import whitelists +from netaddr import IPNetwork, AddrFormatError -__all__ = [ - "make_header", - "Quarantine", - "QuarantineMilter", - "setup_milter", - "reload_config", - "cli", - "mailer", - "notifications", - "storages", - "run", - "whitelists"] - -__version__ = "1.0.10" +from pyquarantine.base import CustomLogger, MilterMessage +from pyquarantine.base import replace_illegal_chars +from pyquarantine.rule import Rule -################################################ -# add charset alias for windows-874 encoding # -################################################ +class ModifyMilter(Milter.Base): + """ModifyMilter based on Milter.Base to implement milter communication""" -aliases = encodings.aliases.aliases + _rules = [] + _loglevel = logging.INFO + _addr_fields = [f for f, v in _default_header_map.items() + if issubclass(v, AddressHeader)] -for alias in ["windows-874", "windows_874"]: - if alias not in aliases: - aliases[alias] = "cp874" + @staticmethod + def set_config(cfg, debug): + ModifyMilter._loglevel = cfg.get_loglevel(debug) -setattr(encodings.aliases, "aliases", aliases) - -################################################ - - -def make_header(decoded_seq, maxlinelen=None, header_name=None, - continuation_ws=' ', errors='strict'): - """Create a Header from a sequence of pairs as returned by decode_header() - - decode_header() takes a header value string and returns a sequence of - pairs of the format (decoded_string, charset) where charset is the string - name of the character set. - - This function takes one of those sequence of pairs and returns a Header - instance. Optional maxlinelen, header_name, and continuation_ws are as in - the Header constructor. - """ - h = Header(maxlinelen=maxlinelen, header_name=header_name, - continuation_ws=continuation_ws) - for s, charset in decoded_seq: - # None means us-ascii but we can simply pass it on to h.append() - if charset is not None and not isinstance(charset, Charset): - charset = Charset(charset) - h.append(s, charset, errors=errors) - return h - - -class Quarantine(object): - """Quarantine class suitable for QuarantineMilter - - The class holds all the objects and functions needed - for QuarantineMilter quarantine. - - """ - - # list of possible actions - _actions = { - "ACCEPT": Milter.ACCEPT, - "REJECT": Milter.REJECT, - "DISCARD": Milter.DISCARD} - - def __init__(self, name, index=0, regex=None, storage=None, whitelist=None, - host_whitelist=[], notification=None, action="ACCEPT", - reject_reason=None): - self.logger = logging.getLogger(__name__) - self.name = name - self.index = index - self.regex = regex - if regex: - self.re = re.compile( - regex, re.MULTILINE + re.DOTALL + re.IGNORECASE) - self.storage = storage - self.whitelist = whitelist - self.host_whitelist = host_whitelist - self.notification = notification - action = action.upper() - assert action in self._actions - self.action = action - self.milter_action = self._actions[action] - self.reject_reason = reject_reason - - def setup_from_cfg(self, global_cfg, cfg, test=False): - defaults = { - "action": "accept", - "reject_reason": "Message rejected", - "storage_type": "none", - "notification_type": "none", - "whitelist_type": "none", - "host_whitelist": "" - } - # check config - for opt in ["regex", "smtp_host", "smtp_port"] + list(defaults.keys()): - if opt in cfg: - continue - if opt in global_cfg: - cfg[opt] = global_cfg[opt] - elif opt in defaults: - cfg[opt] = defaults[opt] - else: - raise RuntimeError( - f"mandatory option '{opt}' not present in " - f"config section '{self.name}' or 'global'") - - # pre-compile regex - self.logger.debug( - f"{self.name}: compiling regex '{cfg['regex']}'") - self.re = re.compile( - cfg["regex"], re.MULTILINE + re.DOTALL + re.IGNORECASE) - - self.smtp_host = cfg["smtp_host"] - self.smtp_port = cfg["smtp_port"] - - # create storage instance - storage_type = cfg["storage_type"].lower() - if storage_type in storages.TYPES: - self.logger.debug( - f"{self.name}: initializing storage " - f"type '{storage_type.upper()}'") - self.storage = storages.TYPES[storage_type]( - self.name, global_cfg, cfg, test) - elif storage_type == "none": - self.logger.debug(f"{self.name}: storage is NONE") - self.storage = None - else: - raise RuntimeError( - f"{self.name}: unknown storage type '{storage_type}'") - - # create whitelist instance - whitelist_type = cfg["whitelist_type"].lower() - if whitelist_type in whitelists.TYPES: - self.logger.debug( - f"{self.name}: initializing whitelist " - f"type '{whitelist_type.upper()}'") - self.whitelist = whitelists.TYPES[whitelist_type]( - self.name, global_cfg, cfg, test) - elif whitelist_type == "none": - self.logger.debug(f"{self.name}: whitelist is NONE") - self.whitelist = None - else: - raise RuntimeError( - f"{self.name}: unknown whitelist type '{whitelist_type}'") - - # create notification instance - notification_type = cfg["notification_type"].lower() - if notification_type in notifications.TYPES: - self.logger.debug( - f"{self.name}: initializing notification " - f"type '{notification_type.upper()}'") - self.notification = notifications.TYPES[notification_type]( - self.name, global_cfg, cfg, test) - elif notification_type == "none": - self.logger.debug(f"{self.name}: notification is NONE") - self.notification = None - else: - raise RuntimeError( - f"{self.name}: unknown notification " - f"type '{notification_type}'") - - # determining milter action for this quarantine - action = cfg["action"].upper() - if action in self._actions: - self.logger.debug(f"{self.name}: action is {action}") - self.action = action - self.milter_action = self._actions[action] - else: - raise RuntimeError( - f"{self.name}: unknown action '{action}'") - - self.reject_reason = cfg["reject_reason"] - - # create host/network whitelist - self.host_whitelist = [] - host_whitelist = set([p.strip() - for p in cfg["host_whitelist"].split(",") if p]) - for host in host_whitelist: - if not host: - continue - # parse network notation - try: - net = IPNetwork(host) - except AddrFormatError as e: - raise RuntimeError( - f"{self.name}: error parsing host_whitelist: {e}") - else: - self.host_whitelist.append(net) - if self.host_whitelist: - whitelist = ", ".join([str(ip) for ip in host_whitelist]) - self.logger.debug( - f"{self.name}: host whitelist: {whitelist}") - - def notify(self, storage_id, recipient=None, synchronous=True): - "Notify recipient about email in storage." - if not self.storage: - raise RuntimeError( - "storage type is set to None, unable to send notification") - - if not self.notification: - raise RuntimeError( - "notification type is set to None, " - "unable to send notification") - - fp, metadata = self.storage.get_mail(storage_id) - - if recipient is not None: - if recipient not in metadata["recipients"]: - raise RuntimeError(f"invalid recipient '{recipient}'") - recipients = [recipient] - else: - recipients = metadata["recipients"] - - self.notification.notify( - metadata["queue_id"], storage_id, metadata["mailfrom"], - recipients, metadata["headers"], fp, - metadata["subgroups"], metadata["named_subgroups"], - synchronous) - fp.close() - - def release(self, storage_id, recipients=None): - "Release email from storage." - if not self.storage: - raise RuntimeError( - "storage type is set to None, unable to release email") - - fp, metadata = self.storage.get_mail(storage_id) try: - mail = fp.read() - fp.close() - except IOError as e: - raise RuntimeError(f"unable to read data file: {e}") + local_addrs = [] + for addr in cfg["local_addrs"]: + local_addrs.append(IPNetwork(addr)) + except AddrFormatError as e: + raise RuntimeError(e) - if recipients and type(recipients) == str: - recipients = [recipients] - else: - recipients = metadata["recipients"] + logger = logging.getLogger(__name__) + logger.setLevel(ModifyMilter._loglevel) + for idx, rule_cfg in enumerate(cfg["rules"]): + if "name" not in rule_cfg: + rule_cfg["name"] = f"rule#{idx}" + if "loglevel" not in rule_cfg: + rule_cfg["loglevel"] = cfg["loglevel"] + if "pretend" not in rule_cfg: + rule_cfg["pretend"] = cfg["pretend"] + rule = Rule(rule_cfg, local_addrs, debug) - for recipient in recipients: - if recipient not in metadata["recipients"]: - raise RuntimeError(f"invalid recipient '{recipient}'") - - try: - mailer.smtp_send( - self.smtp_host, - self.smtp_port, - metadata["mailfrom"], - recipient, - mail) - except Exception as e: - raise RuntimeError( - f"error while sending email to '{recipient}': {e}") - self.storage.delete(storage_id, recipient) - - def get_storage(self): - return self.storage - - def get_notification(self): - return self.notification - - def get_whitelist(self): - return self.whitelist - - def host_in_whitelist(self, hostaddr): - ip = IPAddress(hostaddr[0]) - for entry in self.host_whitelist: - if ip in entry: - return True - return False - - def match(self, header): - return self.re.search(header) - - -class QuarantineMilter(Milter.Base): - """QuarantineMilter based on Milter.Base to implement milter communication - - The class variable quarantines needs to be filled by - runng the setup_milter function. - - """ - quarantines = [] - preferred_action = "first" - - # list of default config files - _cfg_files = [ - "/etc/pyquarantine/pyquarantine.conf", - os.path.expanduser('~/pyquarantine.conf'), - "pyquarantine.conf"] + logger.debug(rule) + ModifyMilter._rules.append(rule) def __init__(self): self.logger = logging.getLogger(__name__) - # save runtime config, it must not change during runtime - self.quarantines = QuarantineMilter.quarantines + self.logger.setLevel(ModifyMilter._loglevel) - def _get_preferred_quarantine(self): - matching_quarantines = [ - q for q in self.rcpts_quarantines.values() if q] - if self.preferred_action == "first": - quarantine = sorted( - matching_quarantines, - key=lambda q: q.index)[0] + def addheader(self, field, value, idx=-1): + value = replace_illegal_chars(Header(s=value).encode()) + self.logger.debug(f"addheader: {field}: {value}") + super().addheader(field, value, idx) + + def chgheader(self, field, value, idx=1): + value = replace_illegal_chars(Header(s=value).encode()) + if value: + self.logger.debug(f"chgheader: {field}[{idx}]: {value}") else: - quarantine = sorted( - matching_quarantines, - key=lambda q: q.index, - reverse=True)[0] - return quarantine + self.logger.debug(f"delheader: {field}[{idx}]") + super().chgheader(field, idx, value) - @staticmethod - def get_cfg_files(): - return QuarantineMilter._cfg_files + def update_headers(self, old_headers): + if self.msg.is_multipart() and not self.msg["MIME-Version"]: + self.msg.add_header("MIME-Version", "1.0") - @staticmethod - def set_cfg_files(cfg_files): - QuarantineMilter._cfg_files = cfg_files + # serialize the message object so it updates its internal strucure + self.msg.as_bytes() - def connect(self, hostname, family, hostaddr): - self.hostaddr = hostaddr - self.logger.debug( - f"accepted milter connection from {hostaddr[0]} " - f"port {hostaddr[1]}") - for quarantine in self.quarantines.copy(): - if quarantine.host_in_whitelist(hostaddr): - self.logger.debug( - f"host {hostaddr[0]} is in whitelist of " - f"quarantine {quarantine.name}") - self.quarantines.remove(quarantine) - if not self.quarantines: - self.logger.debug( - f"host {hostaddr[0]} is in whitelist of all " - f"quarantines, skip further processing") - return Milter.ACCEPT - return Milter.CONTINUE + old_headers = [(f, f.lower(), v) for f, v in old_headers] + headers = [(f, f.lower(), v) for f, v in self.msg.items()] - @Milter.noreply - def envfrom(self, mailfrom, *str): - self.mailfrom = "@".join(parse_addr(mailfrom)).lower() - self.recipients = set() - return Milter.CONTINUE + idx = defaultdict(int) + for field, field_lower, value in old_headers: + idx[field_lower] += 1 + if (field, field_lower, value) not in headers: + self.chgheader(field, "", idx=idx[field_lower]) + idx[field] -= 1 - @Milter.noreply - def envrcpt(self, to, *str): - self.recipients.add("@".join(parse_addr(to)).lower()) - return Milter.CONTINUE + for field, value in self.msg.items(): + field_lower = field.lower() + if (field, field_lower, value) not in old_headers: + self.addheader(field, value) - @Milter.noreply - def data(self): - self.qid = self.getsymval('i') - self.logger.debug( - f"{self.qid}: received queue-id from MTA") - self.recipients = list(self.recipients) - self.logger.debug( - f"{self.qid}: initializing memory buffer to save email data") - # initialize memory buffer to save email data - self.fp = BytesIO() - self.headers = [] - return Milter.CONTINUE + def replacebody(self): + self._body_changed = True - def header(self, name, value): + def _replacebody(self): + if not self._body_changed: + return + data = self.msg.as_bytes() + body_pos = data.find(b"\r\n\r\n") + 4 + self.logger.debug("replace body") + super().replacebody(data[body_pos:]) + del data + + def delrcpt(self, rcpts): + "Remove recipient. May be called from eom callback only." + if not isinstance(rcpts, list): + rcpts = [rcpts] + for rcpt in rcpts: + self.logger.debug(f"delrcpt: {rcpt}") + self.msginfo["rcpts"].remove(rcpt) + super().delrcpt(rcpt) + + def connect(self, IPname, family, hostaddr): try: - # remove surrogates from value - value = value.encode( - errors="surrogateescape").decode(errors="replace") - self.logger.debug(f"{self.qid}: received header: {name}: {value}") - # write email header to memory buffer - self.fp.write(f"{name}: {value}\r\n".encode( - encoding="ascii", errors="replace")) - header = make_header(decode_header(value), errors="replace") - value = str(header).replace("\x00", "") + if hostaddr is None: + self.logger.error(f"received invalid host address {hostaddr}, " + f"unable to proceed") + return Milter.TEMPFAIL + + self.IP = hostaddr[0] + self.port = hostaddr[1] self.logger.debug( - f"{self.qid}: decoded header: {name}: {value}") - self.headers.append((name, value)) - return Milter.CONTINUE + f"accepted milter connection from {self.IP} " + f"port {self.port}") + + # pre-filter rules and actions by the host condition + # also check if the mail body is needed by any upcoming action. + self.rules = [] + self._headersonly = True + for rule in ModifyMilter._rules: + if rule.conditions is None or \ + rule.conditions.match_host(self.IP): + actions = [] + for action in rule.actions: + if action.conditions is None or \ + action.conditions.match_host(self.IP): + actions.append(action) + if not action.headersonly(): + self._headersonly = False + + if actions: + # copy needed rules to preserve configured actions + rule = copy(rule) + rule.actions = actions + self.rules.append(rule) + + if not self.rules: + self.logger.debug( + "host is ignored by all rules, skip further processing") + return Milter.ACCEPT + except Exception as e: self.logger.exception( - f"an exception occured in header function: {e}") + f"an exception occured in connect method: {e}") return Milter.TEMPFAIL + return Milter.CONTINUE + + def hello(self, heloname): + try: + self.heloname = heloname + self.logger.debug(f"received HELO name: {heloname}") + except Exception as e: + self.logger.exception( + f"an exception occured in hello method: {e}") + return Milter.TEMPFAIL + + return Milter.CONTINUE + + def envfrom(self, mailfrom, *str): + try: + self.mailfrom = "@".join(parse_addr(mailfrom)).lower() + self.rcpts = set() + except Exception as e: + self.logger.exception( + f"an exception occured in envfrom method: {e}") + return Milter.TEMPFAIL + + return Milter.CONTINUE + + def envrcpt(self, to, *str): + try: + self.rcpts.add("@".join(parse_addr(to)).lower()) + except Exception as e: + self.logger.exception( + f"an exception occured in envrcpt method: {e}") + return Milter.TEMPFAIL + + return Milter.CONTINUE + + def data(self): + try: + self.qid = self.getsymval('i') + self.logger = CustomLogger( + self.logger, {"qid": self.qid, "name": "milter"}) + self.logger.debug("received queue-id from MTA") + self.fp = BytesIO() + except Exception as e: + self.logger.exception( + f"an exception occured in data method: {e}") + return Milter.TEMPFAIL + + return Milter.CONTINUE + + def header(self, field, value): + try: + # remove CR and LF from address fields, otherwise pythons + # email library throws an exception + if field.lower() in ModifyMilter._addr_fields: + try: + v = str(make_header(decode_header(value))) + except Exception as e: + self.logger.error( + f"unable to decode field '{field}': {e}") + else: + if any(c in v for c in ["\r", "\n"]): + v = v.replace("\r", "").replace("\n", "") + value = Header(s=v).encode() + + # remove surrogates + field = field.encode("ascii", errors="replace") + value = value.encode("ascii", errors="replace") + + self.fp.write(field + b": " + value + b"\r\n") + except Exception as e: + self.logger.exception( + f"an exception occured in header method: {e}") + return Milter.TEMPFAIL + + return Milter.CONTINUE + def eoh(self): try: - self.fp.write("\r\n".encode(encoding="ascii")) - self.wl_cache = whitelists.WhitelistCache() - - # initialize dicts to set quaranines per recipient and keep matches - self.rcpts_quarantines = {} - self.matches = {} - - # iterate email headers - rcpts_to_check = self.recipients.copy() - for name, value in self.headers: - header = f"{name}: {value}" - self.logger.debug( - f"{self.qid}: checking header against configured " - f"quarantines: {header}") - # iterate quarantines - for quarantine in self.quarantines: - if len(self.rcpts_quarantines) == len( - self.recipients): - # every recipient matched a quarantine already - if quarantine.index >= max( - [q.index for q in - self.rcpts_quarantines.values()]): - # all recipients matched a quarantine with at least - # the same precedence already, skip checks against - # quarantines with lower precedence - self.logger.debug( - f"{self.qid}: {quarantine.name}: skip further " - f"checks of this header") - break - - # check email header against quarantine regex - self.logger.debug( - f"{self.qid}: {quarantine.name}: checking header " - f"against regex '{str(quarantine.regex)}'") - match = quarantine.match(header) - if match: - self.logger.debug( - f"{self.qid}: {quarantine.name}: " - f"header matched regex") - # check for whitelisted recipients - whitelist = quarantine.get_whitelist() - if whitelist: - try: - wl_recipients = self.wl_cache.get_recipients( - whitelist, - self.mailfrom, - rcpts_to_check) - except RuntimeError as e: - self.logger.error( - f"{self.qid}: {quarantine.name}: unable " - f"to query whitelist: {e}") - return Milter.TEMPFAIL - else: - wl_recipients = {} - - # iterate recipients - for rcpt in rcpts_to_check.copy(): - if rcpt in wl_recipients: - # recipient is whitelisted in this quarantine - self.logger.debug( - f"{self.qid}: {quarantine.name}: " - f"recipient '{rcpt}' is whitelisted") - continue - - if rcpt not in self.rcpts_quarantines or \ - self.rcpts_quarantines[rcpt].index > \ - quarantine.index: - self.logger.debug( - f"{self.qid}: {quarantine.name}: set " - f"quarantine for recipient '{rcpt}'") - # save match for later use as template - # variables - self.matches[quarantine.name] = match - self.rcpts_quarantines[rcpt] = quarantine - if quarantine.index == 0: - # we do not need to check recipients which - # matched the quarantine with the highest - # precedence already - rcpts_to_check.remove(rcpt) - else: - self.logger.debug( - f"{self.qid}: {quarantine.name}: a " - f"quarantine with same or higher " - f"precedence matched already for " - f"recipient '{rcpt}'") - - if not rcpts_to_check: - self.logger.debug( - f"{self.qid}: all recipients matched the first " - f"quarantine, skipping all remaining header checks") - break - - # check if no quarantine has matched for all recipients - if not self.rcpts_quarantines: - # accept email - self.logger.info( - f"{self.qid}: passed clean for all recipients") - return Milter.ACCEPT - - # check if the mail body is needed - for recipient, quarantine in self.rcpts_quarantines.items(): - if quarantine.get_storage() or quarantine.get_notification(): - # mail body is needed, continue processing - return Milter.CONTINUE - - # quarantine and notification are disabled on all matching - # quarantines, just return configured action - quarantine = self._get_preferred_quarantine() - self.logger.info( - f"{self.qid}: {self.preferred_action} " - f"matching quarantine is '{quarantine.name}', performing " - f"milter action {quarantine.action}") - if quarantine.action == "REJECT": - self.setreply("554", "5.7.0", quarantine.reject_reason) - return quarantine.milter_action - + self.fp.write(b"\r\n") except Exception as e: self.logger.exception( - f"an exception occured in eoh function: {e}") + f"an exception occured in eoh method: {e}") return Milter.TEMPFAIL + return Milter.CONTINUE + def body(self, chunk): try: - # save received body chunk - self.fp.write(chunk) + if not self._headersonly: + self.fp.write(chunk) except Exception as e: self.logger.exception( - f"an exception occured in body function: {e}") + f"an exception occured in body method: {e}") return Milter.TEMPFAIL + return Milter.CONTINUE def eom(self): try: - # processing recipients grouped by quarantines - quarantines = [] - for quarantine, recipients in groupby( - sorted(self.rcpts_quarantines, - key=lambda x: self.rcpts_quarantines[x].index), - lambda x: self.rcpts_quarantines[x]): - quarantines.append((quarantine, list(recipients))) + # msg and msginfo contain the runtime data that + # is read/modified by actions + self.fp.seek(0) + self.msg = message_from_binary_file( + self.fp, _class=MilterMessage, policy=SMTPUTF8.clone( + refold_source='none')) + self.msginfo = { + "mailfrom": self.mailfrom, + "rcpts": [*self.rcpts], + "vars": {}} - # iterate quarantines sorted by index - for quarantine, recipients in sorted( - quarantines, key=lambda x: x[0].index): - headers = defaultdict(str) - for name, value in self.headers: - headers[name.lower()] = value - subgroups = self.matches[quarantine.name].groups( - default="") - named_subgroups = self.matches[quarantine.name].groupdict( - default="") + self._body_changed = False + milter_action = None + for rule in self.rules: + milter_action = rule.execute(self) + self.logger.debug( + f"current template variables: {self.msginfo['vars']}") + if milter_action is not None: + break - rcpts = ", ".join(recipients) - - # check if a storage is configured - storage_id = "" - storage = quarantine.get_storage() - if storage: - # add email to quarantine - self.logger.info( - f"{self.qid}: adding to quarantine " - f"'{quarantine.name}' for: {rcpts}") - try: - storage_id = storage.add( - self.qid, self.mailfrom, recipients, headers, - self.fp, subgroups, named_subgroups) - except RuntimeError as e: - self.logger.error( - f"{self.qid}: unable to add to quarantine " - f"'{quarantine.name}': {e}") - return Milter.TEMPFAIL - - # check if a notification is configured - notification = quarantine.get_notification() - if notification: - # notify - self.logger.info( - f"{self.qid}: sending notification to: {rcpts}") - try: - notification.notify( - self.qid, storage_id, - self.mailfrom, recipients, headers, self.fp, - subgroups, named_subgroups) - except RuntimeError as e: - self.logger.error( - f"{self.qid}: unable to send notification: {e}") - return Milter.TEMPFAIL - - # remove processed recipient - for recipient in recipients: - self.delrcpt(recipient) - self.recipients.remove(recipient) - - self.fp.close() - - # email passed clean for at least one recipient, accepting email - if self.recipients: - rcpts = ", ".join(recipients) - self.logger.info( - f"{self.qid}: passed clean for: {rcpts}") - return Milter.ACCEPT - - # return configured action - quarantine = self._get_preferred_quarantine() - self.logger.info( - f"{self.qid}: {self.preferred_action} matching " - f"quarantine is '{quarantine.name}', performing milter " - f"action {quarantine.action}") - if quarantine.action == "REJECT": - self.setreply("554", "5.7.0", quarantine.reject_reason) - return quarantine.milter_action + if milter_action is None: + self._replacebody() + else: + action, reason = milter_action + if action == "ACCEPT": + self._replacebody() + elif action == "REJECT": + self.setreply("554", "5.7.0", reason) + return Milter.REJECT + elif action == "DISCARD": + return Milter.DISCARD except Exception as e: self.logger.exception( - f"an exception occured in eom function: {e}") + f"an exception occured in eom method: {e}") return Milter.TEMPFAIL - def close(self): - self.logger.debug( - f"disconnect from {self.hostaddr[0]} port {self.hostaddr[1]}") - return Milter.CONTINUE - - -def setup_milter(test=False, cfg_files=[]): - "Generate the configuration for QuarantineMilter class." - logger = logging.getLogger(__name__) - - # read config file - parser = configparser.ConfigParser() - if not cfg_files: - cfg_files = parser.read(QuarantineMilter.get_cfg_files()) - else: - cfg_files = parser.read(cfg_files) - if not cfg_files: - raise RuntimeError("config file not found") - - QuarantineMilter.set_cfg_files(cfg_files) - os.chdir(os.path.dirname(cfg_files[0])) - - # check if mandatory config options in global section are present - if "global" not in parser.sections(): - raise RuntimeError( - "mandatory section 'global' not present in config file") - for option in ["quarantines", "preferred_quarantine_action"]: - if not parser.has_option("global", option): - raise RuntimeError( - f"mandatory option '{option}' not present in config " - f"section 'global'") - - # read global config section - global_cfg = dict(parser.items("global")) - preferred_action = global_cfg["preferred_quarantine_action"].lower() - if preferred_action not in ["first", "last"]: - raise RuntimeError( - "option preferred_quarantine_action has illegal value") - - # read active quarantine names - quarantines = [ - q.strip() for q in global_cfg["quarantines"].split(",")] - if len(quarantines) != len(set(quarantines)): - raise RuntimeError( - "at least one quarantine is specified multiple times " - "in quarantines option") - if "global" in quarantines: - quarantines.remove("global") - logger.warning( - "removed illegal quarantine name 'global' from list of " - "active quarantines") - if not quarantines: - raise RuntimeError("no quarantines configured") - - milter_quarantines = [] - logger.debug("preparing milter configuration ...") - # iterate quarantine names - for index, name in enumerate(quarantines): - # check if config section for current quarantine exists - if name not in parser.sections(): - raise RuntimeError( - f"config section '{name}' does not exist") - - cfg = dict(parser.items(name)) - quarantine = Quarantine(name, index) - quarantine.setup_from_cfg(global_cfg, cfg, test) - milter_quarantines.append(quarantine) - - QuarantineMilter.preferred_action = preferred_action - QuarantineMilter.quarantines = milter_quarantines - - -def reload_config(): - "Reload the configuration of QuarantineMilter class." - logger = logging.getLogger(__name__) - - try: - setup_milter() - except RuntimeError as e: - logger.info(e) - logger.info("daemon is still running with previous configuration") - else: - logger.info("reloaded configuration") + return Milter.ACCEPT diff --git a/pyquarantine/_runtime_patches.py b/pyquarantine/_runtime_patches.py new file mode 100644 index 0000000..aef4656 --- /dev/null +++ b/pyquarantine/_runtime_patches.py @@ -0,0 +1,229 @@ +# pyquarantine is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# pyquarantine is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with pyquarantine. If not, see . +# + +import encodings + + +##################################### +# patch pythons email library # +##################################### +# +# https://bugs.python.org/issue27257 +# https://bugs.python.org/issue30988 +# +# fix: https://github.com/python/cpython/pull/15600 + +import email._header_value_parser +from email._header_value_parser import TokenList, NameAddr +from email._header_value_parser import get_display_name, get_angle_addr +from email._header_value_parser import get_cfws, errors +from email._header_value_parser import CFWS_LEADER, PHRASE_ENDS + + +class DisplayName(email._header_value_parser.DisplayName): + @property + def display_name(self): + res = TokenList(self) + if len(res) == 0: + return res.value + if res[0].token_type == 'cfws': + res.pop(0) + else: + if isinstance(res[0], TokenList) and \ + res[0][0].token_type == 'cfws': + res[0] = TokenList(res[0][1:]) + if res[-1].token_type == 'cfws': + res.pop() + else: + if isinstance(res[-1], TokenList) and \ + res[-1][-1].token_type == 'cfws': + res[-1] = TokenList(res[-1][:-1]) + return res.value + + +def get_name_addr(value): + """ name-addr = [display-name] angle-addr + + """ + name_addr = NameAddr() + # Both the optional display name and the angle-addr can start with cfws. + leader = None + if value[0] in CFWS_LEADER: + leader, value = get_cfws(value) + if not value: + raise errors.HeaderParseError( + "expected name-addr but found '{}'".format(leader)) + if value[0] != '<': + if value[0] in PHRASE_ENDS: + raise errors.HeaderParseError( + "expected name-addr but found '{}'".format(value)) + token, value = get_display_name(value) + if not value: + raise errors.HeaderParseError( + "expected name-addr but found '{}'".format(token)) + if leader is not None: + if isinstance(token[0], TokenList): + token[0][:0] = [leader] + else: + token[:0] = [leader] + leader = None + name_addr.append(token) + token, value = get_angle_addr(value) + if leader is not None: + token[:0] = [leader] + name_addr.append(token) + return name_addr, value + + +setattr(email._header_value_parser, "DisplayName", DisplayName) +setattr(email._header_value_parser, "get_name_addr", get_name_addr) + + +# https://bugs.python.org/issue42484 +# +# fix: https://github.com/python/cpython/pull/24669 + +from email._header_value_parser import DOT, ObsLocalPart, ValueTerminal, get_word + + +def get_obs_local_part(value): + """ obs-local-part = word *("." word) + """ + obs_local_part = ObsLocalPart() + last_non_ws_was_dot = False + while value and (value[0]=='\\' or value[0] not in PHRASE_ENDS): + if value[0] == '.': + if last_non_ws_was_dot: + obs_local_part.defects.append(errors.InvalidHeaderDefect( + "invalid repeated '.'")) + obs_local_part.append(DOT) + last_non_ws_was_dot = True + value = value[1:] + continue + elif value[0]=='\\': + obs_local_part.append(ValueTerminal(value[0], + 'misplaced-special')) + value = value[1:] + obs_local_part.defects.append(errors.InvalidHeaderDefect( + "'\\' character outside of quoted-string/ccontent")) + last_non_ws_was_dot = False + continue + if obs_local_part and obs_local_part[-1].token_type != 'dot': + obs_local_part.defects.append(errors.InvalidHeaderDefect( + "missing '.' between words")) + try: + token, value = get_word(value) + last_non_ws_was_dot = False + except errors.HeaderParseError: + if value[0] not in CFWS_LEADER: + raise + token, value = get_cfws(value) + obs_local_part.append(token) + if not obs_local_part: + return obs_local_part, value + if (obs_local_part[0].token_type == 'dot' or + obs_local_part[0].token_type=='cfws' and + obs_local_part[1].token_type=='dot'): + obs_local_part.defects.append(errors.InvalidHeaderDefect( + "Invalid leading '.' in local part")) + if (obs_local_part[-1].token_type == 'dot' or + obs_local_part[-1].token_type=='cfws' and + obs_local_part[-2].token_type=='dot'): + obs_local_part.defects.append(errors.InvalidHeaderDefect( + "Invalid trailing '.' in local part")) + if obs_local_part.defects: + obs_local_part.token_type = 'invalid-obs-local-part' + return obs_local_part, value + + +setattr(email._header_value_parser, "get_obs_local_part", get_obs_local_part) + + +# https://bugs.python.org/issue30681 +# +# fix: https://github.com/python/cpython/pull/22090 + +import email.errors +from email.errors import HeaderDefect + + +class InvalidDateDefect(HeaderDefect): + """Header has unparseable or invalid date""" + + +setattr(email.errors, "InvalidDateDefect", InvalidDateDefect) + + +import email.utils +from email.utils import _parsedate_tz +import datetime + + +def parsedate_to_datetime(data): + parsed_date_tz = _parsedate_tz(data) + if parsed_date_tz is None: + raise ValueError('Invalid date value or format "%s"' % str(data)) + *dtuple, tz = parsed_date_tz + if tz is None: + return datetime.datetime(*dtuple[:6]) + return datetime.datetime(*dtuple[:6], + tzinfo=datetime.timezone(datetime.timedelta(seconds=tz))) + + +setattr(email.utils, "parsedate_to_datetime", parsedate_to_datetime) + + +import email.headerregistry +from email import utils, _header_value_parser as parser + +@classmethod +def parse(cls, value, kwds): + if not value: + kwds['defects'].append(errors.HeaderMissingRequiredValue()) + kwds['datetime'] = None + kwds['decoded'] = '' + kwds['parse_tree'] = parser.TokenList() + return + if isinstance(value, str): + kwds['decoded'] = value + try: + value = utils.parsedate_to_datetime(value) + except ValueError: + kwds['defects'].append(errors.InvalidDateDefect('Invalid date value or format')) + kwds['datetime'] = None + kwds['parse_tree'] = parser.TokenList() + return + kwds['datetime'] = value + kwds['decoded'] = utils.format_datetime(kwds['datetime']) + kwds['parse_tree'] = cls.value_parser(kwds['decoded']) + + +setattr(email.headerregistry.DateHeader, "parse", parse) + + +####################################### +# add charset alias for windows-874 # +####################################### +# +# https://bugs.python.org/issue17254 +# +# fix: https://github.com/python/cpython/pull/10237 + +aliases = encodings.aliases.aliases + +for alias in ["windows-874", "windows_874"]: + if alias not in aliases: + aliases[alias] = "cp874" + +setattr(encodings.aliases, "aliases", aliases) diff --git a/pyquarantine/action.py b/pyquarantine/action.py new file mode 100644 index 0000000..cb30822 --- /dev/null +++ b/pyquarantine/action.py @@ -0,0 +1,64 @@ +# pyquarantine is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# pyquarantine is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with pyquarantine. If not, see . +# + +__all__ = ["Action"] + +from pyquarantine import modify, notify, storage +from pyquarantine.conditions import Conditions + + +class Action: + """Action to implement a pre-configured action to perform on e-mails.""" + ACTION_TYPES = { + "add_header": modify.Modify, + "mod_header": modify.Modify, + "del_header": modify.Modify, + "add_disclaimer": modify.Modify, + "rewrite_links": modify.Modify, + "store": storage.Store, + "notify": notify.Notify, + "quarantine": storage.Quarantine} + + def __init__(self, cfg, local_addrs, debug): + self.cfg = cfg + + self.conditions = cfg["conditions"] if "conditions" in cfg else None + if self.conditions is not None: + self.conditions["name"] = f"{cfg['name']}: conditions" + self.conditions["loglevel"] = cfg["loglevel"] + self.conditions = Conditions(self.conditions, local_addrs, debug) + + action_type = cfg["type"] + self.action = self.ACTION_TYPES[action_type]( + cfg, local_addrs, debug) + + def __str__(self): + cfg = [] + for key in ["name", "loglevel", "pretend", "type"]: + value = self.cfg[key] + cfg.append(f"{key}={value}") + if self.conditions is not None: + cfg.append(f"conditions={self.conditions}") + cfg.append(f"action={self.action}") + return "Action(" + ", ".join(cfg) + ")" + + def headersonly(self): + """Return the needs of this action.""" + return self.action._headersonly + + def execute(self, milter): + """Execute configured action.""" + if self.conditions is None or \ + self.conditions.match(milter): + return self.action.execute(milter) diff --git a/pyquarantine/base.py b/pyquarantine/base.py new file mode 100644 index 0000000..70be625 --- /dev/null +++ b/pyquarantine/base.py @@ -0,0 +1,71 @@ +# pyquarantine is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# pyquarantine is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with pyquarantine. If not, see . +# + +__all__ = [ + "CustomLogger", + "MilterMessage", + "replace_illegal_chars"] + +import logging + +from email.message import MIMEPart + + +class CustomLogger(logging.LoggerAdapter): + def process(self, msg, kwargs): + if "name" in self.extra: + msg = f"{self.extra['name']}: {msg}" + + if "qid" in self.extra: + msg = f"{self.extra['qid']}: {msg}" + + if self.logger.getEffectiveLevel() != logging.DEBUG: + msg = msg.replace("\n", "").replace("\r", "") + + return msg, kwargs + + +class MilterMessage(MIMEPart): + def replace_header(self, _name, _value, idx=None): + _name = _name.lower() + counter = 0 + for i, (k, v) in zip(range(len(self._headers)), self._headers): + if k.lower() == _name: + counter += 1 + if not idx or counter == idx: + self._headers[i] = self.policy.header_store_parse( + k, _value) + break + + else: + raise KeyError(_name) + + def remove_header(self, name, idx=None): + name = name.lower() + newheaders = [] + counter = 0 + for k, v in self._headers: + if k.lower() == name: + counter += 1 + if counter != idx: + newheaders.append((k, v)) + else: + newheaders.append((k, v)) + + self._headers = newheaders + + +def replace_illegal_chars(string): + """Remove illegal characters from header values.""" + return "".join(string.replace("\x00", "").splitlines()) diff --git a/pyquarantine/cli.py b/pyquarantine/cli.py index cba3cb7..d7d733e 100644 --- a/pyquarantine/cli.py +++ b/pyquarantine/cli.py @@ -20,7 +20,7 @@ import logging.handlers import sys import time -from pyquarantine import QuarantineMilter, setup_milter +from pyquarantine.config import get_milter_config from pyquarantine import __version__ as version @@ -304,6 +304,9 @@ class StdOutFilter(logging.Filter): def main(): + python_version = ".".join([str(v) for v in sys.version_info[0:3]]) + python_version = f"{python_version}-{sys.version_info[3]}" + "PyQuarantine command-line interface." # parse command line def formatter_class(prog): return argparse.HelpFormatter( @@ -312,10 +315,8 @@ def main(): description="PyQuarantine CLI", formatter_class=formatter_class) parser.add_argument( - "-c", "--config", - help="Config files to read.", - nargs="+", metavar="CFG", - default=QuarantineMilter.get_cfg_files()) + "-c", "--config", help="Config file to read.", + default="/etc/pyquarantine/pyquarantine.conf") parser.add_argument( "-d", "--debug", help="Log debugging messages.", @@ -324,7 +325,7 @@ def main(): "-v", "--version", help="Print version.", action="version", - version=f"%(prog)s ({version})") + version=f"%(prog)s {version} (python {python_version})") parser.set_defaults(syslog=False) subparsers = parser.add_subparsers( dest="command", @@ -578,14 +579,29 @@ def main(): root_logger.addHandler(stderrhandler) logger = logging.getLogger(__name__) - # try to generate milter configs try: - setup_milter( - cfg_files=args.config, test=True) - except RuntimeError as e: - logger.error(e) + logger.debug("read milter configuration") + cfg = get_milter_config(args.config) + if not cfg["rules"]: + raise RuntimeError("no rules configured") + + for rule in cfg["rules"]: + if not rule["actions"]: + raise RuntimeError( + f"{rule['name']}: no actions configured") + except (RuntimeError, AssertionError) as e: + logger.error(f"config error: {e}") sys.exit(255) + quarantines = [] + for rule in cfg["rules"]: + for action in rule["actions"]: + if action["type"] == "quarantine": + quarantines.append(action) + + print(quarantines) + sys.exit(0) + if args.syslog: # setup syslog sysloghandler = logging.handlers.SysLogHandler( @@ -602,7 +618,7 @@ def main(): # call the commands function try: - args.func(QuarantineMilter.quarantines, args) + args.func(cfg, args) except RuntimeError as e: logger.error(e) sys.exit(1) diff --git a/pyquarantine/conditions.py b/pyquarantine/conditions.py new file mode 100644 index 0000000..3ec4b88 --- /dev/null +++ b/pyquarantine/conditions.py @@ -0,0 +1,193 @@ +# pyquarantine is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# pyquarantine is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with pyquarantine. If not, see . +# + +__all__ = ["Conditions"] + +import logging +import re + +from netaddr import IPAddress, IPNetwork, AddrFormatError +from pyquarantine import CustomLogger +from pyquarantine.whitelist import DatabaseWhitelist + + +class Conditions: + """Conditions to implement conditions for rules and actions.""" + + def __init__(self, cfg, local_addrs, debug): + self.cfg = cfg + self.local_addrs = local_addrs + + self.logger = logging.getLogger(cfg["name"]) + self.logger.setLevel(cfg.get_loglevel(debug)) + + for arg in ("local", "hosts", "envfrom", "envto", "header", "metavar", + "var"): + if arg not in cfg: + setattr(self, arg, None) + continue + + if arg == "hosts": + try: + self.hosts = [] + for host in cfg["hosts"]: + self.hosts.append(IPNetwork(host)) + except AddrFormatError as e: + raise RuntimeError(e) + elif arg in ("envfrom", "envto"): + try: + setattr(self, arg, re.compile( + cfg[arg], re.IGNORECASE)) + except re.error as e: + raise RuntimeError(e) + elif arg == "header": + try: + self.header = re.compile( + cfg["header"], + re.IGNORECASE + re.DOTALL + re.MULTILINE) + except re.error as e: + raise RuntimeError(e) + else: + setattr(self, arg, cfg[arg]) + + self.whitelist = cfg["whitelist"] if "whitelist" in cfg else None + if self.whitelist is not None: + self.whitelist["name"] = f"{cfg['name']}: whitelist" + self.whitelist["loglevel"] = cfg["loglevel"] + if self.whitelist["type"] == "db": + self.whitelist = DatabaseWhitelist(self.whitelist, debug) + else: + raise RuntimeError("invalid whitelist type") + + def __str__(self): + cfg = [] + for arg in ("local", "hosts", "envfrom", "envto", "header", + "var", "metavar"): + if arg in self.cfg: + cfg.append(f"{arg}={self.cfg[arg]}") + if self.whitelist is not None: + cfg.append(f"whitelist={self.whitelist}") + return "Conditions(" + ", ".join(cfg) + ")" + + def match_host(self, host): + logger = CustomLogger( + self.logger, {"name": self.cfg["name"]}) + ip = IPAddress(host) + + if self.local is not None: + is_local = False + for addr in self.local_addrs: + if ip in addr: + is_local = True + break + + if is_local != self.local: + logger.debug( + f"ignore host {host}, " + f"local does not match") + return False + + logger.debug( + f"local matches for host {host}") + + if self.hosts is not None: + found = False + for addr in self.hosts: + if ip in addr: + found = True + break + + if not found: + logger.debug( + f"ignore host {host}, " + f"hosts does not match") + return False + + logger.debug( + f"hosts matches for host {host}") + + return True + + def get_wl_rcpts(self, mailfrom, rcpts, logger): + if not self.whitelist: + return {} + + wl_rcpts = [] + for rcpt in rcpts: + if self.whitelist.check(mailfrom, rcpt, logger): + wl_rcpts.append(rcpt) + + return wl_rcpts + + def match(self, milter): + logger = CustomLogger( + self.logger, {"qid": milter.qid, "name": self.cfg["name"]}) + + if self.envfrom is not None: + envfrom = milter.msginfo["mailfrom"] + if not self.envfrom.match(envfrom): + logger.debug( + f"ignore envelope-from address {envfrom}, " + f"envfrom does not match") + return False + + logger.debug( + f"envfrom matches for " + f"envelope-from address {envfrom}") + + if self.envto is not None: + envto = milter.msginfo["rcpts"] + if not isinstance(envto, list): + envto = [envto] + + for to in envto: + if not self.envto.match(to): + logger.debug( + f"ignore envelope-to address {envto}, " + f"envto does not match") + return False + + logger.debug( + f"envto matches for " + f"envelope-to address {envto}") + + if self.header is not None: + match = None + for field, value in milter.msg.items(): + header = f"{field}: {value}" + match = self.header.search(header) + if match: + logger.debug( + f"header matches for " + f"header: {header}") + if self.metavar is not None: + named_subgroups = match.groupdict(default=None) + for group, value in named_subgroups.items(): + if value is None: + continue + name = f"{self.metavar}_{group}" + milter.msginfo["vars"][name] = value + break + + if not match: + logger.debug( + "ignore message, " + "header does not match") + return False + + if self.var is not None: + if self.var not in milter.msginfo["vars"]: + return False + + return True diff --git a/pyquarantine/config.py b/pyquarantine/config.py new file mode 100644 index 0000000..c729ad3 --- /dev/null +++ b/pyquarantine/config.py @@ -0,0 +1,354 @@ +# pyquarantine is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# pyquarantine is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with pyquarantine. If not, see . +# + +__all__ = [ + "BaseConfig", + "ConditionsConfig", + "AddHeaderConfig", + "ModHeaderConfig", + "DelHeaderConfig", + "AddDisclaimerConfig", + "RewriteLinksConfig", + "StoreConfig", + "NotifyConfig", + "WhitelistConfig", + "QuarantineConfig", + "ActionConfig", + "RuleConfig", + "MilterConfig", + "get_milter_config"] + +import json +import jsonschema +import logging +import re + + +class BaseConfig: + JSON_SCHEMA = { + "type": "object", + "required": [], + "additionalProperties": True, + "properties": { + "loglevel": {"type": "string", "default": "info"}}} + + def __init__(self, config): + required = self.JSON_SCHEMA["required"] + properties = self.JSON_SCHEMA["properties"] + for p in properties.keys(): + if p in required: + continue + elif p not in config and "default" in properties[p]: + config[p] = properties[p]["default"] + try: + jsonschema.validate(config, self.JSON_SCHEMA) + except jsonschema.exceptions.ValidationError as e: + raise RuntimeError(e) + + self._config = config + + def __getitem__(self, key): + return self._config[key] + + def __setitem__(self, key, value): + self._config[key] = value + + def __delitem__(self, key): + del self._config[key] + + def __contains__(self, key): + return key in self._config + + def keys(self): + return self._config.keys() + + def items(self): + return self._config.items() + + def get_loglevel(self, debug): + if debug: + level = logging.DEBUG + else: + level = getattr(logging, self["loglevel"].upper(), None) + assert isinstance(level, int), \ + "loglevel: invalid value" + return level + + def get_config(self): + return self._config + + +class WhitelistConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": ["type"], + "additionalProperties": True, + "properties": { + "type": {"enum": ["db"]}}, + "if": {"properties": {"type": {"const": "db"}}}, + "then": { + "required": ["connection", "table"], + "additionalProperties": False, + "properties": { + "type": {"type": "string"}, + "connection": {"type": "string"}, + "table": {"type": "string"}}}} + + +class ConditionsConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": [], + "additionalProperties": False, + "properties": { + "metavar": {"type": "string"}, + "local": {"type": "boolean"}, + "hosts": {"type": "array", + "items": {"type": "string"}}, + "envfrom": {"type": "string"}, + "envto": {"type": "string"}, + "header": {"type": "string"}, + "var": {"type": "string"}, + "whitelist": {"type": "object"}}} + + def __init__(self, config, rec=True): + super().__init__(config) + if rec: + if "whitelist" in self: + self["whitelist"] = WhitelistConfig(self["whitelist"]) + + +class AddHeaderConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": ["field", "value"], + "additionalProperties": False, + "properties": { + "field": {"type": "string"}, + "value": {"type": "string"}}} + + +class ModHeaderConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": ["field", "value"], + "additionalProperties": False, + "properties": { + "field": {"type": "string"}, + "value": {"type": "string"}, + "search": {"type": "string"}}} + + +class DelHeaderConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": ["field"], + "additionalProperties": False, + "properties": { + "field": {"type": "string"}, + "value": {"type": "string"}}} + + +class AddDisclaimerConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": ["action", "html_template", "text_template"], + "additionalProperties": False, + "properties": { + "action": {"type": "string"}, + "html_template": {"type": "string"}, + "text_template": {"type": "string"}, + "error_policy": {"type": "string"}}} + + +class RewriteLinksConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": ["repl"], + "additionalProperties": False, + "properties": { + "repl": {"type": "string"}}} + + +class StoreConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": ["type"], + "additionalProperties": True, + "properties": { + "type": {"enum": ["file"]}}, + "if": {"properties": {"type": {"const": "file"}}}, + "then": { + "required": ["directory"], + "additionalProperties": False, + "properties": { + "type": {"type": "string"}, + "directory": {"type": "string"}, + "mode": {"type": "string"}, + "metavar": {"type": "string"}, + "original": {"type": "boolean", "default": True}}}} + + +class NotifyConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": ["type"], + "additionalProperties": True, + "properties": { + "type": {"enum": ["email"]}}, + "if": {"properties": {"type": {"const": "email"}}}, + "then": { + "required": ["smtp_host", "smtp_port", "envelope_from", + "from_header", "subject", "template"], + "additionalProperties": False, + "properties": { + "type": {"type": "string"}, + "smtp_host": {"type": "string"}, + "smtp_port": {"type": "number"}, + "envelope_from": {"type": "string"}, + "from_header": {"type": "string"}, + "subject": {"type": "string"}, + "template": {"type": "string"}, + "repl_img": {"type": "string"}, + "embed_imgs": { + "type": "array", + "items": {"type": "string"}, + "default": True}}}} + + +class QuarantineConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": ["store"], + "additionalProperties": False, + "properties": { + "name": {"type": "string"}, + "notify": {"type": "object"}, + "milter_action": {"type": "string"}, + "reject_reason": {"type": "string"}, + "whitelist": {"type": "object"}, + "store": {"type": "object"}}} + + def __init__(self, config, rec=True): + super().__init__(config) + if rec: + self["store"] = StoreConfig(self["store"]) + if "notify" in self: + self["notify"] = NotifyConfig(self["notify"]) + if "whitelist" in self: + self["whitelist"] = ConditionsConfig( + {"whitelist": self["whitelist"]}, rec) + + +class ActionConfig(BaseConfig): + ACTION_TYPES = { + "add_header": AddHeaderConfig, + "mod_header": ModHeaderConfig, + "del_header": DelHeaderConfig, + "add_disclaimer": AddDisclaimerConfig, + "rewrite_links": RewriteLinksConfig, + "store": StoreConfig, + "notify": NotifyConfig, + "quarantine": QuarantineConfig} + + JSON_SCHEMA = { + "type": "object", + "required": ["type", "args"], + "additionalProperties": False, + "properties": { + "name": {"type": "string", "default": "action"}, + "loglevel": {"type": "string", "default": "info"}, + "pretend": {"type": "boolean", "default": False}, + "conditions": {"type": "object"}, + "type": {"enum": list(ACTION_TYPES.keys())}, + "args": {"type": "object"}}} + + def __init__(self, config, rec=True): + super().__init__(config) + if rec: + if "conditions" in self: + self["conditions"] = ConditionsConfig(self["conditions"]) + self["action"] = self.ACTION_TYPES[self["type"]](self["args"]) + + +class RuleConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": ["actions"], + "additionalProperties": False, + "properties": { + "name": {"type": "string", "default": "rule"}, + "loglevel": {"type": "string", "default": "info"}, + "pretend": {"type": "boolean", "default": False}, + "conditions": {"type": "object"}, + "actions": {"type": "array"}}} + + def __init__(self, config, rec=True): + super().__init__(config) + if rec: + if "conditions" in self: + self["conditions"] = ConditionsConfig(self["conditions"]) + + actions = [] + for idx, action in enumerate(self["actions"]): + actions.append(ActionConfig(action, rec)) + self["actions"] = actions + + +class MilterConfig(BaseConfig): + JSON_SCHEMA = { + "type": "object", + "required": ["rules"], + "additionalProperties": False, + "properties": { + "socket": {"type": "string"}, + "local_addrs": {"type": "array", + "items": {"type": "string"}, + "default": [ + "fe80::/64", + "::1/128", + "127.0.0.0/8", + "10.0.0.0/8", + "172.16.0.0/12", + "192.168.0.0/16"]}, + "loglevel": {"type": "string", "default": "info"}, + "pretend": {"type": "boolean", "default": False}, + "rules": {"type": "array"}}} + + def __init__(self, config, rec=True): + super().__init__(config) + if rec: + rules = [] + for idx, rule in enumerate(self["rules"]): + rules.append(RuleConfig(rule, rec)) + self["rules"] = rules + + +def get_milter_config(cfgfile): + try: + with open(cfgfile, "r") as fh: + # remove lines with leading # (comments), they + # are not allowed in json + cfg = re.sub(r"(?m)^\s*#.*\n?", "", fh.read()) + except IOError as e: + raise RuntimeError(f"unable to open/read config file: {e}") + + try: + cfg = json.loads(cfg) + except json.JSONDecodeError as e: + cfg_text = [f"{n+1}: {l}" for n, l in enumerate(cfg.splitlines())] + msg = "\n".join(cfg_text) + raise RuntimeError(f"{e}\n{msg}") + + return MilterConfig(cfg) diff --git a/pyquarantine/docs/pyquarantine.conf.example b/pyquarantine/docs/pyquarantine.conf.example index 5105d25..c062f75 100644 --- a/pyquarantine/docs/pyquarantine.conf.example +++ b/pyquarantine/docs/pyquarantine.conf.example @@ -1,180 +1,221 @@ # This is an example /etc/pyquarantine/pyquarantine.conf file. # Copy it into place before use. # -# Comments: use '#' for comment lines and ';' (following a space) for inline comments. +# The file is in JSON format. # -# If an option is not present in a quarantine section, it will be read from -# the global section. +# The global option 'log' can be overriden per rule or per modification. # +{ + # Section: global + # Notes: Global options. + # + "global": { + # Option: socket + # Type: String + # Notes: The socket used to communicate with the MTA. + # + # Examples: + # unix:/path/to/socket a named pipe + # inet:8899 listen on ANY interface + # inet:8899@localhost listen on a specific interface + # inet6:8899 listen on ANY interface + # inet6:8899@[2001:db8:1234::1] listen on a specific interface + # Value: [ SOCKET ] + "socket": "inet:8898@127.0.0.1", + # Option: local_addrs + # Type: List + # Notes: A list of local hosts and networks. + # Value: [ LIST ] + # + "local_addrs": ["fe80::/64", "::1/128", "127.0.0.0/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"], -[global] + # Option: loglevel + # Type: String + # Notes: Set loglevel for rules and actions. + # Value: [ error | warning | info | debug ] + # + "loglevel": "info", -# Option: quarantines -# Notes: Set active quarantines (comma-separated). -# Each active quarantine must have a section with the same name below. -# The quarantine name 'global' is forbidden and will be ignored. -# Quarantine names must be unique. -# Values: [ ACTIVE ] -# -quarantines = spam + # Option: pretend + # Type: Bool + # Notes: Just pretend to do the actions, for test purposes. + # Value: [ true | false ] + # + "pretend": true + }, -# Option: quarantine_action_precedence -# Notes: Set if the action of the first or the last matching quarantine should -# be used if multiple recipients match multiple quarantines. If an original -# email is delivered to at least one recipient due to whitelisting, the -# email will always be accepted. -# Values: [ first | last ] -# -preferred_quarantine_action = last + # Section: rules + # Notes: Rules and related actions. + # + "rules": [ + { + # Option: name + # Type: String + # Notes: Name of the rule. + # Value: [ NAME ] + # + "name": "myrule", + # Section: conditions + # Notes: Optional conditions to process the rule. + # If multiple conditions are set, they all + # have to be true to process the rule. + # + "conditions": { + # Option: local + # Type: Bool + # Notes: Condition wheter the senders host address is listed in local_addrs. + # Value: [ true | false ] + # + "local": false, -[spam] -# Option: host_whitelist -# Notes: Set a list of host and network addresses to be ignored by this quarantine. -# All the common host/network notations are supported, including IPv6. -# Value: [ HOST ] -# -host_whitelist = 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16 + # Option: hosts + # Type: String + # Notes: Condition wheter the senders host address is listed in this list. + # Value: [ LIST ] + # + "hosts": [ "127.0.0.1" ], -# Option: regex -# Notes: Set the case insensitive regular expression to match against email headers. -# If the regex matches any of the email headers, the email -# will be processed by this quarantine. -# Values: [ REGEX ] -# -regex = ^X-Spam-Flag: YES + # Option: envfrom + # Type: String + # Notes: Condition wheter the envelop-from address matches this regular expression. + # Value: [ REGEX ] + # + "envfrom": "^.+@mypartner\\.com$", -# Option: smtp_host -# Notes: Set the SMTP host. It will be used to (re-)inject emails. -# Values: [ HOSTNAME | IP_ADDRESS ] -# -smtp_host = 127.0.0.1 + # Option: envto + # Type: String + # Notes: Condition wheter the envelop-to address matches this regular expression. + # Value: [ REGEX ] + # + "envto": "^postmaster@.+$" + }, -# Option: smtp_port -# Notes: Set the SMTP port. -# Values: [ PORT ] -# -smtp_port = 25 + # Section: actions + # Notes: Actions of the rule. + # + "actions": [ + { + # Option: name + # Type: String + # Notes: Name of the modification. + # Value: [ NAME ] + # + "name": "add_test_header", -# Option: storage_type -# Notes: Set the storage type. -# Values: [ file | none ] -# -storage_type = file + # Option: type + # Type: String + # Notes: Type of the modification. + # Value: [ add_header | del_header | mod_header ] + # + "type": "add_header", -# Option: storage_directory -# Notes: Set the directory to store quarantined emails. -# This option is needed by quarantine type 'file'. -# Values: [ DIRECTORY ] -# -storage_directory = /var/lib/pyquarantine/spam + # Option: field + # Type: String + # Notes: Name of the header. + # Value: [ NAME ] + # + "field": "X-Test-Header", -# Option: action -# Notes: Set the milter action to perform if email is processed by this quarantine. -# Values: [ accept | discard | reject ] -# -action = discard + # Option: value + # Type: String + # Notes: Value of the header. + # Value: [ VALUE ] + # + "value": "true" + }, { + "name": "modify_subject", -# Option: reject_reason -# Notes: Optionally set the reason to return if action is set to reject. -# Values: [ REASON ] -# -reject_reason = Message rejected + "type": "mod_header", -# Option: notification -# Notes: Set the notification type. -# Values: [ email | none ] -# -notification_type = email + # Option: field + # Type: String + # Notes: Regular expression to match against header lines (e.g. Subject: Test-Subject). + # Value: [ REGEX ] + # + "field": "^Subject$", -# Option: notification_email_smtp_host -# Notes: Set the SMTP host. It will be used to send notification e-mails. -# Values: [ HOSTNAME | IP_ADDRESS ] -# -notification_email_smtp_host = 127.0.0.1 + # Option: search + # Type: String + # Notes: Regular expression to match against the headers value. + # Values: [ VALUE ] + # + "search": "(?P.*)", -# Option: notification_email_smtp_port -# Notes: Set the SMTP port. -# Values: [ PORT ] -# -notification_email_smtp_port = 25 + # Option: value + # Type: String + # Notes: New value of the header. + # Values: [ VALUE ] + "value": "[EXTERNAL] \\g" + }, { + "name": "delete_received_header", -# Option: notification_email_envelope_from -# Notes: Set the envelope-from address used when sending notification emails. -# This option is needed by notification type 'email'. -# Values: [ ENVELOPE_FROM_ADDRESS ] -# -notification_email_envelope_from = notification@domain.tld + "type": "del_header", -# Option: notification_email_from -# Notes: Set the from header used when sending notification emails. -# This option is needed by notification type 'email'. -# Values: [ FROM_HEADER ] -# -notification_email_from = Notification + # Option: field + # Type: String + # Notes: Regular expression to match against header lines (e.g. Subject: Test-Subject). + # Value: [ REGEX ] + # + "field": "^Received$" + }, { + "name": "add_disclaimer", -# Option: notification_email_usbject -# Notes: Set the subject used when sending notification emails. -# This option is needed by notification type 'email'. -# Values: [ SUBJECT ] -# -notification_email_subject = Spam Quarantine Notification + "type": "add_disclaimer", -# Option: notification_email_template -# Notes: Set the template used when sending notification emails. -# A relative path to this config file can be used. -# This option is needed by notification type 'email'. -# Values: [ TEMPLATE_PATH ] -# -notification_email_template = templates/notification.template + # Option: action + # Type: String + # Notes: Action to perform with the disclaimer. + # Value: [ append | prepend ] + # + "action": "prepend", -# Option: notification_email_strip_images -# Notes: Optionally enable this option to strip img tags from emails. -# Values: [ TRUE | ON | YES | FALSE | OFF | NO ] -# -notification_email_strip_images = False + # Option: html_template + # Type: String + # Notes: Path to a file which contains the html representation of the disclaimer. + # Value: [ FILE_PATH ] + # + "html_template": "/etc/pyquarantine/templates/disclaimer_html.template", -# Option: notification_email_replacement_img -# Notes: Optionally set the path to a replacement image for img tags within emails. -# A relative path to this config file can be used. -# Values: [ IMAGE_PATH ] -# -notification_email_replacement_img = templates/removed.png + # Option: text_template + # Type: String + # Notes: Path to a file which contains the text representation of the disclaimer. + # Value: [ FILE_PATH ] + # + "text_template": "/etc/pyquarantine/templates/disclaimer_text.template", -# Option: notification_email_embedded_imgs -# Notes: Set a list of paths to images to embed in e-mails (comma-separated). -# Relative paths to this config file can be used. -# This option is needed by notification type 'email'. -# Values: [ IMAGE_PATH ] -# -notification_email_embedded_imgs = templates/logo.png + # Option: error_policy + # Type: String + # Notes: Set what should be done if the modification fails (e.g. no message body present). + # Value: [ wrap | ignore | reject ] + # + "error_policy": "wrap" + }, { + "name": "store_message", -# Option: notification_email_parser_lib -# Notes: Optionally set the parser library used to parse -# the text part of emails. -# Values: [ lxml | html.parser ] -# -notification_email_parser_lib = lxml + "type": "store", -# Option: whitelist_type -# Notes: Set the whitelist type. -# Values: [ db | none ] -# -whitelist_type = db + # Option: storage_type + # Type: String + # Notes: The storage type used to store e-mails. + # Value: [ file ] + "storage_type": "file", -# Option: whitelist_db_connection -# Notes: Set the connection string to connect to the database. -# The configured user must have read/write access to -# the whitelist_db_table configured below. -# This option is needed by whitelist type 'db'. -# Values: [ DB_CONNECTION_STRING | none ] -# -whitelist_db_connection = mysql://user:password@localhost/database + # Option: directory + # Type: String + # Notes: Directory used to store e-mails. + # Value: [ file ] + "directory": "/mnt/messages", -# Option: whitelist_db_table -# Notes: Set the database table name. -# This option is needed by whitelist type 'db'. -# Values: [ DATABASE_TABLE] -# -whitelist_db_table = whitelist + # Option: original + # Type: Bool + # Notes: If set to true, store the message as received by the MTA instead of storing the current state + # of the message, that may was modified already by other actions. + # Value: [ true | false ] + "original": true + } + ] + } + ] +} diff --git a/pyquarantine/docs/templates/disclaimer_html.template b/pyquarantine/docs/templates/disclaimer_html.template new file mode 100644 index 0000000..827f148 --- /dev/null +++ b/pyquarantine/docs/templates/disclaimer_html.template @@ -0,0 +1,9 @@ + + + + +
+ CAUTION: This email originated from outside the organization. + Do not follow guidance, click links or open attachments unless you recognize the sender and know the content is safe. +
+

diff --git a/pyquarantine/docs/templates/disclaimer_text.template b/pyquarantine/docs/templates/disclaimer_text.template new file mode 100644 index 0000000..6d9532d --- /dev/null +++ b/pyquarantine/docs/templates/disclaimer_text.template @@ -0,0 +1,4 @@ + +CAUTION: This email originated from outside the organization. Do not follow guidance, click links or open attachments unless you recognize the sender and know the content is safe. + + diff --git a/pyquarantine/docs/templates/notification.template b/pyquarantine/docs/templates/notification.template index 46a4d64..f41a4ad 100644 --- a/pyquarantine/docs/templates/notification.template +++ b/pyquarantine/docs/templates/notification.template @@ -4,30 +4,26 @@ - + - + - + - + - - - - - +
Envelope-From:{EMAIL_ENVELOPE_FROM}{ENVELOPE_FROM}
From:{EMAIL_FROM}{FROM}
Envelope-To:{EMAIL_ENVELOPE_TO}{ENVELOPE_TO}
To:{EMAIL_TO}{TO}
Subject:{EMAIL_SUBJECT}
Quarantine ID:{EMAIL_QUARANTINE_ID}{SUBJECT}

Preview of the original e-mail

- {EMAIL_HTML_TEXT} + {HTML_TEXT} diff --git a/pyquarantine/mailer.py b/pyquarantine/mailer.py index 803dd64..b5bea6c 100644 --- a/pyquarantine/mailer.py +++ b/pyquarantine/mailer.py @@ -1,15 +1,15 @@ -# PyQuarantine-Milter is free software: you can redistribute it and/or modify +# pyquarantine is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # -# PyQuarantine-Milter is distributed in the hope that it will be useful, +# pyquarantine is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License -# along with PyQuarantineMilter. If not, see . +# along with pyquarantine. If not, see . # import logging diff --git a/pyquarantine/misc/openrc/pyquarantine-milter.confd b/pyquarantine/misc/openrc/pyquarantine-milter.confd index 38d1d35..277e442 100644 --- a/pyquarantine/misc/openrc/pyquarantine-milter.confd +++ b/pyquarantine/misc/openrc/pyquarantine-milter.confd @@ -1,17 +1,8 @@ -# /etc/conf.d/pyquarantine-milter: config file for /etc/init.d/pyquarantine-milter - -# Set the socket used to communicate with the MTA. -# Examples: -# unix:/path/to/socket a named pipe -# inet:8899 listen on ANY interface -# inet:8899@localhost listen on a specific interface -# inet6:8899 listen on ANY interface -# inet6:8899@[2001:db8:1234::1] listen on a specific interface -SOCKET="inet:8899@127.0.0.1" +# /etc/conf.d/pyquarantine: config file for /etc/init.d/pyquarantine # Start the daemon as the user. You can optionally append a group name here also. # USER="daemon" # USER="daemon:nobody" -# Optional parameters for pyquarantine-milter +# Optional parameters for pyquarantine # MILTER_OPTS="" diff --git a/pyquarantine/misc/openrc/pyquarantine-milter.initd b/pyquarantine/misc/openrc/pyquarantine-milter.initd index 5ab34fe..8bf697f 100755 --- a/pyquarantine/misc/openrc/pyquarantine-milter.initd +++ b/pyquarantine/misc/openrc/pyquarantine-milter.initd @@ -1,16 +1,15 @@ #!/sbin/openrc-run user=${USER:-daemon} -socket="${SOCKET:-}" milter_opts="${MILTER_OPTS:-}" pidfile="/run/${RC_SVCNAME}.pid" command="/usr/bin/pyquarantine-milter" -command_args="-s ${socket} ${milter_opts}" +command_args="${milter_opts}" command_background=true -start_stop_daemon_args="--user ${user}" - +command_user="${user}" extra_commands="configtest" +start_stop_daemon_args="--wait 500" depend() { need net @@ -18,9 +17,6 @@ depend() { } checkconfig() { - if [ -z "${socket}" ]; then - eerror "No socket specified in config!" - fi OUTPUT=$( ${command} ${command_args} -t 2>&1 ) ret=$? if [ $ret -ne 0 ]; then @@ -44,7 +40,7 @@ start_pre() { } stop_pre() { - if [ "${RC_CMD}" != "restart" ]; then + if [ "${RC_CMD}" == "restart" ]; then checkconfig || return $? fi } diff --git a/pyquarantine/modify.py b/pyquarantine/modify.py new file mode 100644 index 0000000..f57e460 --- /dev/null +++ b/pyquarantine/modify.py @@ -0,0 +1,391 @@ +# pyquarantine is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# pyquarantine is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with pyquarantine. If not, see . +# + +__all__ = [ + "AddHeader", + "ModHeader", + "DelHeader", + "AddDisclaimer", + "RewriteLinks", + "Modify"] + +import logging +import re + +from base64 import b64encode +from bs4 import BeautifulSoup +from collections import defaultdict +from copy import copy +from email.message import MIMEPart +from email.policy import SMTPUTF8 + +from pyquarantine import replace_illegal_chars +from pyquarantine.base import CustomLogger + + +class AddHeader: + """Add a mail header field.""" + _headersonly = True + + def __init__(self, field, value, pretend=False): + self.field = field + self.value = value + self.pretend = pretend + + def execute(self, milter, logger): + header = f"{self.field}: {self.value}" + if logger.getEffectiveLevel() == logging.DEBUG: + logger.debug(f"add_header: {header}") + else: + logger.info(f"add_header: {header[0:70]}") + + milter.msg.add_header(self.field, self.value) + if not self.pretend: + milter.addheader(self.field, self.value) + + +class ModHeader: + """Change the value of a mail header field.""" + _headersonly = True + + def __init__(self, field, value, search=None, pretend=False): + try: + self.field = re.compile(field, re.IGNORECASE) + self.search = search + if self.search is not None: + self.search = re.compile( + self.search, re.MULTILINE + re.DOTALL + re.IGNORECASE) + + except re.error as e: + raise RuntimeError(e) + + self.value = value + self.pretend = pretend + + def execute(self, milter, logger): + idx = defaultdict(int) + + for i, (field, value) in enumerate(milter.msg.items()): + field_lower = field.lower() + idx[field_lower] += 1 + + if not self.field.match(field): + continue + + new_value = value + if self.search is not None: + new_value = self.search.sub(self.value, value).strip() + else: + new_value = self.value + + if not new_value: + logger.warning( + "mod_header: resulting value is empty, " + "skip modification") + continue + + if new_value == value: + continue + + header = f"{field}: {value}" + new_header = f"{field}: {new_value}" + + if logger.getEffectiveLevel() == logging.DEBUG: + logger.debug(f"mod_header: {header}: {new_header}") + else: + logger.info( + f"mod_header: {header[0:70]}: {new_header[0:70]}") + + milter.msg.replace_header( + field, replace_illegal_chars(new_value), idx=idx[field_lower]) + + if not self.pretend: + milter.chgheader(field, new_value, idx=idx[field_lower]) + + +class DelHeader: + """Delete a mail header field.""" + _headersonly = True + + def __init__(self, field, value=None, pretend=False): + try: + self.field = re.compile(field, re.IGNORECASE) + self.value = value + if self.value is not None: + self.value = re.compile( + value, re.MULTILINE + re.DOTALL + re.IGNORECASE) + except re.error as e: + raise RuntimeError(e) + + self.pretend = pretend + + def execute(self, milter, logger): + idx = defaultdict(int) + + for field, value in milter.msg.items(): + field_lower = field.lower() + idx[field_lower] += 1 + + if not self.field.match(field): + continue + + if self.value is not None and not self.value.search(value): + continue + + header = f"{field}: {value}" + if logger.getEffectiveLevel() == logging.DEBUG: + logger.debug(f"del_header: {header}") + else: + logger.info(f"del_header: {header[0:70]}") + milter.msg.remove_header(field, idx=idx[field_lower]) + + if not self.pretend: + milter.chgheader(field, "", idx=idx[field_lower]) + + idx[field_lower] -= 1 + + +def _get_body_content(msg, pref): + part = None + content = None + if not msg.is_multipart() and msg.get_content_type() == f"text/{pref}": + part = msg + else: + part = msg.get_body(preferencelist=(pref)) + + if part is not None: + content = part.get_content() + + return (part, content) + + +def _has_content_before_body_tag(soup): + s = copy(soup) + for element in s.find_all("head") + s.find_all("body"): + element.extract() + + if len(s.text.strip()) > 0: + return True + + return False + + +def _inject_body(milter): + if not milter.msg.is_multipart(): + milter.msg.make_mixed() + + attachments = [] + for attachment in milter.msg.iter_attachments(): + if "content-disposition" not in attachment: + attachment["Content-Disposition"] = "attachment" + attachments.append(attachment) + + milter.msg.clear_content() + milter.msg.set_content("") + milter.msg.add_alternative("", subtype="html") + milter.msg.make_mixed() + + for attachment in attachments: + milter.msg.attach(attachment) + + +def _wrap_message(milter): + attachment = MIMEPart(policy=SMTPUTF8) + attachment.set_content(milter.msg.as_bytes(), + maintype="plain", subtype="text", + disposition="attachment", + filename=f"{milter.qid}.eml", + params={"name": f"{milter.qid}.eml"}) + + milter.msg.clear_content() + milter.msg.set_content( + "Please see the original email attached.") + milter.msg.add_alternative( + "Please see the original email attached.", + subtype="html") + milter.msg.make_mixed() + milter.msg.attach(attachment) + + +class AddDisclaimer: + """Append or prepend a disclaimer to the mail body.""" + _headersonly = False + + def __init__(self, text_template, html_template, action, error_policy, + pretend=False): + self.text_template_path = text_template + self.html_template_path = html_template + try: + with open(text_template, "r") as f: + self.text_template = f.read() + + with open(html_template, "r") as f: + html = BeautifulSoup(f.read(), "html.parser") + + except IOError as e: + raise RuntimeError(e) + body = html.find('body') + self.html_template = body or html + self.action = action + self.error_policy = error_policy + self.pretend = pretend + + def patch_message_body(self, milter, logger): + text_body, text_content = _get_body_content(milter.msg, "plain") + html_body, html_content = _get_body_content(milter.msg, "html") + + if text_content is None and html_content is None: + raise RuntimeError("message does not contain any body part") + + if text_content is not None: + logger.info(f"{self.action} text disclaimer") + + if self.action == "prepend": + content = f"{self.text_template}{text_content}" + else: + content = f"{text_content}{self.text_template}" + + text_body.set_content( + content.encode(), maintype="text", subtype="plain") + text_body.set_param("charset", "UTF-8", header="Content-Type") + del text_body["MIME-Version"] + + if html_content is not None: + logger.info(f"{self.action} html disclaimer") + + soup = BeautifulSoup(html_content, "html.parser") + + body = soup.find('body') + if not body: + body = soup + elif _has_content_before_body_tag(soup): + body = soup + + if self.action == "prepend": + body.insert(0, copy(self.html_template)) + else: + body.append(self.html_template) + + html_body.set_content( + str(soup).encode(), maintype="text", subtype="html") + html_body.set_param("charset", "UTF-8", header="Content-Type") + del html_body["MIME-Version"] + + def execute(self, milter, logger): + old_headers = milter.msg.items() + + try: + try: + self.patch_message_body(milter, logger) + except RuntimeError as e: + logger.info(f"{e}, inject empty plain and html body") + _inject_body(milter) + self.patch_message_body(milter, logger) + + except Exception as e: + logger.warning(e) + if self.error_policy == "ignore": + logger.info( + "unable to add disclaimer to message body, " + "ignore error according to policy") + return + elif self.error_policy == "reject": + logger.info( + "unable to add disclaimer to message body, " + "reject message according to policy") + return [ + ("reject", "Message rejected due to error")] + + logger.info("wrap original message in a new message envelope") + try: + _wrap_message(milter) + self.patch_message_body(milter, logger) + except Exception as e: + logger.error(e) + raise Exception( + "unable to wrap message in a new message envelope, " + "give up ...") + + if not self.pretend: + milter.update_headers(old_headers) + milter.replacebody() + + +class RewriteLinks: + """Rewrite link targets in the mail html body.""" + _headersonly = False + + def __init__(self, repl, pretend=False): + self.repl = repl + self.pretend = pretend + + def execute(self, milter, logger): + html_body, html_content = _get_body_content(milter.msg, "html") + if html_content is not None: + soup = BeautifulSoup(html_content, "html.parser") + + rewritten = 0 + for link in soup.find_all("a", href=True): + if not link["href"]: + continue + + if "{URL_B64}" in self.repl: + url_b64 = b64encode(link["href"].encode()).decode() + target = self.repl.replace("{URL_B64}", url_b64) + else: + target = self.repl + + link["href"] = target + rewritten += 1 + + if rewritten: + logger.info(f"rewrote {rewritten} link(s) in html body") + + html_body.set_content( + str(soup).encode(), maintype="text", subtype="html") + html_body.set_param("charset", "UTF-8", header="Content-Type") + del html_body["MIME-Version"] + + if not self.pretend: + milter.replacebody() + + +class Modify: + MODIFICATION_TYPES = { + "add_header": AddHeader, + "mod_header": ModHeader, + "del_header": DelHeader, + "add_disclaimer": AddDisclaimer, + "rewrite_links": RewriteLinks} + + def __init__(self, cfg, local_addrs, debug): + self.cfg = cfg + self.logger = logging.getLogger(cfg["name"]) + self.logger.setLevel(cfg.get_loglevel(debug)) + cfg["args"]["pretend"] = cfg["pretend"] + self._modification = self.MODIFICATION_TYPES[cfg["type"]]( + **cfg["args"]) + self._headersonly = self._modification._headersonly + + def __str__(self): + cfg = [] + for key, value in self.cfg["args"].items(): + cfg.append(f"{key}={value}") + class_name = type(self._modification).__name__ + return f"{class_name}(" + ", ".join(cfg) + ")" + + def execute(self, milter): + logger = CustomLogger( + self.logger, {"name": self.cfg["name"], "qid": milter.qid}) + self._modification.execute(milter, logger) diff --git a/pyquarantine/notify.py b/pyquarantine/notify.py index b6fcf82..56ca4ba 100644 --- a/pyquarantine/notify.py +++ b/pyquarantine/notify.py @@ -1,24 +1,28 @@ -# PyQuarantine-Milter is free software: you can redistribute it and/or modify +# pyquarantine is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # -# PyQuarantine-Milter is distributed in the hope that it will be useful, +# pyquarantine is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License -# along with PyQuarantineMilter. If not, see . +# along with pyquarantine. If not, see . # +__all__ = [ + "BaseNotification", + "EMailNotification", + "Notify"] + import email import logging import re from bs4 import BeautifulSoup from collections import defaultdict -from email.policy import default as default_policy from email.mime.multipart import MIMEMultipart from email.mime.text import MIMEText from email.mime.image import MIMEImage @@ -26,26 +30,24 @@ from html import escape from os.path import basename from urllib.parse import quote +from pyquarantine.base import CustomLogger from pyquarantine import mailer -class BaseNotification(object): +class BaseNotification: "Notification base class" - notification_type = "base" + _headersonly = True - def __init__(self, name, global_cfg, cfg, test=False): - self.name = name - self.logger = logging.getLogger(__name__) + def __init__(self, pretend=False): + self.pretend = pretend - def notify(self, qid, storage_id, mailfrom, recipients, headers, - fp, subgroups=None, named_subgroups=None, synchronous=False): - fp.seek(0) - pass + def execute(self, milter, logger): + return class EMailNotification(BaseNotification): "Notification class to send notifications via mail." - notification_type = "email" + _headersonly = False _bad_tags = [ "applet", "embed", @@ -109,122 +111,48 @@ class EMailNotification(BaseNotification): "width" ] - def __init__(self, name, global_cfg, cfg, test=False): - super(EMailNotification, self).__init__( - name, global_cfg, cfg, test) - - defaults = { - "notification_email_replacement_img": "", - "notification_email_strip_images": "false", - "notification_email_parser_lib": "lxml" - } - # check config - for opt in [ - "notification_email_smtp_host", - "notification_email_smtp_port", - "notification_email_envelope_from", - "notification_email_from", - "notification_email_subject", - "notification_email_template", - "notification_email_embedded_imgs"] + list(defaults.keys()): - if opt in cfg: - continue - if opt in global_cfg: - cfg[opt] = global_cfg[opt] - elif opt in defaults: - cfg[opt] = defaults[opt] - else: - raise RuntimeError( - f"mandatory option '{opt}' not present in config " - f"section '{self.name}' or 'global'") - - self.smtp_host = cfg["notification_email_smtp_host"] - self.smtp_port = cfg["notification_email_smtp_port"] - self.mailfrom = cfg["notification_email_envelope_from"] - self.from_header = cfg["notification_email_from"] - self.subject = cfg["notification_email_subject"] - - testvars = defaultdict(str, test="TEST") - - # test-parse from header + def __init__(self, smtp_host, smtp_port, envelope_from, from_header, + subject, template, embed_imgs=[], repl_img=None, + strip_imgs=False, parser_lib="lxml", pretend=False): + super().__init__(pretend) + self.smtp_host = smtp_host + self.smtp_port = smtp_port + self.mailfrom = envelope_from + self.from_header = from_header + self.subject = subject try: - self.from_header.format_map(testvars) - except ValueError as e: - raise RuntimeError( - f"error parsing notification_email_from: {e}") - - # test-parse subject - try: - self.subject.format_map(testvars) - except ValueError as e: - raise RuntimeError( - f"error parsing notification_email_subject: {e}") - - # read and parse email notification template - try: - self.template = open( - cfg["notification_email_template"], "r").read() - self.template.format_map(testvars) - except IOError as e: - raise RuntimeError(f"error reading template: {e}") - except ValueError as e: - raise RuntimeError(f"error parsing template: {e}") - - strip_images = cfg["notification_email_strip_images"].strip().upper() - if strip_images in ["TRUE", "ON", "YES"]: - self.strip_images = True - elif strip_images in ["FALSE", "OFF", "NO"]: - self.strip_images = False - else: - raise RuntimeError( - "error parsing notification_email_strip_images: unknown value") - - self.parser_lib = cfg["notification_email_parser_lib"].strip() - if self.parser_lib not in ["lxml", "html.parser"]: - raise RuntimeError( - "error parsing notification_email_parser_lib: unknown value") - - # read email replacement image if specified - replacement_img = cfg["notification_email_replacement_img"].strip() - if not self.strip_images and replacement_img: - try: - self.replacement_img = MIMEImage( - open(replacement_img, "rb").read()) - except IOError as e: - raise RuntimeError( - f"error reading replacement image: {e}") - else: - self.replacement_img.add_header( - "Content-ID", "") - else: - self.replacement_img = None - - # read images to embed if specified - embedded_img_paths = [ - p.strip() for p in cfg["notification_email_embedded_imgs"].split( - ",") if p] - self.embedded_imgs = [] - for img_path in embedded_img_paths: - # read image - try: + self.template = open(template, "r").read() + self.embed_imgs = [] + for img_path in embed_imgs: img = MIMEImage(open(img_path, "rb").read()) - except IOError as e: - raise RuntimeError(f"error reading image: {e}") - else: filename = basename(img_path) img.add_header("Content-ID", f"<{filename}>") - self.embedded_imgs.append(img) + self.embed_imgs.append(img) - def get_email_body_soup(self, qid, msg): + self.replacement_img = repl_img + self.strip_images = strip_imgs + + if not strip_imgs and repl_img: + self.replacement_img = MIMEImage( + open(repl_img, "rb").read()) + self.replacement_img.add_header( + "Content-ID", "") + + except IOError as e: + raise RuntimeError(e) + + self.parser_lib = parser_lib + + def get_email_body_soup(self, msg, logger): "Extract and decode email body and return it as BeautifulSoup object." # try to find the body part - self.logger.debug(f"{qid}: trying to find email body") + logger.debug("trying to find email body") try: body = msg.get_body(preferencelist=("html", "plain")) except Exception as e: - self.logger.error( - f"{qid}: an error occured in " - f"email.message.EmailMessage.get_body: {e}") + logger.error( + f"an error occured in email.message.EmailMessage.get_body: " + f"{e}") body = None if body: @@ -233,55 +161,51 @@ class EMailNotification(BaseNotification): try: content = content.decode(encoding=charset, errors="replace") except LookupError: - self.logger.info( - f"{qid}: unknown encoding '{charset}', " - f"falling back to UTF-8") + logger.info( + f"unknown encoding '{charset}', falling back to UTF-8") content = content.decode("utf-8", errors="replace") content_type = body.get_content_type() if content_type == "text/plain": # convert text/plain to text/html - self.logger.debug( - f"{qid}: content type is {content_type}, " + logger.debug( + f"content type is {content_type}, " f"converting to text/html") content = re.sub(r"^(.*)$", r"\1
", escape(content, quote=False), flags=re.MULTILINE) else: - self.logger.debug( - f"{qid}: content type is {content_type}") + logger.debug(f"content type is {content_type}") else: - self.logger.error( - f"{qid}: unable to find email body") + logger.error("unable to find email body") content = "ERROR: unable to find email body" # create BeautifulSoup object length = len(content) - self.logger.debug( - f"{qid}: trying to create BeatufilSoup object with " + logger.debug( + f"trying to create BeatufilSoup object with " f"parser lib {self.parser_lib}, " f"text length is {length} bytes") soup = BeautifulSoup(content, self.parser_lib) - self.logger.debug( - f"{qid}: sucessfully created BeautifulSoup object") + logger.debug("sucessfully created BeautifulSoup object") return soup - def sanitize(self, qid, soup): + def sanitize(self, soup, logger): "Sanitize mail html text." - self.logger.debug(f"{qid}: sanitizing email text") + logger.debug("sanitizing email text") # completly remove bad elements for element in soup(EMailNotification._bad_tags): - self.logger.debug( - f"{qid}: removing dangerous tag '{element.name}' " + logger.debug( + f"removing dangerous tag '{element.name}' " f"and its content") element.extract() # remove not whitelisted elements, but keep their content for element in soup.find_all(True): if element.name not in EMailNotification._good_tags: - self.logger.debug( - f"{qid}: removing tag '{element.name}', keep its content") + logger.debug( + f"removing tag '{element.name}', keep its content") element.replaceWithChildren() # remove not whitelisted attributes @@ -289,126 +213,130 @@ class EMailNotification(BaseNotification): for attribute in list(element.attrs.keys()): if attribute not in EMailNotification._good_attributes: if element.name == "a" and attribute == "href": - self.logger.debug( - f"{qid}: setting attribute href to '#' " + logger.debug( + f"setting attribute href to '#' " f"on tag '{element.name}'") element["href"] = "#" else: - self.logger.debug( - f"{qid}: removing attribute '{attribute}' " + logger.debug( + f"removing attribute '{attribute}' " f"from tag '{element.name}'") del(element.attrs[attribute]) return soup - def notify(self, qid, storage_id, mailfrom, recipients, headers, fp, - subgroups=None, named_subgroups=None, synchronous=False): + def notify(self, msg, qid, mailfrom, recipients, logger, + template_vars=defaultdict(str), synchronous=False): "Notify recipients via email." - super( - EMailNotification, - self).notify( - qid, - storage_id, - mailfrom, - recipients, - headers, - fp, - subgroups, - named_subgroups, - synchronous) - # extract body from email - soup = self.get_email_body_soup( - qid, email.message_from_binary_file(fp, policy=default_policy)) + soup = self.get_email_body_soup(msg, logger) # replace picture sources image_replaced = False if self.strip_images: - self.logger.debug( - f"{qid}: looking for images to strip") + logger.debug("looking for images to strip") for element in soup("img"): if "src" in element.attrs.keys(): - self.logger.debug( - f"{qid}: strip image: {element['src']}") + logger.debug(f"strip image: {element['src']}") element.extract() elif self.replacement_img: - self.logger.debug( - f"{qid}: looking for images to replace") + logger.debug("looking for images to replace") for element in soup("img"): if "src" in element.attrs.keys(): - self.logger.debug( - f"{qid}: replacing image: {element['src']}") + logger.debug(f"replacing image: {element['src']}") element["src"] = "cid:removed_for_security_reasons" image_replaced = True # sanitizing email text of original email - sanitized_text = self.sanitize(qid, soup) + sanitized_text = self.sanitize(soup, logger) del soup # sending email notifications for recipient in recipients: - self.logger.debug( - f"{qid}: generating notification email for '{recipient}'") - self.logger.debug(f"{qid}: parsing email template") + logger.debug( + f"generating notification email for '{recipient}'") + logger.debug("parsing email template") # generate dict containing all template variables - variables = defaultdict( - str, - EMAIL_HTML_TEXT=sanitized_text, - EMAIL_FROM=escape(headers["from"], quote=False), - EMAIL_ENVELOPE_FROM=escape(mailfrom, quote=False), - EMAIL_ENVELOPE_FROM_URL=escape(quote(mailfrom), - quote=False), - EMAIL_TO=escape(headers["to"], quote=False), - EMAIL_ENVELOPE_TO=escape(recipient, quote=False), - EMAIL_ENVELOPE_TO_URL=escape(quote(recipient)), - EMAIL_SUBJECT=escape(headers["subject"], quote=False), - EMAIL_QUARANTINE_ID=storage_id) - - if subgroups: - number = 0 - for subgroup in subgroups: - variables[f"SUBGROUP_{number}"] = escape(subgroup, - quote=False) - if named_subgroups: - for key, value in named_subgroups.items(): - named_subgroups[key] = escape(value, quote=False) - variables.update(named_subgroups) + variables = defaultdict(str, template_vars) + variables.update({ + "HTML_TEXT": sanitized_text, + "FROM": escape(msg["from"], quote=False), + "ENVELOPE_FROM": escape(mailfrom, quote=False), + "ENVELOPE_FROM_URL": escape(quote(mailfrom), + quote=False), + "TO": escape(msg["to"], quote=False), + "ENVELOPE_TO": escape(recipient, quote=False), + "ENVELOPE_TO_URL": escape(quote(recipient)), + "SUBJECT": escape(msg["subject"], quote=False)}) # parse template htmltext = self.template.format_map(variables) - msg = MIMEMultipart('related') - msg["From"] = self.from_header.format_map( - defaultdict(str, EMAIL_FROM=headers["from"])) - msg["To"] = headers["to"] - msg["Subject"] = self.subject.format_map( - defaultdict(str, EMAIL_SUBJECT=headers["subject"])) - msg["Date"] = email.utils.formatdate() - msg.attach(MIMEText(htmltext, "html", 'UTF-8')) + newmsg = MIMEMultipart('related') + newmsg["From"] = self.from_header.format_map( + defaultdict(str, FROM=msg["from"])) + newmsg["To"] = msg["to"] + newmsg["Subject"] = self.subject.format_map( + defaultdict(str, SUBJECT=msg["subject"])) + newmsg["Date"] = email.utils.formatdate() + newmsg.attach(MIMEText(htmltext, "html", 'UTF-8')) if image_replaced: - self.logger.debug( - f"{qid}: attaching notification_replacement_img") - msg.attach(self.replacement_img) + logger.debug("attaching notification_replacement_img") + newmsg.attach(self.replacement_img) - for img in self.embedded_imgs: - self.logger.debug(f"{qid}: attaching imgage") - msg.attach(img) + for img in self.embed_imgs: + logger.debug("attaching imgage") + newmsg.attach(img) - self.logger.debug( - f"{qid}: sending notification email to: {recipient}") + logger.debug(f"sending notification email to: {recipient}") if synchronous: try: mailer.smtp_send(self.smtp_host, self.smtp_port, - self.mailfrom, recipient, msg.as_string()) + self.mailfrom, recipient, + newmsg.as_string()) except Exception as e: raise RuntimeError( f"error while sending email to '{recipient}': {e}") else: mailer.sendmail(self.smtp_host, self.smtp_port, qid, - self.mailfrom, recipient, msg.as_string(), + self.mailfrom, recipient, newmsg.as_string(), "notification email") + def execute(self, milter, logger): + super().execute(milter, logger) -# list of notification types and their related notification classes -TYPES = {"email": EMailNotification} + self.notify(msg=milter.msg, qid=milter.qid, + mailfrom=milter.msginfo["mailfrom"], + recipients=milter.msginfo["rcpts"], + template_vars=milter.msginfo["vars"], + logger=logger) + + +class Notify: + NOTIFICATION_TYPES = { + "email": EMailNotification} + + def __init__(self, cfg, local_addrs, debug): + self.cfg = cfg + self.logger = logging.getLogger(cfg["name"]) + self.logger.setLevel(cfg.get_loglevel(debug)) + + nodification_type = cfg["args"]["type"] + del cfg["args"]["type"] + cfg["args"]["pretend"] = cfg["pretend"] + self._notification = self.NOTIFICATION_TYPES[nodification_type]( + **cfg["args"]) + self._headersonly = self._notification._headersonly + + def __str__(self): + cfg = [] + for key, value in self.cfg["args"].items(): + cfg.append(f"{key}={value}") + class_name = type(self._notification).__name__ + return f"{class_name}(" + ", ".join(cfg) + ")" + + def execute(self, milter): + logger = CustomLogger( + self.logger, {"name": self.cfg["name"], "qid": milter.qid}) + self._notification.execute(milter, logger) diff --git a/pyquarantine/rule.py b/pyquarantine/rule.py new file mode 100644 index 0000000..e0e003b --- /dev/null +++ b/pyquarantine/rule.py @@ -0,0 +1,66 @@ +# pyquarantine is free software: you can redistribute it and/or modify +# it under the terms of the GNU General Public License as published by +# the Free Software Foundation, either version 3 of the License, or +# (at your option) any later version. +# +# pyquarantine is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the +# GNU General Public License for more details. +# +# You should have received a copy of the GNU General Public License +# along with pyquarantine. If not, see . +# + +__all__ = ["Rule"] + +from pyquarantine.action import Action +from pyquarantine.conditions import Conditions + + +class Rule: + """ + Rule to implement multiple actions on emails. + """ + def __init__(self, cfg, local_addrs, debug): + self.cfg = cfg + + self.conditions = cfg["conditions"] if "conditions" in cfg else None + if self.conditions is not None: + self.conditions["name"] = f"{cfg['name']}: condition" + self.conditions["loglevel"] = cfg["loglevel"] + self.conditions = Conditions(self.conditions, local_addrs, debug) + + self.actions = [] + for idx, action_cfg in enumerate(cfg["actions"]): + if "name" in action_cfg: + action_cfg["name"] = f"{cfg['name']}: {action_cfg['name']}" + else: + action_cfg["name"] = f"action#{idx}" + if "loglevel" not in action_cfg: + action_cfg["loglevel"] = cfg["loglevel"] + if "pretend" not in action_cfg: + action_cfg["pretend"] = cfg["pretend"] + self.actions.append(Action(action_cfg, local_addrs, debug)) + + def __str__(self): + cfg = [] + for key in ["name", "loglevel", "pretend"]: + value = self.cfg[key] + cfg.append(f"{key}={value}") + if self.conditions is not None: + cfg.append(f"conditions={self.conditions}") + actions = [] + for action in self.actions: + actions.append(str(action)) + cfg.append("actions=[" + ", ".join(actions) + "]") + return "Rule(" + ", ".join(cfg) + ")" + + def execute(self, milter): + """Execute all actions of this rule.""" + if self.conditions is None or \ + self.conditions.match(milter): + for action in self.actions: + milter_action = action.execute(milter) + if milter_action is not None: + return milter_action diff --git a/pyquarantine/run.py b/pyquarantine/run.py index 45e8edf..4cbce22 100644 --- a/pyquarantine/run.py +++ b/pyquarantine/run.py @@ -1,127 +1,152 @@ -#!/usr/bin/env python -# -# PyQuarantine-Milter is free software: you can redistribute it and/or modify +# pyquarantine is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # -# PyQuarantine-Milter is distributed in the hope that it will be useful, +# pyquarantine is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License -# along with PyQuarantineMilter. If not, see . +# along with pyquarantine. If not, see . # +__all__ = ["main"] + import Milter import argparse import logging import logging.handlers import sys -import pyquarantine - +from pyquarantine import mailer +from pyquarantine import ModifyMilter from pyquarantine import __version__ as version +from pyquarantine.config import get_milter_config def main(): - "Run PyQuarantine-Milter." - # parse command line + python_version = ".".join([str(v) for v in sys.version_info[0:3]]) + python_version = f"{python_version}-{sys.version_info[3]}" + + "Run pyquarantine." parser = argparse.ArgumentParser( - description="PyQuarantine milter daemon", + description="pyquarantine-milter daemon", formatter_class=lambda prog: argparse.HelpFormatter( prog, max_help_position=45, width=140)) + parser.add_argument( - "-c", "--config", - help="List of config files to read.", - nargs="+", - default=pyquarantine.QuarantineMilter.get_cfg_files()) + "-c", "--config", help="Config file to read.", + default="/etc/pyquarantine/pyquarantine.conf") + parser.add_argument( - "-s", "--socket", + "-s", + "--socket", help="Socket used to communicate with the MTA.", - default="inet:8899@127.0.0.1") + default="") + parser.add_argument( - "-d", "--debug", + "-d", + "--debug", help="Log debugging messages.", action="store_true") + parser.add_argument( - "-t", "--test", + "-t", + "--test", help="Check configuration.", action="store_true") + parser.add_argument( "-v", "--version", help="Print version.", action="version", - version=f"%(prog)s ({version})") + version=f"%(prog)s {version} (python {python_version})") + args = parser.parse_args() - # setup logging - loglevel = logging.INFO - logname = "pyquarantine-milter" - syslog_name = logname - if args.debug: - loglevel = logging.DEBUG - logname = f"{logname}[%(name)s]" - syslog_name = f"{syslog_name}: [%(name)s] %(levelname)s" - - # set config files for milter class - pyquarantine.QuarantineMilter.set_cfg_files(args.config) root_logger = logging.getLogger() - root_logger.setLevel(loglevel) + root_logger.setLevel(logging.DEBUG) # setup console log stdouthandler = logging.StreamHandler(sys.stdout) - stdouthandler.setLevel(logging.DEBUG) - formatter = logging.Formatter("%(message)s") + formatter = logging.Formatter("%(levelname)s: %(message)s") stdouthandler.setFormatter(formatter) root_logger.addHandler(stdouthandler) + logger = logging.getLogger(__name__) - if args.test: - try: - pyquarantine.setup_milter(test=args.test) - print("Configuration ok") - except RuntimeError as e: - logger.error(e) - sys.exit(255) + + if not args.debug: + logger.setLevel(logging.INFO) + + try: + logger.debug("read milter configuration") + cfg = get_milter_config(args.config) + logger.setLevel(cfg.get_loglevel(args.debug)) + + if args.socket: + socket = args.socket + elif "socket" in cfg: + socket = cfg["socket"] else: - sys.exit(0) - formatter = logging.Formatter( - f"%(asctime)s {logname}: [%(levelname)s] %(message)s", - datefmt="%Y-%m-%d %H:%M:%S") + raise RuntimeError( + "listening socket is neither specified on the command line " + "nor in the configuration file") + + if not cfg["rules"]: + raise RuntimeError("no rules configured") + + for rule in cfg["rules"]: + if not rule["actions"]: + raise RuntimeError( + f"{rule['name']}: no actions configured") + + except (RuntimeError, AssertionError) as e: + logger.error(f"config error: {e}") + sys.exit(255) + + try: + ModifyMilter.set_config(cfg, args.debug) + except RuntimeError as e: + logger.error(e) + sys.exit(254) + + if args.test: + print("Configuration OK") + sys.exit(0) + + # setup console log for runtime + formatter = logging.Formatter("%(asctime)s - %(levelname)s: %(message)s") stdouthandler.setFormatter(formatter) + stdouthandler.setLevel(logging.DEBUG) # setup syslog sysloghandler = logging.handlers.SysLogHandler( address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_MAIL) - sysloghandler.setLevel(loglevel) - formatter = logging.Formatter(f"{syslog_name}: %(message)s") - sysloghandler.setFormatter(formatter) + sysloghandler.setFormatter( + logging.Formatter("pyquarantine: %(message)s")) root_logger.addHandler(sysloghandler) - logger.info("PyQuarantine-Milter starting") - try: - # generate milter config - pyquarantine.setup_milter() - except RuntimeError as e: - logger.error(e) - sys.exit(255) + logger.info("pyquarantine-milter starting") - # register to have the Milter factory create instances of your class: - Milter.factory = pyquarantine.QuarantineMilter + # register milter factory class + Milter.factory = ModifyMilter Milter.set_exception_policy(Milter.TEMPFAIL) - # run milter + if args.debug: + Milter.setdbg(1) + rc = 0 try: - Milter.runmilter("pyquarantine-milter", socketname=args.socket, - timeout=300) + Milter.runmilter("pyquarantine", socketname=socket, timeout=600) except Milter.milter.error as e: logger.error(e) rc = 255 - pyquarantine.mailer.queue.put(None) - logger.info("PyQuarantine-Milter terminated") + mailer.queue.put(None) + logger.info("pyquarantine-milter stopped") + sys.exit(rc) diff --git a/pyquarantine/storage.py b/pyquarantine/storage.py index 8114473..597e2dd 100644 --- a/pyquarantine/storage.py +++ b/pyquarantine/storage.py @@ -1,17 +1,23 @@ -# PyQuarantine-Milter is free software: you can redistribute it and/or modify +# pyquarantine is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # -# PyQuarantine-Milter is distributed in the hope that it will be useful, +# pyquarantine is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License -# along with PyQuarantineMilter. If not, see . +# along with pyquarantine. If not, see . # +__all__ = [ + "BaseMailStorage", + "FileMailStorage", + "Store", + "Quarantine"] + import json import logging import os @@ -19,23 +25,31 @@ import os from calendar import timegm from datetime import datetime from glob import glob -from shutil import copyfileobj from time import gmtime +from pyquarantine.base import CustomLogger +from pyquarantine.conditions import Conditions +from pyquarantine.config import ActionConfig +from pyquarantine.notify import Notify -class BaseMailStorage(object): + +class BaseMailStorage: "Mail storage base class" - storage_type = "base" + _headersonly = True - def __init__(self, name, global_cfg, cfg, test=False): - self.name = name - self.logger = logging.getLogger(__name__) + def __init__(self, original=False, metadata=False, metavar=None, + pretend=False): + self.original = original + self.metadata = metadata + self.metavar = metavar + self.pretend = False - def add(self, qid, mailfrom, recipients, headers, - fp, subgroups=None, named_subgroups=None): + def add(self, data, qid, mailfrom="", recipients=[]): "Add email to storage." - fp.seek(0) - return "" + return ("", "") + + def execute(self, milter, logger): + return def find(self, mailfrom=None, recipients=None, older_than=None): "Find emails in storage." @@ -50,116 +64,158 @@ class BaseMailStorage(object): return def get_mail(self, storage_id): - "Return a file pointer to the email and metadata." + "Return email and metadata." return class FileMailStorage(BaseMailStorage): "Storage class to store mails on filesystem." - storage_type = "file" + _headersonly = False - def __init__(self, name, global_cfg, cfg, test=False): - super(FileMailStorage, self).__init__(name, global_cfg, cfg, test) - - defaults = {} - # check config - - for opt in ["storage_directory"] + list(defaults.keys()): - if opt in cfg: - continue - if opt in global_cfg: - cfg[opt] = global_cfg[opt] - elif opt in defaults: - cfg[opt] = defaults[opt] - else: - raise RuntimeError( - f"mandatory option '{opt}' not present in config " - f"section '{self.name}' or 'global'") - self.directory = cfg["storage_directory"] - - # check if quarantine directory exists and is writable - if not os.path.isdir(self.directory) or not os.access( - self.directory, os.W_OK): + def __init__(self, directory, original=False, metadata=False, metavar=None, + mode=None, pretend=False): + super().__init__(original, metadata, metavar, pretend) + # check if directory exists and is writable + if not os.path.isdir(directory) or \ + not os.access(directory, os.W_OK): raise RuntimeError( - f"file quarantine directory '{self.directory}' does " - f"not exist or is not writable") + f"directory '{directory}' does not exist or is " + f"not writable") + self.directory = directory + try: + self.mode = int(mode, 8) if mode is not None else None + if self.mode is not None and self.mode > 511: + raise ValueError + except ValueError: + raise RuntimeError(f"invalid mode '{mode}'") + self._metadata_suffix = ".metadata" - def _save_datafile(self, storage_id, fp): + def __str__(self): + cfg = [] + cfg.append(f"metadata={self.metadata}") + cfg.append(f"metavar={self.metavar}") + cfg.append(f"pretend={self.pretend}") + cfg.append(f"directory={self.directory}") + cfg.append(f"original={self.original}") + return "FileMailStorage(" + ", ".join(cfg) + ")" + + def get_storageid(self, qid): + timestamp = datetime.now().strftime("%Y%m%d%H%M%S") + return f"{timestamp}_{qid}" + + def _get_file_paths(self, storage_id): datafile = os.path.join(self.directory, storage_id) + metafile = f"{datafile}{self._metadata_suffix}" + return metafile, datafile + + def _save_datafile(self, datafile, data): try: - with open(datafile, "wb") as f: - copyfileobj(fp, f) + if self.mode is None: + with open(datafile, "wb") as f: + f.write(data) + else: + umask = os.umask(0) + with open( + os.open(datafile, os.O_CREAT | os.O_WRONLY, self.mode), + "wb") as f: + f.write(data) + os.umask(umask) + except IOError as e: raise RuntimeError(f"unable save data file: {e}") - def _save_metafile(self, storage_id, metadata): - metafile = os.path.join( - self.directory, f"{storage_id}{self._metadata_suffix}") + def _save_metafile(self, metafile, metadata): try: - with open(metafile, "w") as f: - json.dump(metadata, f, indent=2) + if self.mode is None: + with open(metafile, "w") as f: + json.dump(metadata, f, indent=2) + else: + umask = os.umask(0) + with open( + os.open(metafile, os.O_CREAT | os.O_WRONLY, self.mode), + "w") as f: + json.dump(metadata, f, indent=2) + os.umask(umask) + except IOError as e: raise RuntimeError(f"unable to save metadata file: {e}") def _remove(self, storage_id): - datafile = os.path.join(self.directory, storage_id) - metafile = f"{datafile}{self._metadata_suffix}" + metafile, datafile = self._get_file_paths(storage_id) try: - os.remove(metafile) - except IOError as e: - raise RuntimeError(f"unable to remove metadata file: {e}") + if self.metadata: + os.remove(metafile) - try: os.remove(datafile) except IOError as e: - raise RuntimeError(f"unable to remove data file: {e}") + raise RuntimeError(f"unable to remove file: {e}") - def add(self, qid, mailfrom, recipients, headers, - fp, subgroups=None, named_subgroups=None): + def add(self, data, qid, mailfrom="", recipients=[], subject=""): "Add email to file storage and return storage id." - super( - FileMailStorage, - self).add( - qid, - mailfrom, - recipients, - headers, - fp, - subgroups, - named_subgroups) - timestamp = datetime.now().strftime("%Y%m%d%H%M%S") - storage_id = f"{timestamp}_{qid}" + super().add(data, qid, mailfrom, recipients) + + storage_id = self.get_storageid(qid) + metafile, datafile = self._get_file_paths(storage_id) # save mail - self._save_datafile(storage_id, fp) + self._save_datafile(datafile, data) + + if not self.metadata: + return storage_id, None, datafile # save metadata metadata = { "mailfrom": mailfrom, "recipients": recipients, - "headers": headers, - "date": timegm(gmtime()), - "queue_id": qid, - "subgroups": subgroups, - "named_subgroups": named_subgroups - } + "subject": subject, + "timestamp": timegm(gmtime()), + "queue_id": qid} + try: - self._save_metafile(storage_id, metadata) + self._save_metafile(metafile, metadata) except RuntimeError as e: - datafile = os.path.join(self.directory, storage_id) os.remove(datafile) raise e - return storage_id + return storage_id, metafile, datafile + + def execute(self, milter, logger): + if self.original: + milter.fp.seek(0) + data = milter.fp.read + mailfrom = milter.mailfrom + recipients = list(milter.rcpts) + subject = "" + else: + data = milter.msg.as_bytes + mailfrom = milter.msginfo["mailfrom"] + recipients = list(milter.msginfo["rcpts"]) + subject = milter.msg["subject"] or "" + + if not self.pretend: + storage_id, metafile, datafile = self.add( + data(), milter.qid, mailfrom, recipients, subject) + logger.info(f"stored message in file {datafile}") + else: + storage_id = self.get_storageid(milter.qid) + metafile, datafile = self._get_file_paths(storage_id) + + if self.metavar: + milter.msginfo["vars"][f"{self.metavar}_ID"] = storage_id + milter.msginfo["vars"][f"{self.metavar}_DATAFILE"] = datafile + if self.metadata: + milter.msginfo["vars"][f"{self.metavar}_METAFILE"] = metafile def get_metadata(self, storage_id): "Return metadata of email in storage." - super(FileMailStorage, self).get_metadata(storage_id) + super().get_metadata(storage_id) - metafile = os.path.join( - self.directory, f"{storage_id}{self._metadata_suffix}") + if not self.metadata: + return None + + metafile, _ = self._get_file_paths(storage_id) if not os.path.isfile(metafile): raise RuntimeError( f"invalid storage id '{storage_id}'") @@ -177,12 +233,15 @@ class FileMailStorage(BaseMailStorage): def find(self, mailfrom=None, recipients=None, older_than=None): "Find emails in storage." - super(FileMailStorage, self).find(mailfrom, recipients, older_than) + super().find(mailfrom, recipients, older_than) if isinstance(mailfrom, str): mailfrom = [mailfrom] if isinstance(recipients, str): recipients = [recipients] + if not self.metadata: + return {} + emails = {} metafiles = glob(os.path.join( self.directory, f"*{self._metadata_suffix}")) @@ -215,38 +274,158 @@ class FileMailStorage(BaseMailStorage): def delete(self, storage_id, recipients=None): "Delete email from storage." - super(FileMailStorage, self).delete(storage_id, recipients) + super().delete(storage_id, recipients) + + if not recipients or not self.metadata: + self._remove(storage_id) + return try: metadata = self.get_metadata(storage_id) except RuntimeError as e: raise RuntimeError(f"unable to delete email: {e}") - if not recipients: - self._remove(storage_id) - else: - if type(recipients) == str: - recipients = [recipients] - for recipient in recipients: - if recipient not in metadata["recipients"]: - raise RuntimeError(f"invalid recipient '{recipient}'") - metadata["recipients"].remove(recipient) - if not metadata["recipients"]: - self._remove(storage_id) - else: - self._save_metafile(storage_id, metadata) + metafile, _ = self._get_file_paths(storage_id) + + if type(recipients) == str: + recipients = [recipients] + + for recipient in recipients: + if recipient not in metadata["recipients"]: + raise RuntimeError(f"invalid recipient '{recipient}'") + metadata["recipients"].remove(recipient) + if not metadata["recipients"]: + self._remove(storage_id) + else: + self._save_metafile(metafile, metadata) def get_mail(self, storage_id): - super(FileMailStorage, self).get_mail(storage_id) + super().get_mail(storage_id) metadata = self.get_metadata(storage_id) - datafile = os.path.join(self.directory, storage_id) + _, datafile = self._get_file_paths(storage_id) try: - fp = open(datafile, "rb") + data = open(datafile, "rb").read() except IOError as e: raise RuntimeError(f"unable to open email data file: {e}") - return (fp, metadata) + return (metadata, data) -# list of storage types and their related storage classes -TYPES = {"file": FileMailStorage} +class Store: + STORAGE_TYPES = { + "file": FileMailStorage} + + def __init__(self, cfg, local_addrs, debug): + self.cfg = cfg + self.logger = logging.getLogger(cfg["name"]) + self.logger.setLevel(cfg.get_loglevel(debug)) + + storage_type = cfg["args"]["type"] + del cfg["args"]["type"] + cfg["args"]["pretend"] = cfg["pretend"] + self._storage = self.STORAGE_TYPES[storage_type]( + **cfg["args"]) + self._headersonly = self._storage._headersonly + + def __str__(self): + cfg = [] + for key, value in self.cfg["args"].items(): + cfg.append(f"{key}={value}") + class_name = type(self._storage).__name__ + return f"{class_name}(" + ", ".join(cfg) + ")" + + def execute(self, milter): + logger = CustomLogger( + self.logger, {"name": self.cfg["name"], "qid": milter.qid}) + self._storage.execute(milter, logger) + + +class Quarantine: + "Quarantine class." + _headersonly = False + + def __init__(self, cfg, local_addrs, debug): + self.cfg = cfg + self.logger = logging.getLogger(cfg["name"]) + self.logger.setLevel(cfg.get_loglevel(debug)) + + store_cfg = ActionConfig({ + "name": cfg["name"], + "loglevel": cfg["loglevel"], + "pretend": cfg["pretend"], + "type": "store", + "args": cfg["args"]["store"].get_config()}) + store_cfg["args"]["metadata"] = True + self.store = Store(store_cfg, local_addrs, debug) + + self.notify = None + if "notify" in cfg["args"]: + notify_cfg = ActionConfig({ + "name": cfg["name"], + "loglevel": cfg["loglevel"], + "pretend": cfg["pretend"], + "type": "notify", + "args": cfg["args"]["notify"].get_config()}) + self.notify = Notify(notify_cfg, local_addrs, debug) + + self.whitelist = None + if "whitelist" in cfg["args"]: + whitelist_cfg = cfg["args"]["whitelist"] + whitelist_cfg["name"] = cfg["name"] + whitelist_cfg["loglevel"] = cfg["loglevel"] + self.whitelist = Conditions( + whitelist_cfg, + local_addrs=[], + debug=debug) + + self.milter_action = None + if "milter_action" in cfg["args"]: + self.milter_action = cfg["args"]["milter_action"] + self.reject_reason = None + if "reject_reason" in cfg["args"]: + self.reject_reason = cfg["args"]["reject_reason"] + + def __str__(self): + cfg = [] + cfg.append(f"store={str(self.store)}") + if self.notify is not None: + cfg.append(f"notify={str(self.notify)}") + if self.whitelist is not None: + cfg.append(f"whitelist={str(self.whitelist)}") + for key in ["milter_action", "reject_reason"]: + if key not in self.cfg["args"]: + continue + value = self.cfg["args"][key] + cfg.append(f"{key}={value}") + class_name = type(self).__name__ + return f"{class_name}(" + ", ".join(cfg) + ")" + + def execute(self, milter): + logger = CustomLogger( + self.logger, {"name": self.cfg["name"], "qid": milter.qid}) + wl_rcpts = [] + if self.whitelist: + wl_rcpts = self.whitelist.get_wl_rcpts( + milter.msginfo["mailfrom"], milter.msginfo["rcpts"], logger) + logger.info(f"whitelisted recipients: {wl_rcpts}") + + rcpts = [ + rcpt for rcpt in milter.msginfo["rcpts"] if rcpt not in wl_rcpts] + + if not rcpts: + # all recipients whitelisted + return + + logger.info(f"add to quarantine for recipients: {rcpts}") + milter.msginfo["rcpts"] = rcpts + + self.store.execute(milter) + + if self.notify is not None: + self.notify.execute(milter) + + milter.msginfo["rcpts"].extend(wl_rcpts) + milter.delrcpt(rcpts) + + if self.milter_action is not None and not milter.msginfo["rcpts"]: + return (self.milter_action, self.reject_reason) diff --git a/pyquarantine/whitelist.py b/pyquarantine/whitelist.py index 014b67f..34a923a 100644 --- a/pyquarantine/whitelist.py +++ b/pyquarantine/whitelist.py @@ -1,17 +1,21 @@ -# PyQuarantine-Milter is free software: you can redistribute it and/or modify +# pyquarantine is free software: you can redistribute it and/or modify # it under the terms of the GNU General Public License as published by # the Free Software Foundation, either version 3 of the License, or # (at your option) any later version. # -# PyQuarantine-Milter is distributed in the hope that it will be useful, +# pyquarantine is distributed in the hope that it will be useful, # but WITHOUT ANY WARRANTY; without even the implied warranty of # MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the # GNU General Public License for more details. # # You should have received a copy of the GNU General Public License -# along with PyQuarantineMilter. If not, see . +# along with pyquarantine. If not, see . # +__all__ = [ + "DatabaseWhitelist", + "WhitelistBase"] + import logging import peewee import re @@ -20,17 +24,23 @@ from datetime import datetime from playhouse.db_url import connect -class WhitelistBase(object): +class WhitelistBase: "Whitelist base class" + def __init__(self, cfg, debug): + self.cfg = cfg + self.logger = logging.getLogger(cfg["name"]) + self.logger.setLevel(cfg.get_loglevel(debug)) - whitelist_type = "base" + peewee_logger = logging.getLogger("peewee") + peewee_logger.setLevel(cfg.get_loglevel(debug)) - def __init__(self, name, global_cfg, cfg, test=False): - self.name = name - self.test = test - self.logger = logging.getLogger(__name__) self.valid_entry_regex = re.compile( r"^[a-zA-Z0-9_.=+-]*?(@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+)?$") + self.batv_regex = re.compile( + r"^prvs=[0-9]{4}[0-9A-Fa-f]{6}=(?P.+?)@") + + def remove_batv(self, addr): + return self.batv_regex.sub(r"\g", addr, count=1) def check(self, mailfrom, recipient): "Check if mailfrom/recipient combination is whitelisted." @@ -63,7 +73,7 @@ class WhitelistModel(peewee.Model): permanent = peewee.BooleanField(default=False) -class Meta(object): +class Meta: indexes = ( # trailing comma is mandatory if only one index should be created (('mailfrom', 'recipient'), True), @@ -76,32 +86,11 @@ class DatabaseWhitelist(WhitelistBase): _db_connections = {} _db_tables = {} - def __init__(self, name, global_cfg, cfg, test=False): - super( - DatabaseWhitelist, - self).__init__( - global_cfg, - cfg, - test) + def __init__(self, cfg, debug): + super().__init__(cfg, debug) - defaults = {} - - # check config - for opt in ["whitelist_db_connection", - "whitelist_db_table"] + list(defaults.keys()): - if opt in cfg: - continue - if opt in global_cfg: - cfg[opt] = global_cfg[opt] - elif opt in defaults: - cfg[opt] = defaults[opt] - else: - raise RuntimeError( - f"mandatory option '{opt}' not present in config " - f"section '{self.name}' or 'global'") - - tablename = cfg["whitelist_db_table"] - connection_string = cfg["whitelist_db_connection"] + tablename = cfg["table"] + connection_string = cfg["connection"] if connection_string in DatabaseWhitelist._db_connections.keys(): db = DatabaseWhitelist._db_connections[connection_string] @@ -125,21 +114,28 @@ class DatabaseWhitelist(WhitelistBase): self.meta = Meta self.meta.database = db self.meta.table_name = tablename - self.model = type(f"WhitelistModel_{self.name}", (WhitelistModel,), { - "Meta": self.meta - }) + self.model = type( + f"WhitelistModel_{self.cfg['name']}", + (WhitelistModel,), + {"Meta": self.meta}) if connection_string not in DatabaseWhitelist._db_tables.keys(): DatabaseWhitelist._db_tables[connection_string] = [] if tablename not in DatabaseWhitelist._db_tables[connection_string]: DatabaseWhitelist._db_tables[connection_string].append(tablename) - if not self.test: - try: - db.create_tables([self.model]) - except Exception as e: - raise RuntimeError( - f"unable to initialize table '{tablename}': {e}") + try: + db.create_tables([self.model]) + except Exception as e: + raise RuntimeError( + f"unable to initialize table '{tablename}': {e}") + + def __str__(self): + cfg = [] + for arg in ("connection", "table"): + if arg in self.cfg: + cfg.append(f"{arg}={self.cfg[arg]}") + return "DatabaseWhitelist(" + ", ".join(cfg) + ")" def _entry_to_dict(self, entry): result = {} @@ -163,12 +159,14 @@ class DatabaseWhitelist(WhitelistBase): value += 1 return value - def check(self, mailfrom, recipient): + def check(self, mailfrom, recipient, logger): # check if mailfrom/recipient combination is whitelisted - super(DatabaseWhitelist, self).check(mailfrom, recipient) + super().check(mailfrom, recipient) + mailfrom = self.remove_batv(mailfrom) + recipient = self.remove_batv(recipient) # generate list of possible mailfroms - self.logger.debug( + logger.debug( f"query database for whitelist entries from <{mailfrom}> " f"to <{recipient}>") mailfroms = [""] @@ -212,7 +210,7 @@ class DatabaseWhitelist(WhitelistBase): def find(self, mailfrom=None, recipients=None, older_than=None): "Find whitelist entries." - super(DatabaseWhitelist, self).find(mailfrom, recipients, older_than) + super().find(mailfrom, recipients, older_than) if isinstance(mailfrom, str): mailfrom = [mailfrom] @@ -243,14 +241,15 @@ class DatabaseWhitelist(WhitelistBase): def add(self, mailfrom, recipient, comment, permanent): "Add entry to whitelist." - super( - DatabaseWhitelist, - self).add( + super().add( mailfrom, recipient, comment, permanent) + mailfrom = self.remove_batv(mailfrom) + recipient = self.remove_batv(recipient) + try: self.model.create( mailfrom=mailfrom, @@ -262,7 +261,7 @@ class DatabaseWhitelist(WhitelistBase): def delete(self, whitelist_id): "Delete entry from whitelist." - super(DatabaseWhitelist, self).delete(whitelist_id) + super().delete(whitelist_id) try: query = self.model.delete().where(self.model.id == whitelist_id) @@ -273,32 +272,3 @@ class DatabaseWhitelist(WhitelistBase): if deleted == 0: raise RuntimeError("invalid whitelist id") - - -class WhitelistCache(object): - def __init__(self): - self.cache = {} - - def load(self, whitelist, mailfrom, recipients): - for recipient in recipients: - self.check(whitelist, mailfrom, recipient) - - def check(self, whitelist, mailfrom, recipient): - if whitelist not in self.cache.keys(): - self.cache[whitelist] = {} - if recipient not in self.cache[whitelist].keys(): - self.cache[whitelist][recipient] = None - if self.cache[whitelist][recipient] is None: - self.cache[whitelist][recipient] = whitelist.check( - mailfrom, recipient) - return self.cache[whitelist][recipient] - - def get_recipients(self, whitelist, mailfrom, recipients): - self.load(whitelist, mailfrom, recipients) - return list(filter( - lambda x: self.cache[whitelist][x], - self.cache[whitelist].keys())) - - -# list of whitelist types and their related whitelist classes -TYPES = {"db": DatabaseWhitelist} diff --git a/setup.py b/setup.py index dbd1eee..a27fe09 100644 --- a/setup.py +++ b/setup.py @@ -4,17 +4,16 @@ def read_file(fname): with open(fname, 'r') as f: return f.read() - setup(name = "pyquarantine", author = "Thomas Oettli", author_email = "spacefreak@noop.ch", description = "A pymilter based sendmail/postfix pre-queue filter.", license = "GPL 3", - keywords = "quarantine milter", - url = "https://github.com/spacefreak86/pyquarantine-milter", + keywords = "header quarantine milter", + url = "https://github.com/spacefreak86/pyquarantine", packages = ["pyquarantine"], long_description = read_file("README.md"), - long_description_content_type="text/markdown", + long_description_content_type = "text/markdown", classifiers = [ # 3 - Alpha # 4 - Beta @@ -26,12 +25,29 @@ setup(name = "pyquarantine", "Programming Language :: Python :: 3", "Topic :: Communications :: Email :: Filters" ], + include_package_data = True, entry_points = { "console_scripts": [ "pyquarantine-milter=pyquarantine.run:main", - "pyquarantine=pyquarantine.cli:main" + "pyquarantine=pyquarantine.run:main", ] }, - install_requires = ["pymilter", "netaddr", "beautifulsoup4[lxml]", "peewee"], - python_requires = ">=3.6" + data_files = [ + ( + "/etc/pyquarantine", + [ + "pyquarantine/docs/pyquarantine.conf.example" + ] + ), ( + "/etc/pyquarantine/templates", + [ + "pyquarantine/docs/templates/disclaimer_html.template", + "pyquarantine/docs/templates/disclaimer_text.template", + "pyquarantine/docs/templates/notification.template", + "pyquarantine/docs/templates/removed.png" + ] + ) + ], + install_requires = ["pymilter", "jsonschema", "netaddr", "beautifulsoup4[lxml]", "peewee"], + python_requires = ">=3.8" )