spacefreak/pyquarantine-milter
1
0
Fork 0
Code Issues Pull Requests Releases Activity

merge with project pymodmilter

Browse Source
This commit is contained in:
Thomas Oettli
2021-09-30 18:18:00 +02:00
parent 461a4bdb7c 0bd88f7cf4
commit ba3ac65b21
22 changed files with 2578 additions and 1449 deletions
Download Patch File Download Diff File Expand all files Collapse all files

265
README.md
View File

@@ -1,157 +1,140 @@
# pyquarantine-milter
A pymilter based sendmail/postfix pre-queue filter with the ability to quarantine e-mails, sending notifications
to recipients and respond with a milter-action (ACCEPT, DISCARD or REJECT).
# pyquarantine
A pymilter based sendmail/postfix pre-queue filter with the ability to add, remove and modify e-mail headers.
The project is currently in beta status, but it is already used in a productive enterprise environment that processes about a million e-mails per month.
It is useful in many cases because of its felxible configuration and the ability to handle any number of quarantines.
The MTA can check e-mail headers using regular expressions to determine if and which quarantine to use.
Each quarantine can be configured with a quarantine type, notification type, whitelist and an action to respond with.
The basic idea is to define rules with conditions and actions which are processed when all conditions are true.
Addionally, pyquarantine-milter provides a sanitized, harmless version of the text parts of e-mails, which can be embedded in e-mail notifications. This makes it easier for users to decide, if a match is a false-positive or not. If a matching quarantine provides a quarantine ID of the original e-mail, it is also available as a template variable. This is useful if you want to add links to a webservice to notification e-mails, to give your users the ability to release e-mails or whitelist the from-address for example. The webservice then releases the e-mail from the quarantine.
## Dependencies
pyquarantine is depending on these python packages, but they are installed automatically if you are working with pip.
* [jsonschema](https://github.com/Julian/jsonschema)
* [pymilter](https://github.com/sdgathman/pymilter)
* [netaddr](https://github.com/drkjam/netaddr)
* [peewee](https://github.com/coleifer/peewee)
* [BeautifulSoup](https://www.crummy.com/software/BeautifulSoup/)
The project is currently in beta status, but it is already used in a productive enterprise environment which processes about a million e-mails per month.
## Installation
* Install pyquarantine with pip and copy the example config file.
```sh
pip install pyquarantine
cp /etc/pyquarantine/pyquarantine.conf.example /etc/pyquarantine/pyquarantine.conf
```
* Modify /etc/pyquarantine/pyquarantine.conf according to your needs.
## Requirements
* pymilter <https://pythonhosted.org/pymilter/>
* netaddr <https://github.com/drkjam/netaddr/>
* peewee <https://github.com/coleifer/peewee/>
* BeautifulSoup <https://www.crummy.com/software/BeautifulSoup/>
## Configuration options
pyquarantine uses a config file in JSON format. The config file has to be JSON valid with the exception of allowed comment lines starting with **#**. The options are described below.
Rules and actions are processed in the given order.
## Configuration
The pyquarantine module uses an INI-style configuration file. The sections are described below. If you have to specify a path in the config, you can always use a relative path to the last loaded config file.
### Section "global"
Any available configuration option can be set in the global section as default instead of in a quarantine section.
The following configuration options are mandatory in the global section:
* **quarantines**
Comma-separated, ordered list of active quarantines. For each, there must be a section of the same name in the configuration.
* **preferred_quarantin_action**
Defines which quarantine action should be preferred if multiple quarantines are matching for multiple recipients.
If at least one recipient receives the original e-mail due to whitelisting, the action is always ACCEPT.
Possible values are:
* **first**
* **last**
### Quarantine sections
The following configuration options are mandatory in each quarantine section:
* **regex**
Case insensitive regular expression to filter e-mail headers.
* **storage_type**
One of the storage types described below.
* **action**
One of the actions described below.
* **notification_type**
One of the notification types described below.
* **whitelist_type**
One of the whitelist types described below.
* **smtp_host**
SMTP host used to release original e-mails from the quarantine.
* **smtp_port**
SMTP port
The following configuration options are optional in each quarantine section:
* **host_whitelist**
Comma-separated list of host and network addresses to be ignored by this quarantine.
* **reject_reason**
Reason to return to the client if action is set to reject.
### Storage types
* **NONE**
Original e-mails scrapped, sent to nirvana, black-holed or however you want to call it.
* **FILE**
Original e-mails are stored on the filesystem with a unique filename. The filename is available as a
template variable used in notifiaction templates.
The following configuration options are mandatory for this quarantine type:
* **storage_directory**
The directory in which quarantined e-mails are stored.
### Notification types
* **NONE**
No quarantine notifications will be sent.
* **EMAIL**
Quarantine e-mail notifications are sent to recipients. The SMTP host and port, E-mail template, from-address and the subject are configurable for each quarantine. The templates must contain the notification e-mail text in HTML form.
The following template variables are available:
* **{EMAIL_ENVELOPE_FROM}**
E-mail sender address received by the milter.
* **{EMAIL_ENVELOPE_FROM_URL}**
Like EMAIL_ENVELOPE_FROM, but URL encoded
* **{EMAIL_FROM}**
Value of the FROM header of the original e-mail.
* **{EMAIL_ENVELOPE_TO}**
E-mail recipient address of this notification.
* **{EMAIL_ENVELOPE_TO_URL}**
Like EMAIL_ENVELOPE_TO, but URL encoded
* **{EMAIL_TO}**
Value of the TO header of the original e-mail.
* **{EMAIL_SUBJECT}**
Configured e-mail subject.
* **{EMAIL_QUARANTINE_ID}**
Quarantine-ID of the original e-mail if available, empty otherwise.
* **{EMAIL_HTML_TEXT}**
Sanitized version of the e-mail text part of the original e-mail. Only harmless HTML tags and attributes are included. Images are optionally stripped or replaced with the image set by notification_email_replacement_img option.
Some template variables are only available if the regex of the matching quarantine contains subgroups or named subgroups (python syntax). This is useful to include information (e.g. virus names, spam points, ...) of the matching header within the notification.
The following dynamic template variables are available:
* **{SUBGROUP_n}**
Content of a subgroup, 'n' will be replaced by the index number of each subgroup, starting with 0.
* **{subgroup_name}**
Content of a named subgroup, 'subgroup_name' will be replaced by its name.
The following configuration options are mandatory for this notification type:
* **notification_email_smtp_host**
SMTP host used to send notification e-mails.
* **notification_email_smtp_port**
SMTP port.
* **notification_email_envelope_from**
Notification e-mail envelope from-address.
* **notification_email_from**
Value of the notification e-mail from header. Optionally, you may use the EMAIL_FROM template variable described above.
* **notification_email_subject**
Notification e-mail subject. Optionally, you may use the EMAIL_SUBJECT template variable described above.
* **notification_email_template**
Path to the notification e-mail template. It is hold in memory during runtime.
* **notification_email_embedded_imgs**
Comma-separated list of images to embed into the notification e-mail. The Content-ID of each image will be set to the filename, so you can reference it from the e-mail template. All images are hold in memory during runtime.
Leave empty to disable.
The following configuration options are optional for this notification type:
* **notification_email_strip_images**
Enable to strip images from e-mails. This option superseeds notification_email_replacement_img.
* **notification_email_replacement_img**
Path to an image to replace images in e-mails. It is hold in memory during runtime.
* **notification_email_parser_lib**
HTML parser library used to parse text part of emails.
### Global
Config options in **global** section:
* **socket** (optional)
The socket used to communicate with the MTA. If it is not specified in the config, it has to be set as command line option.
* **local_addrs** (optional)
A list of hosts and network addresses which are considered local. It is used to for the condition option [local](#Conditions).
Default: **fe80::/64, ::1/128, 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16**
* **loglevel** (optional)
Set the log level. This option may be overriden by any rule or action object. Possible values are:
* **error**
* **warning**
* **info**
* **debug**
Default: **info**
* **pretend** (optional)
Pretend actions, for test purposes. This option may be overriden by any rule or action object.
### Rules
Config options for **rule** objects:
* **name** (optional)
Name of the rule.
Default: **Rule #n**
* **actions**
A list of action objects which are processed in the given order.
* **conditions** (optional)
A list of conditions which all have to be true to process the actions.
* **loglevel** (optional)
As described above in the [Global](#Global) section.
* **pretend** (optional)
As described above in the [Global](#Global) section.
### Actions
Every quarantine responds with a milter-action if an e-mail header matches the configured regular expression. Please think carefully what you set here or your MTA will do something you do not want.
The following actions are available:
* **ACCEPT**
Tell the MTA to continue processing the e-mail.
* **DISCARD**
Tell the MTA to silently discard the e-mail.
* **REJECT**
Tell the MTA to reject the e-mail.
Config options for **action** objects:
* **name** (optional)
Name of the action.
Default: **Action #n**
* **type**
Action type. Possible values are:
* **add_header**
* **del_header**
* **mod_header**
* **add_disclaimer**
* **store**
* **conditions** (optional)
A list of conditions which all have to be true to process the action.
* **loglevel** (optional)
As described above in the [Global](#Global) section.
* **pretend** (optional)
As described above in the [Global](#Global) section.
Config options for **add_header** actions:
* **field**
Name of the header.
* **value**
Value of the header.
### Whitelist types
* **NONE**
No whitelist will be used.
Config options for **del_header** actions:
* **field**
Regular expression to match against header names.
* **value** (optional)
Regular expression to match against the headers value.
* **DB**
A database whitelist will be used. All database types supported by peewee are available.
Config options for **mod_header** actions:
* **field**
Regular expression to match against header names.
* **search** (optional)
Regular expression to match against header values. You may use subgroups or named subgroups (python syntax) to include parts of the original value in the new value.
* **value**
New value of the header.
The following configuration options are mandatory for this whitelist type:
* **whitelist_db_connection**
Database connection string (e.g. mysql://user:password@host:port).
Config options for **add_disclaimer** actions:
* **action**
Action to perform with the disclaimer. Possible values are:
* append
* prepend
* **html_template**
Path to a file which contains the html representation of the disclaimer.
* **text_template**
Path to a file which contains the text representation of the disclaimer.
* **error_policy** (optional)
Set the error policy in case the disclaimer cannot be added (e.g. if no body part is present in the e-mail). Possible values are:
* **wrap**
A new e-mail body is generated with the disclaimer as body and the original e-mail attached.
* **ignore**
Ignore the error and do nothing.
* **reject**
Reject the e-mail.
Default: **wrap**
* **whitelist_db_table**
Database table to use.
Config options for **store** actions:
* **storage_type**
Storage type. Possible values are:
* **file**
* **original** (optional)
Default: **false**
If set to true, store the message as received by the MTA instead of storing the current state of the message, that may was modified already by other actions.
Config options for **file** storage:
* **directory**
Directory used to store e-mails.
### Conditions
Config options for **conditions** objects:
* **local** (optional)
If set to true, the rule is only executed for e-mails originating from addresses defined in local_addrs and vice versa.
* **hosts** (optional)
A list of hosts and network addresses for which the rule should be executed.
* **envfrom** (optional)
A regular expression to match against the evenlope-from addresses for which the rule should be executed.
* **envto** (optional)
A regular expression to match against all evenlope-to addresses. All addresses must match to fulfill the condition.
## Developer information
Everyone who wants to improve or extend this project is very welcome.

905
pyquarantine/__init__.py
View File

@@ -1,717 +1,324 @@
# PyQuarantine-Milter is free software: you can redistribute it and/or modify
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# PyQuarantine-Milter is distributed in the hope that it will be useful,
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with PyQuarantineMilter. If not, see <http://www.gnu.org/licenses/>.
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = [
"action",
"base",
"cli",
"conditions",
"config",
"mailer",
"modify",
"notify",
"rule",
"run",
"storage",
"whitelist",
"ModifyMilter"]
__version__ = "2.0.0"
from pyquarantine import _runtime_patches
import Milter
import configparser
import logging
import os
import re
import encodings
from Milter.utils import parse_addr
from collections import defaultdict
from email.charset import Charset
from email.header import Header, decode_header
from copy import copy
from email import message_from_binary_file
from email.header import Header, decode_header, make_header
from email.headerregistry import AddressHeader, _default_header_map
from email.policy import SMTPUTF8
from io import BytesIO
from itertools import groupby
from netaddr import IPAddress, IPNetwork, AddrFormatError
from pyquarantine import mailer
from pyquarantine import notifications
from pyquarantine import storages
from pyquarantine import whitelists
from netaddr import IPNetwork, AddrFormatError
__all__ = [
"make_header",
"Quarantine",
"QuarantineMilter",
"setup_milter",
"reload_config",
"cli",
"mailer",
"notifications",
"storages",
"run",
"whitelists"]
__version__ = "1.0.10"
from pyquarantine.base import CustomLogger, MilterMessage
from pyquarantine.base import replace_illegal_chars
from pyquarantine.rule import Rule
################################################
# add charset alias for windows-874 encoding #
################################################
class ModifyMilter(Milter.Base):
"""ModifyMilter based on Milter.Base to implement milter communication"""
aliases = encodings.aliases.aliases
_rules = []
_loglevel = logging.INFO
_addr_fields = [f for f, v in _default_header_map.items()
if issubclass(v, AddressHeader)]
for alias in ["windows-874", "windows_874"]:
if alias not in aliases:
aliases[alias] = "cp874"
@staticmethod
def set_config(cfg, debug):
ModifyMilter._loglevel = cfg.get_loglevel(debug)
setattr(encodings.aliases, "aliases", aliases)
################################################
def make_header(decoded_seq, maxlinelen=None, header_name=None,
continuation_ws=' ', errors='strict'):
"""Create a Header from a sequence of pairs as returned by decode_header()
decode_header() takes a header value string and returns a sequence of
pairs of the format (decoded_string, charset) where charset is the string
name of the character set.
This function takes one of those sequence of pairs and returns a Header
instance. Optional maxlinelen, header_name, and continuation_ws are as in
the Header constructor.
"""
h = Header(maxlinelen=maxlinelen, header_name=header_name,
continuation_ws=continuation_ws)
for s, charset in decoded_seq:
# None means us-ascii but we can simply pass it on to h.append()
if charset is not None and not isinstance(charset, Charset):
charset = Charset(charset)
h.append(s, charset, errors=errors)
return h
class Quarantine(object):
"""Quarantine class suitable for QuarantineMilter
The class holds all the objects and functions needed
for QuarantineMilter quarantine.
"""
# list of possible actions
_actions = {
"ACCEPT": Milter.ACCEPT,
"REJECT": Milter.REJECT,
"DISCARD": Milter.DISCARD}
def __init__(self, name, index=0, regex=None, storage=None, whitelist=None,
host_whitelist=[], notification=None, action="ACCEPT",
reject_reason=None):
self.logger = logging.getLogger(__name__)
self.name = name
self.index = index
self.regex = regex
if regex:
self.re = re.compile(
regex, re.MULTILINE + re.DOTALL + re.IGNORECASE)
self.storage = storage
self.whitelist = whitelist
self.host_whitelist = host_whitelist
self.notification = notification
action = action.upper()
assert action in self._actions
self.action = action
self.milter_action = self._actions[action]
self.reject_reason = reject_reason
def setup_from_cfg(self, global_cfg, cfg, test=False):
defaults = {
"action": "accept",
"reject_reason": "Message rejected",
"storage_type": "none",
"notification_type": "none",
"whitelist_type": "none",
"host_whitelist": ""
}
# check config
for opt in ["regex", "smtp_host", "smtp_port"] + list(defaults.keys()):
if opt in cfg:
continue
if opt in global_cfg:
cfg[opt] = global_cfg[opt]
elif opt in defaults:
cfg[opt] = defaults[opt]
else:
raise RuntimeError(
f"mandatory option '{opt}' not present in "
f"config section '{self.name}' or 'global'")
# pre-compile regex
self.logger.debug(
f"{self.name}: compiling regex '{cfg['regex']}'")
self.re = re.compile(
cfg["regex"], re.MULTILINE + re.DOTALL + re.IGNORECASE)
self.smtp_host = cfg["smtp_host"]
self.smtp_port = cfg["smtp_port"]
# create storage instance
storage_type = cfg["storage_type"].lower()
if storage_type in storages.TYPES:
self.logger.debug(
f"{self.name}: initializing storage "
f"type '{storage_type.upper()}'")
self.storage = storages.TYPES[storage_type](
self.name, global_cfg, cfg, test)
elif storage_type == "none":
self.logger.debug(f"{self.name}: storage is NONE")
self.storage = None
else:
raise RuntimeError(
f"{self.name}: unknown storage type '{storage_type}'")
# create whitelist instance
whitelist_type = cfg["whitelist_type"].lower()
if whitelist_type in whitelists.TYPES:
self.logger.debug(
f"{self.name}: initializing whitelist "
f"type '{whitelist_type.upper()}'")
self.whitelist = whitelists.TYPES[whitelist_type](
self.name, global_cfg, cfg, test)
elif whitelist_type == "none":
self.logger.debug(f"{self.name}: whitelist is NONE")
self.whitelist = None
else:
raise RuntimeError(
f"{self.name}: unknown whitelist type '{whitelist_type}'")
# create notification instance
notification_type = cfg["notification_type"].lower()
if notification_type in notifications.TYPES:
self.logger.debug(
f"{self.name}: initializing notification "
f"type '{notification_type.upper()}'")
self.notification = notifications.TYPES[notification_type](
self.name, global_cfg, cfg, test)
elif notification_type == "none":
self.logger.debug(f"{self.name}: notification is NONE")
self.notification = None
else:
raise RuntimeError(
f"{self.name}: unknown notification "
f"type '{notification_type}'")
# determining milter action for this quarantine
action = cfg["action"].upper()
if action in self._actions:
self.logger.debug(f"{self.name}: action is {action}")
self.action = action
self.milter_action = self._actions[action]
else:
raise RuntimeError(
f"{self.name}: unknown action '{action}'")
self.reject_reason = cfg["reject_reason"]
# create host/network whitelist
self.host_whitelist = []
host_whitelist = set([p.strip()
for p in cfg["host_whitelist"].split(",") if p])
for host in host_whitelist:
if not host:
continue
# parse network notation
try:
net = IPNetwork(host)
local_addrs = []
for addr in cfg["local_addrs"]:
local_addrs.append(IPNetwork(addr))
except AddrFormatError as e:
raise RuntimeError(
f"{self.name}: error parsing host_whitelist: {e}")
else:
self.host_whitelist.append(net)
if self.host_whitelist:
whitelist = ", ".join([str(ip) for ip in host_whitelist])
self.logger.debug(
f"{self.name}: host whitelist: {whitelist}")
raise RuntimeError(e)
def notify(self, storage_id, recipient=None, synchronous=True):
"Notify recipient about email in storage."
if not self.storage:
raise RuntimeError(
"storage type is set to None, unable to send notification")
logger = logging.getLogger(__name__)
logger.setLevel(ModifyMilter._loglevel)
for idx, rule_cfg in enumerate(cfg["rules"]):
if "name" not in rule_cfg:
rule_cfg["name"] = f"rule#{idx}"
if "loglevel" not in rule_cfg:
rule_cfg["loglevel"] = cfg["loglevel"]
if "pretend" not in rule_cfg:
rule_cfg["pretend"] = cfg["pretend"]
rule = Rule(rule_cfg, local_addrs, debug)
if not self.notification:
raise RuntimeError(
"notification type is set to None, "
"unable to send notification")
fp, metadata = self.storage.get_mail(storage_id)
if recipient is not None:
if recipient not in metadata["recipients"]:
raise RuntimeError(f"invalid recipient '{recipient}'")
recipients = [recipient]
else:
recipients = metadata["recipients"]
self.notification.notify(
metadata["queue_id"], storage_id, metadata["mailfrom"],
recipients, metadata["headers"], fp,
metadata["subgroups"], metadata["named_subgroups"],
synchronous)
fp.close()
def release(self, storage_id, recipients=None):
"Release email from storage."
if not self.storage:
raise RuntimeError(
"storage type is set to None, unable to release email")
fp, metadata = self.storage.get_mail(storage_id)
try:
mail = fp.read()
fp.close()
except IOError as e:
raise RuntimeError(f"unable to read data file: {e}")
if recipients and type(recipients) == str:
recipients = [recipients]
else:
recipients = metadata["recipients"]
for recipient in recipients:
if recipient not in metadata["recipients"]:
raise RuntimeError(f"invalid recipient '{recipient}'")
try:
mailer.smtp_send(
self.smtp_host,
self.smtp_port,
metadata["mailfrom"],
recipient,
mail)
except Exception as e:
raise RuntimeError(
f"error while sending email to '{recipient}': {e}")
self.storage.delete(storage_id, recipient)
def get_storage(self):
return self.storage
def get_notification(self):
return self.notification
def get_whitelist(self):
return self.whitelist
def host_in_whitelist(self, hostaddr):
ip = IPAddress(hostaddr[0])
for entry in self.host_whitelist:
if ip in entry:
return True
return False
def match(self, header):
return self.re.search(header)
class QuarantineMilter(Milter.Base):
"""QuarantineMilter based on Milter.Base to implement milter communication
The class variable quarantines needs to be filled by
runng the setup_milter function.
"""
quarantines = []
preferred_action = "first"
# list of default config files
_cfg_files = [
"/etc/pyquarantine/pyquarantine.conf",
os.path.expanduser('~/pyquarantine.conf'),
"pyquarantine.conf"]
logger.debug(rule)
ModifyMilter._rules.append(rule)
def __init__(self):
self.logger = logging.getLogger(__name__)
# save runtime config, it must not change during runtime
self.quarantines = QuarantineMilter.quarantines
self.logger.setLevel(ModifyMilter._loglevel)
def _get_preferred_quarantine(self):
matching_quarantines = [
q for q in self.rcpts_quarantines.values() if q]
if self.preferred_action == "first":
quarantine = sorted(
matching_quarantines,
key=lambda q: q.index)[0]
def addheader(self, field, value, idx=-1):
value = replace_illegal_chars(Header(s=value).encode())
self.logger.debug(f"addheader: {field}: {value}")
super().addheader(field, value, idx)
def chgheader(self, field, value, idx=1):
value = replace_illegal_chars(Header(s=value).encode())
if value:
self.logger.debug(f"chgheader: {field}[{idx}]: {value}")
else:
quarantine = sorted(
matching_quarantines,
key=lambda q: q.index,
reverse=True)[0]
return quarantine
self.logger.debug(f"delheader: {field}[{idx}]")
super().chgheader(field, idx, value)
@staticmethod
def get_cfg_files():
return QuarantineMilter._cfg_files
def update_headers(self, old_headers):
if self.msg.is_multipart() and not self.msg["MIME-Version"]:
self.msg.add_header("MIME-Version", "1.0")
@staticmethod
def set_cfg_files(cfg_files):
QuarantineMilter._cfg_files = cfg_files
# serialize the message object so it updates its internal strucure
self.msg.as_bytes()
def connect(self, hostname, family, hostaddr):
self.hostaddr = hostaddr
self.logger.debug(
f"accepted milter connection from {hostaddr[0]} "
f"port {hostaddr[1]}")
for quarantine in self.quarantines.copy():
if quarantine.host_in_whitelist(hostaddr):
self.logger.debug(
f"host {hostaddr[0]} is in whitelist of "
f"quarantine {quarantine.name}")
self.quarantines.remove(quarantine)
if not self.quarantines:
self.logger.debug(
f"host {hostaddr[0]} is in whitelist of all "
f"quarantines, skip further processing")
return Milter.ACCEPT
return Milter.CONTINUE
old_headers = [(f, f.lower(), v) for f, v in old_headers]
headers = [(f, f.lower(), v) for f, v in self.msg.items()]
@Milter.noreply
def envfrom(self, mailfrom, *str):
self.mailfrom = "@".join(parse_addr(mailfrom)).lower()
self.recipients = set()
return Milter.CONTINUE
idx = defaultdict(int)
for field, field_lower, value in old_headers:
idx[field_lower] += 1
if (field, field_lower, value) not in headers:
self.chgheader(field, "", idx=idx[field_lower])
idx[field] -= 1
@Milter.noreply
def envrcpt(self, to, *str):
self.recipients.add("@".join(parse_addr(to)).lower())
return Milter.CONTINUE
for field, value in self.msg.items():
field_lower = field.lower()
if (field, field_lower, value) not in old_headers:
self.addheader(field, value)
@Milter.noreply
def data(self):
self.qid = self.getsymval('i')
self.logger.debug(
f"{self.qid}: received queue-id from MTA")
self.recipients = list(self.recipients)
self.logger.debug(
f"{self.qid}: initializing memory buffer to save email data")
# initialize memory buffer to save email data
self.fp = BytesIO()
self.headers = []
return Milter.CONTINUE
def replacebody(self):
self._body_changed = True
def header(self, name, value):
def _replacebody(self):
if not self._body_changed:
return
data = self.msg.as_bytes()
body_pos = data.find(b"\r\n\r\n") + 4
self.logger.debug("replace body")
super().replacebody(data[body_pos:])
del data
def delrcpt(self, rcpts):
"Remove recipient. May be called from eom callback only."
if not isinstance(rcpts, list):
rcpts = [rcpts]
for rcpt in rcpts:
self.logger.debug(f"delrcpt: {rcpt}")
self.msginfo["rcpts"].remove(rcpt)
super().delrcpt(rcpt)
def connect(self, IPname, family, hostaddr):
try:
# remove surrogates from value
value = value.encode(
errors="surrogateescape").decode(errors="replace")
self.logger.debug(f"{self.qid}: received header: {name}: {value}")
# write email header to memory buffer
self.fp.write(f"{name}: {value}\r\n".encode(
encoding="ascii", errors="replace"))
header = make_header(decode_header(value), errors="replace")
value = str(header).replace("\x00", "")
if hostaddr is None:
self.logger.error(f"received invalid host address {hostaddr}, "
f"unable to proceed")
return Milter.TEMPFAIL
self.IP = hostaddr[0]
self.port = hostaddr[1]
self.logger.debug(
f"{self.qid}: decoded header: {name}: {value}")
self.headers.append((name, value))
return Milter.CONTINUE
f"accepted milter connection from {self.IP} "
f"port {self.port}")
# pre-filter rules and actions by the host condition
# also check if the mail body is needed by any upcoming action.
self.rules = []
self._headersonly = True
for rule in ModifyMilter._rules:
if rule.conditions is None or \
rule.conditions.match_host(self.IP):
actions = []
for action in rule.actions:
if action.conditions is None or \
action.conditions.match_host(self.IP):
actions.append(action)
if not action.headersonly():
self._headersonly = False
if actions:
# copy needed rules to preserve configured actions
rule = copy(rule)
rule.actions = actions
self.rules.append(rule)
if not self.rules:
self.logger.debug(
"host is ignored by all rules, skip further processing")
return Milter.ACCEPT
except Exception as e:
self.logger.exception(
f"an exception occured in header function: {e}")
f"an exception occured in connect method: {e}")
return Milter.TEMPFAIL
return Milter.CONTINUE
def hello(self, heloname):
try:
self.heloname = heloname
self.logger.debug(f"received HELO name: {heloname}")
except Exception as e:
self.logger.exception(
f"an exception occured in hello method: {e}")
return Milter.TEMPFAIL
return Milter.CONTINUE
def envfrom(self, mailfrom, *str):
try:
self.mailfrom = "@".join(parse_addr(mailfrom)).lower()
self.rcpts = set()
except Exception as e:
self.logger.exception(
f"an exception occured in envfrom method: {e}")
return Milter.TEMPFAIL
return Milter.CONTINUE
def envrcpt(self, to, *str):
try:
self.rcpts.add("@".join(parse_addr(to)).lower())
except Exception as e:
self.logger.exception(
f"an exception occured in envrcpt method: {e}")
return Milter.TEMPFAIL
return Milter.CONTINUE
def data(self):
try:
self.qid = self.getsymval('i')
self.logger = CustomLogger(
self.logger, {"qid": self.qid, "name": "milter"})
self.logger.debug("received queue-id from MTA")
self.fp = BytesIO()
except Exception as e:
self.logger.exception(
f"an exception occured in data method: {e}")
return Milter.TEMPFAIL
return Milter.CONTINUE
def header(self, field, value):
try:
# remove CR and LF from address fields, otherwise pythons
# email library throws an exception
if field.lower() in ModifyMilter._addr_fields:
try:
v = str(make_header(decode_header(value)))
except Exception as e:
self.logger.error(
f"unable to decode field '{field}': {e}")
else:
if any(c in v for c in ["\r", "\n"]):
v = v.replace("\r", "").replace("\n", "")
value = Header(s=v).encode()
# remove surrogates
field = field.encode("ascii", errors="replace")
value = value.encode("ascii", errors="replace")
self.fp.write(field + b": " + value + b"\r\n")
except Exception as e:
self.logger.exception(
f"an exception occured in header method: {e}")
return Milter.TEMPFAIL
return Milter.CONTINUE
def eoh(self):
try:
self.fp.write("\r\n".encode(encoding="ascii"))
self.wl_cache = whitelists.WhitelistCache()
# initialize dicts to set quaranines per recipient and keep matches
self.rcpts_quarantines = {}
self.matches = {}
# iterate email headers
rcpts_to_check = self.recipients.copy()
for name, value in self.headers:
header = f"{name}: {value}"
self.logger.debug(
f"{self.qid}: checking header against configured "
f"quarantines: {header}")
# iterate quarantines
for quarantine in self.quarantines:
if len(self.rcpts_quarantines) == len(
self.recipients):
# every recipient matched a quarantine already
if quarantine.index >= max(
[q.index for q in
self.rcpts_quarantines.values()]):
# all recipients matched a quarantine with at least
# the same precedence already, skip checks against
# quarantines with lower precedence
self.logger.debug(
f"{self.qid}: {quarantine.name}: skip further "
f"checks of this header")
break
# check email header against quarantine regex
self.logger.debug(
f"{self.qid}: {quarantine.name}: checking header "
f"against regex '{str(quarantine.regex)}'")
match = quarantine.match(header)
if match:
self.logger.debug(
f"{self.qid}: {quarantine.name}: "
f"header matched regex")
# check for whitelisted recipients
whitelist = quarantine.get_whitelist()
if whitelist:
try:
wl_recipients = self.wl_cache.get_recipients(
whitelist,
self.mailfrom,
rcpts_to_check)
except RuntimeError as e:
self.logger.error(
f"{self.qid}: {quarantine.name}: unable "
f"to query whitelist: {e}")
return Milter.TEMPFAIL
else:
wl_recipients = {}
# iterate recipients
for rcpt in rcpts_to_check.copy():
if rcpt in wl_recipients:
# recipient is whitelisted in this quarantine
self.logger.debug(
f"{self.qid}: {quarantine.name}: "
f"recipient '{rcpt}' is whitelisted")
continue
if rcpt not in self.rcpts_quarantines or \
self.rcpts_quarantines[rcpt].index > \
quarantine.index:
self.logger.debug(
f"{self.qid}: {quarantine.name}: set "
f"quarantine for recipient '{rcpt}'")
# save match for later use as template
# variables
self.matches[quarantine.name] = match
self.rcpts_quarantines[rcpt] = quarantine
if quarantine.index == 0:
# we do not need to check recipients which
# matched the quarantine with the highest
# precedence already
rcpts_to_check.remove(rcpt)
else:
self.logger.debug(
f"{self.qid}: {quarantine.name}: a "
f"quarantine with same or higher "
f"precedence matched already for "
f"recipient '{rcpt}'")
if not rcpts_to_check:
self.logger.debug(
f"{self.qid}: all recipients matched the first "
f"quarantine, skipping all remaining header checks")
break
# check if no quarantine has matched for all recipients
if not self.rcpts_quarantines:
# accept email
self.logger.info(
f"{self.qid}: passed clean for all recipients")
return Milter.ACCEPT
# check if the mail body is needed
for recipient, quarantine in self.rcpts_quarantines.items():
if quarantine.get_storage() or quarantine.get_notification():
# mail body is needed, continue processing
return Milter.CONTINUE
# quarantine and notification are disabled on all matching
# quarantines, just return configured action
quarantine = self._get_preferred_quarantine()
self.logger.info(
f"{self.qid}: {self.preferred_action} "
f"matching quarantine is '{quarantine.name}', performing "
f"milter action {quarantine.action}")
if quarantine.action == "REJECT":
self.setreply("554", "5.7.0", quarantine.reject_reason)
return quarantine.milter_action
self.fp.write(b"\r\n")
except Exception as e:
self.logger.exception(
f"an exception occured in eoh function: {e}")
f"an exception occured in eoh method: {e}")
return Milter.TEMPFAIL
return Milter.CONTINUE
def body(self, chunk):
try:
# save received body chunk
if not self._headersonly:
self.fp.write(chunk)
except Exception as e:
self.logger.exception(
f"an exception occured in body function: {e}")
f"an exception occured in body method: {e}")
return Milter.TEMPFAIL
return Milter.CONTINUE
def eom(self):
try:
# processing recipients grouped by quarantines
quarantines = []
for quarantine, recipients in groupby(
sorted(self.rcpts_quarantines,
key=lambda x: self.rcpts_quarantines[x].index),
lambda x: self.rcpts_quarantines[x]):
quarantines.append((quarantine, list(recipients)))
# msg and msginfo contain the runtime data that
# is read/modified by actions
self.fp.seek(0)
self.msg = message_from_binary_file(
self.fp, _class=MilterMessage, policy=SMTPUTF8.clone(
refold_source='none'))
self.msginfo = {
"mailfrom": self.mailfrom,
"rcpts": [*self.rcpts],
"vars": {}}
# iterate quarantines sorted by index
for quarantine, recipients in sorted(
quarantines, key=lambda x: x[0].index):
headers = defaultdict(str)
for name, value in self.headers:
headers[name.lower()] = value
subgroups = self.matches[quarantine.name].groups(
default="")
named_subgroups = self.matches[quarantine.name].groupdict(
default="")
self._body_changed = False
milter_action = None
for rule in self.rules:
milter_action = rule.execute(self)
self.logger.debug(
f"current template variables: {self.msginfo['vars']}")
if milter_action is not None:
break
rcpts = ", ".join(recipients)
# check if a storage is configured
storage_id = ""
storage = quarantine.get_storage()
if storage:
# add email to quarantine
self.logger.info(
f"{self.qid}: adding to quarantine "
f"'{quarantine.name}' for: {rcpts}")
try:
storage_id = storage.add(
self.qid, self.mailfrom, recipients, headers,
self.fp, subgroups, named_subgroups)
except RuntimeError as e:
self.logger.error(
f"{self.qid}: unable to add to quarantine "
f"'{quarantine.name}': {e}")
return Milter.TEMPFAIL
# check if a notification is configured
notification = quarantine.get_notification()
if notification:
# notify
self.logger.info(
f"{self.qid}: sending notification to: {rcpts}")
try:
notification.notify(
self.qid, storage_id,
self.mailfrom, recipients, headers, self.fp,
subgroups, named_subgroups)
except RuntimeError as e:
self.logger.error(
f"{self.qid}: unable to send notification: {e}")
return Milter.TEMPFAIL
# remove processed recipient
for recipient in recipients:
self.delrcpt(recipient)
self.recipients.remove(recipient)
self.fp.close()
# email passed clean for at least one recipient, accepting email
if self.recipients:
rcpts = ", ".join(recipients)
self.logger.info(
f"{self.qid}: passed clean for: {rcpts}")
return Milter.ACCEPT
# return configured action
quarantine = self._get_preferred_quarantine()
self.logger.info(
f"{self.qid}: {self.preferred_action} matching "
f"quarantine is '{quarantine.name}', performing milter "
f"action {quarantine.action}")
if quarantine.action == "REJECT":
self.setreply("554", "5.7.0", quarantine.reject_reason)
return quarantine.milter_action
if milter_action is None:
self._replacebody()
else:
action, reason = milter_action
if action == "ACCEPT":
self._replacebody()
elif action == "REJECT":
self.setreply("554", "5.7.0", reason)
return Milter.REJECT
elif action == "DISCARD":
return Milter.DISCARD
except Exception as e:
self.logger.exception(
f"an exception occured in eom function: {e}")
f"an exception occured in eom method: {e}")
return Milter.TEMPFAIL
def close(self):
self.logger.debug(
f"disconnect from {self.hostaddr[0]} port {self.hostaddr[1]}")
return Milter.CONTINUE
def setup_milter(test=False, cfg_files=[]):
"Generate the configuration for QuarantineMilter class."
logger = logging.getLogger(__name__)
# read config file
parser = configparser.ConfigParser()
if not cfg_files:
cfg_files = parser.read(QuarantineMilter.get_cfg_files())
else:
cfg_files = parser.read(cfg_files)
if not cfg_files:
raise RuntimeError("config file not found")
QuarantineMilter.set_cfg_files(cfg_files)
os.chdir(os.path.dirname(cfg_files[0]))
# check if mandatory config options in global section are present
if "global" not in parser.sections():
raise RuntimeError(
"mandatory section 'global' not present in config file")
for option in ["quarantines", "preferred_quarantine_action"]:
if not parser.has_option("global", option):
raise RuntimeError(
f"mandatory option '{option}' not present in config "
f"section 'global'")
# read global config section
global_cfg = dict(parser.items("global"))
preferred_action = global_cfg["preferred_quarantine_action"].lower()
if preferred_action not in ["first", "last"]:
raise RuntimeError(
"option preferred_quarantine_action has illegal value")
# read active quarantine names
quarantines = [
q.strip() for q in global_cfg["quarantines"].split(",")]
if len(quarantines) != len(set(quarantines)):
raise RuntimeError(
"at least one quarantine is specified multiple times "
"in quarantines option")
if "global" in quarantines:
quarantines.remove("global")
logger.warning(
"removed illegal quarantine name 'global' from list of "
"active quarantines")
if not quarantines:
raise RuntimeError("no quarantines configured")
milter_quarantines = []
logger.debug("preparing milter configuration ...")
# iterate quarantine names
for index, name in enumerate(quarantines):
# check if config section for current quarantine exists
if name not in parser.sections():
raise RuntimeError(
f"config section '{name}' does not exist")
cfg = dict(parser.items(name))
quarantine = Quarantine(name, index)
quarantine.setup_from_cfg(global_cfg, cfg, test)
milter_quarantines.append(quarantine)
QuarantineMilter.preferred_action = preferred_action
QuarantineMilter.quarantines = milter_quarantines
def reload_config():
"Reload the configuration of QuarantineMilter class."
logger = logging.getLogger(__name__)
try:
setup_milter()
except RuntimeError as e:
logger.info(e)
logger.info("daemon is still running with previous configuration")
else:
logger.info("reloaded configuration")
return Milter.ACCEPT

229
pyquarantine/_runtime_patches.py Normal file
View File

@@ -0,0 +1,229 @@
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
import encodings
#####################################
# patch pythons email library #
#####################################
#
# https://bugs.python.org/issue27257
# https://bugs.python.org/issue30988
#
# fix: https://github.com/python/cpython/pull/15600
import email._header_value_parser
from email._header_value_parser import TokenList, NameAddr
from email._header_value_parser import get_display_name, get_angle_addr
from email._header_value_parser import get_cfws, errors
from email._header_value_parser import CFWS_LEADER, PHRASE_ENDS
class DisplayName(email._header_value_parser.DisplayName):
@property
def display_name(self):
res = TokenList(self)
if len(res) == 0:
return res.value
if res[0].token_type == 'cfws':
res.pop(0)
else:
if isinstance(res[0], TokenList) and \
res[0][0].token_type == 'cfws':
res[0] = TokenList(res[0][1:])
if res[-1].token_type == 'cfws':
res.pop()
else:
if isinstance(res[-1], TokenList) and \
res[-1][-1].token_type == 'cfws':
res[-1] = TokenList(res[-1][:-1])
return res.value
def get_name_addr(value):
""" name-addr = [display-name] angle-addr
"""
name_addr = NameAddr()
# Both the optional display name and the angle-addr can start with cfws.
leader = None
if value[0] in CFWS_LEADER:
leader, value = get_cfws(value)
if not value:
raise errors.HeaderParseError(
"expected name-addr but found '{}'".format(leader))
if value[0] != '<':
if value[0] in PHRASE_ENDS:
raise errors.HeaderParseError(
"expected name-addr but found '{}'".format(value))
token, value = get_display_name(value)
if not value:
raise errors.HeaderParseError(
"expected name-addr but found '{}'".format(token))
if leader is not None:
if isinstance(token[0], TokenList):
token[0][:0] = [leader]
else:
token[:0] = [leader]
leader = None
name_addr.append(token)
token, value = get_angle_addr(value)
if leader is not None:
token[:0] = [leader]
name_addr.append(token)
return name_addr, value
setattr(email._header_value_parser, "DisplayName", DisplayName)
setattr(email._header_value_parser, "get_name_addr", get_name_addr)
# https://bugs.python.org/issue42484
#
# fix: https://github.com/python/cpython/pull/24669
from email._header_value_parser import DOT, ObsLocalPart, ValueTerminal, get_word
def get_obs_local_part(value):
""" obs-local-part = word *("." word)
"""
obs_local_part = ObsLocalPart()
last_non_ws_was_dot = False
while value and (value[0]=='\\' or value[0] not in PHRASE_ENDS):
if value[0] == '.':
if last_non_ws_was_dot:
obs_local_part.defects.append(errors.InvalidHeaderDefect(
"invalid repeated '.'"))
obs_local_part.append(DOT)
last_non_ws_was_dot = True
value = value[1:]
continue
elif value[0]=='\\':
obs_local_part.append(ValueTerminal(value[0],
'misplaced-special'))
value = value[1:]
obs_local_part.defects.append(errors.InvalidHeaderDefect(
"'\\' character outside of quoted-string/ccontent"))
last_non_ws_was_dot = False
continue
if obs_local_part and obs_local_part[-1].token_type != 'dot':
obs_local_part.defects.append(errors.InvalidHeaderDefect(
"missing '.' between words"))
try:
token, value = get_word(value)
last_non_ws_was_dot = False
except errors.HeaderParseError:
if value[0] not in CFWS_LEADER:
raise
token, value = get_cfws(value)
obs_local_part.append(token)
if not obs_local_part:
return obs_local_part, value
if (obs_local_part[0].token_type == 'dot' or
obs_local_part[0].token_type=='cfws' and
obs_local_part[1].token_type=='dot'):
obs_local_part.defects.append(errors.InvalidHeaderDefect(
"Invalid leading '.' in local part"))
if (obs_local_part[-1].token_type == 'dot' or
obs_local_part[-1].token_type=='cfws' and
obs_local_part[-2].token_type=='dot'):
obs_local_part.defects.append(errors.InvalidHeaderDefect(
"Invalid trailing '.' in local part"))
if obs_local_part.defects:
obs_local_part.token_type = 'invalid-obs-local-part'
return obs_local_part, value
setattr(email._header_value_parser, "get_obs_local_part", get_obs_local_part)
# https://bugs.python.org/issue30681
#
# fix: https://github.com/python/cpython/pull/22090
import email.errors
from email.errors import HeaderDefect
class InvalidDateDefect(HeaderDefect):
"""Header has unparseable or invalid date"""
setattr(email.errors, "InvalidDateDefect", InvalidDateDefect)
import email.utils
from email.utils import _parsedate_tz
import datetime
def parsedate_to_datetime(data):
parsed_date_tz = _parsedate_tz(data)
if parsed_date_tz is None:
raise ValueError('Invalid date value or format "%s"' % str(data))
*dtuple, tz = parsed_date_tz
if tz is None:
return datetime.datetime(*dtuple[:6])
return datetime.datetime(*dtuple[:6],
tzinfo=datetime.timezone(datetime.timedelta(seconds=tz)))
setattr(email.utils, "parsedate_to_datetime", parsedate_to_datetime)
import email.headerregistry
from email import utils, _header_value_parser as parser
@classmethod
def parse(cls, value, kwds):
if not value:
kwds['defects'].append(errors.HeaderMissingRequiredValue())
kwds['datetime'] = None
kwds['decoded'] = ''
kwds['parse_tree'] = parser.TokenList()
return
if isinstance(value, str):
kwds['decoded'] = value
try:
value = utils.parsedate_to_datetime(value)
except ValueError:
kwds['defects'].append(errors.InvalidDateDefect('Invalid date value or format'))
kwds['datetime'] = None
kwds['parse_tree'] = parser.TokenList()
return
kwds['datetime'] = value
kwds['decoded'] = utils.format_datetime(kwds['datetime'])
kwds['parse_tree'] = cls.value_parser(kwds['decoded'])
setattr(email.headerregistry.DateHeader, "parse", parse)
#######################################
# add charset alias for windows-874 #
#######################################
#
# https://bugs.python.org/issue17254
#
# fix: https://github.com/python/cpython/pull/10237
aliases = encodings.aliases.aliases
for alias in ["windows-874", "windows_874"]:
if alias not in aliases:
aliases[alias] = "cp874"
setattr(encodings.aliases, "aliases", aliases)

64
pyquarantine/action.py Normal file
View File

@@ -0,0 +1,64 @@
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = ["Action"]
from pyquarantine import modify, notify, storage
from pyquarantine.conditions import Conditions
class Action:
"""Action to implement a pre-configured action to perform on e-mails."""
ACTION_TYPES = {
"add_header": modify.Modify,
"mod_header": modify.Modify,
"del_header": modify.Modify,
"add_disclaimer": modify.Modify,
"rewrite_links": modify.Modify,
"store": storage.Store,
"notify": notify.Notify,
"quarantine": storage.Quarantine}
def __init__(self, cfg, local_addrs, debug):
self.cfg = cfg
self.conditions = cfg["conditions"] if "conditions" in cfg else None
if self.conditions is not None:
self.conditions["name"] = f"{cfg['name']}: conditions"
self.conditions["loglevel"] = cfg["loglevel"]
self.conditions = Conditions(self.conditions, local_addrs, debug)
action_type = cfg["type"]
self.action = self.ACTION_TYPES[action_type](
cfg, local_addrs, debug)
def __str__(self):
cfg = []
for key in ["name", "loglevel", "pretend", "type"]:
value = self.cfg[key]
cfg.append(f"{key}={value}")
if self.conditions is not None:
cfg.append(f"conditions={self.conditions}")
cfg.append(f"action={self.action}")
return "Action(" + ", ".join(cfg) + ")"
def headersonly(self):
"""Return the needs of this action."""
return self.action._headersonly
def execute(self, milter):
"""Execute configured action."""
if self.conditions is None or \
self.conditions.match(milter):
return self.action.execute(milter)

71
pyquarantine/base.py Normal file
View File

@@ -0,0 +1,71 @@
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = [
"CustomLogger",
"MilterMessage",
"replace_illegal_chars"]
import logging
from email.message import MIMEPart
class CustomLogger(logging.LoggerAdapter):
def process(self, msg, kwargs):
if "name" in self.extra:
msg = f"{self.extra['name']}: {msg}"
if "qid" in self.extra:
msg = f"{self.extra['qid']}: {msg}"
if self.logger.getEffectiveLevel() != logging.DEBUG:
msg = msg.replace("\n", "").replace("\r", "")
return msg, kwargs
class MilterMessage(MIMEPart):
def replace_header(self, _name, _value, idx=None):
_name = _name.lower()
counter = 0
for i, (k, v) in zip(range(len(self._headers)), self._headers):
if k.lower() == _name:
counter += 1
if not idx or counter == idx:
self._headers[i] = self.policy.header_store_parse(
k, _value)
break
else:
raise KeyError(_name)
def remove_header(self, name, idx=None):
name = name.lower()
newheaders = []
counter = 0
for k, v in self._headers:
if k.lower() == name:
counter += 1
if counter != idx:
newheaders.append((k, v))
else:
newheaders.append((k, v))
self._headers = newheaders
def replace_illegal_chars(string):
"""Remove illegal characters from header values."""
return "".join(string.replace("\x00", "").splitlines())

40
pyquarantine/cli.py
View File

@@ -20,7 +20,7 @@ import logging.handlers
import sys
import time
from pyquarantine import QuarantineMilter, setup_milter
from pyquarantine.config import get_milter_config
from pyquarantine import __version__ as version
@@ -304,6 +304,9 @@ class StdOutFilter(logging.Filter):
def main():
python_version = ".".join([str(v) for v in sys.version_info[0:3]])
python_version = f"{python_version}-{sys.version_info[3]}"
"PyQuarantine command-line interface."
# parse command line
def formatter_class(prog): return argparse.HelpFormatter(
@@ -312,10 +315,8 @@ def main():
description="PyQuarantine CLI",
formatter_class=formatter_class)
parser.add_argument(
"-c", "--config",
help="Config files to read.",
nargs="+", metavar="CFG",
default=QuarantineMilter.get_cfg_files())
"-c", "--config", help="Config file to read.",
default="/etc/pyquarantine/pyquarantine.conf")
parser.add_argument(
"-d", "--debug",
help="Log debugging messages.",
@@ -324,7 +325,7 @@ def main():
"-v", "--version",
help="Print version.",
action="version",
version=f"%(prog)s ({version})")
version=f"%(prog)s {version} (python {python_version})")
parser.set_defaults(syslog=False)
subparsers = parser.add_subparsers(
dest="command",
@@ -578,14 +579,29 @@ def main():
root_logger.addHandler(stderrhandler)
logger = logging.getLogger(__name__)
# try to generate milter configs
try:
setup_milter(
cfg_files=args.config, test=True)
except RuntimeError as e:
logger.error(e)
logger.debug("read milter configuration")
cfg = get_milter_config(args.config)
if not cfg["rules"]:
raise RuntimeError("no rules configured")
for rule in cfg["rules"]:
if not rule["actions"]:
raise RuntimeError(
f"{rule['name']}: no actions configured")
except (RuntimeError, AssertionError) as e:
logger.error(f"config error: {e}")
sys.exit(255)
quarantines = []
for rule in cfg["rules"]:
for action in rule["actions"]:
if action["type"] == "quarantine":
quarantines.append(action)
print(quarantines)
sys.exit(0)
if args.syslog:
# setup syslog
sysloghandler = logging.handlers.SysLogHandler(
@@ -602,7 +618,7 @@ def main():
# call the commands function
try:
args.func(QuarantineMilter.quarantines, args)
args.func(cfg, args)
except RuntimeError as e:
logger.error(e)
sys.exit(1)

193
pyquarantine/conditions.py Normal file
View File

@@ -0,0 +1,193 @@
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = ["Conditions"]
import logging
import re
from netaddr import IPAddress, IPNetwork, AddrFormatError
from pyquarantine import CustomLogger
from pyquarantine.whitelist import DatabaseWhitelist
class Conditions:
"""Conditions to implement conditions for rules and actions."""
def __init__(self, cfg, local_addrs, debug):
self.cfg = cfg
self.local_addrs = local_addrs
self.logger = logging.getLogger(cfg["name"])
self.logger.setLevel(cfg.get_loglevel(debug))
for arg in ("local", "hosts", "envfrom", "envto", "header", "metavar",
"var"):
if arg not in cfg:
setattr(self, arg, None)
continue
if arg == "hosts":
try:
self.hosts = []
for host in cfg["hosts"]:
self.hosts.append(IPNetwork(host))
except AddrFormatError as e:
raise RuntimeError(e)
elif arg in ("envfrom", "envto"):
try:
setattr(self, arg, re.compile(
cfg[arg], re.IGNORECASE))
except re.error as e:
raise RuntimeError(e)
elif arg == "header":
try:
self.header = re.compile(
cfg["header"],
re.IGNORECASE + re.DOTALL + re.MULTILINE)
except re.error as e:
raise RuntimeError(e)
else:
setattr(self, arg, cfg[arg])
self.whitelist = cfg["whitelist"] if "whitelist" in cfg else None
if self.whitelist is not None:
self.whitelist["name"] = f"{cfg['name']}: whitelist"
self.whitelist["loglevel"] = cfg["loglevel"]
if self.whitelist["type"] == "db":
self.whitelist = DatabaseWhitelist(self.whitelist, debug)
else:
raise RuntimeError("invalid whitelist type")
def __str__(self):
cfg = []
for arg in ("local", "hosts", "envfrom", "envto", "header",
"var", "metavar"):
if arg in self.cfg:
cfg.append(f"{arg}={self.cfg[arg]}")
if self.whitelist is not None:
cfg.append(f"whitelist={self.whitelist}")
return "Conditions(" + ", ".join(cfg) + ")"
def match_host(self, host):
logger = CustomLogger(
self.logger, {"name": self.cfg["name"]})
ip = IPAddress(host)
if self.local is not None:
is_local = False
for addr in self.local_addrs:
if ip in addr:
is_local = True
break
if is_local != self.local:
logger.debug(
f"ignore host {host}, "
f"local does not match")
return False
logger.debug(
f"local matches for host {host}")
if self.hosts is not None:
found = False
for addr in self.hosts:
if ip in addr:
found = True
break
if not found:
logger.debug(
f"ignore host {host}, "
f"hosts does not match")
return False
logger.debug(
f"hosts matches for host {host}")
return True
def get_wl_rcpts(self, mailfrom, rcpts, logger):
if not self.whitelist:
return {}
wl_rcpts = []
for rcpt in rcpts:
if self.whitelist.check(mailfrom, rcpt, logger):
wl_rcpts.append(rcpt)
return wl_rcpts
def match(self, milter):
logger = CustomLogger(
self.logger, {"qid": milter.qid, "name": self.cfg["name"]})
if self.envfrom is not None:
envfrom = milter.msginfo["mailfrom"]
if not self.envfrom.match(envfrom):
logger.debug(
f"ignore envelope-from address {envfrom}, "
f"envfrom does not match")
return False
logger.debug(
f"envfrom matches for "
f"envelope-from address {envfrom}")
if self.envto is not None:
envto = milter.msginfo["rcpts"]
if not isinstance(envto, list):
envto = [envto]
for to in envto:
if not self.envto.match(to):
logger.debug(
f"ignore envelope-to address {envto}, "
f"envto does not match")
return False
logger.debug(
f"envto matches for "
f"envelope-to address {envto}")
if self.header is not None:
match = None
for field, value in milter.msg.items():
header = f"{field}: {value}"
match = self.header.search(header)
if match:
logger.debug(
f"header matches for "
f"header: {header}")
if self.metavar is not None:
named_subgroups = match.groupdict(default=None)
for group, value in named_subgroups.items():
if value is None:
continue
name = f"{self.metavar}_{group}"
milter.msginfo["vars"][name] = value
break
if not match:
logger.debug(
"ignore message, "
"header does not match")
return False
if self.var is not None:
if self.var not in milter.msginfo["vars"]:
return False
return True

354
pyquarantine/config.py Normal file
View File

@@ -0,0 +1,354 @@
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = [
"BaseConfig",
"ConditionsConfig",
"AddHeaderConfig",
"ModHeaderConfig",
"DelHeaderConfig",
"AddDisclaimerConfig",
"RewriteLinksConfig",
"StoreConfig",
"NotifyConfig",
"WhitelistConfig",
"QuarantineConfig",
"ActionConfig",
"RuleConfig",
"MilterConfig",
"get_milter_config"]
import json
import jsonschema
import logging
import re
class BaseConfig:
JSON_SCHEMA = {
"type": "object",
"required": [],
"additionalProperties": True,
"properties": {
"loglevel": {"type": "string", "default": "info"}}}
def __init__(self, config):
required = self.JSON_SCHEMA["required"]
properties = self.JSON_SCHEMA["properties"]
for p in properties.keys():
if p in required:
continue
elif p not in config and "default" in properties[p]:
config[p] = properties[p]["default"]
try:
jsonschema.validate(config, self.JSON_SCHEMA)
except jsonschema.exceptions.ValidationError as e:
raise RuntimeError(e)
self._config = config
def __getitem__(self, key):
return self._config[key]
def __setitem__(self, key, value):
self._config[key] = value
def __delitem__(self, key):
del self._config[key]
def __contains__(self, key):
return key in self._config
def keys(self):
return self._config.keys()
def items(self):
return self._config.items()
def get_loglevel(self, debug):
if debug:
level = logging.DEBUG
else:
level = getattr(logging, self["loglevel"].upper(), None)
assert isinstance(level, int), \
"loglevel: invalid value"
return level
def get_config(self):
return self._config
class WhitelistConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": ["type"],
"additionalProperties": True,
"properties": {
"type": {"enum": ["db"]}},
"if": {"properties": {"type": {"const": "db"}}},
"then": {
"required": ["connection", "table"],
"additionalProperties": False,
"properties": {
"type": {"type": "string"},
"connection": {"type": "string"},
"table": {"type": "string"}}}}
class ConditionsConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": [],
"additionalProperties": False,
"properties": {
"metavar": {"type": "string"},
"local": {"type": "boolean"},
"hosts": {"type": "array",
"items": {"type": "string"}},
"envfrom": {"type": "string"},
"envto": {"type": "string"},
"header": {"type": "string"},
"var": {"type": "string"},
"whitelist": {"type": "object"}}}
def __init__(self, config, rec=True):
super().__init__(config)
if rec:
if "whitelist" in self:
self["whitelist"] = WhitelistConfig(self["whitelist"])
class AddHeaderConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": ["field", "value"],
"additionalProperties": False,
"properties": {
"field": {"type": "string"},
"value": {"type": "string"}}}
class ModHeaderConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": ["field", "value"],
"additionalProperties": False,
"properties": {
"field": {"type": "string"},
"value": {"type": "string"},
"search": {"type": "string"}}}
class DelHeaderConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": ["field"],
"additionalProperties": False,
"properties": {
"field": {"type": "string"},
"value": {"type": "string"}}}
class AddDisclaimerConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": ["action", "html_template", "text_template"],
"additionalProperties": False,
"properties": {
"action": {"type": "string"},
"html_template": {"type": "string"},
"text_template": {"type": "string"},
"error_policy": {"type": "string"}}}
class RewriteLinksConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": ["repl"],
"additionalProperties": False,
"properties": {
"repl": {"type": "string"}}}
class StoreConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": ["type"],
"additionalProperties": True,
"properties": {
"type": {"enum": ["file"]}},
"if": {"properties": {"type": {"const": "file"}}},
"then": {
"required": ["directory"],
"additionalProperties": False,
"properties": {
"type": {"type": "string"},
"directory": {"type": "string"},
"mode": {"type": "string"},
"metavar": {"type": "string"},
"original": {"type": "boolean", "default": True}}}}
class NotifyConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": ["type"],
"additionalProperties": True,
"properties": {
"type": {"enum": ["email"]}},
"if": {"properties": {"type": {"const": "email"}}},
"then": {
"required": ["smtp_host", "smtp_port", "envelope_from",
"from_header", "subject", "template"],
"additionalProperties": False,
"properties": {
"type": {"type": "string"},
"smtp_host": {"type": "string"},
"smtp_port": {"type": "number"},
"envelope_from": {"type": "string"},
"from_header": {"type": "string"},
"subject": {"type": "string"},
"template": {"type": "string"},
"repl_img": {"type": "string"},
"embed_imgs": {
"type": "array",
"items": {"type": "string"},
"default": True}}}}
class QuarantineConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": ["store"],
"additionalProperties": False,
"properties": {
"name": {"type": "string"},
"notify": {"type": "object"},
"milter_action": {"type": "string"},
"reject_reason": {"type": "string"},
"whitelist": {"type": "object"},
"store": {"type": "object"}}}
def __init__(self, config, rec=True):
super().__init__(config)
if rec:
self["store"] = StoreConfig(self["store"])
if "notify" in self:
self["notify"] = NotifyConfig(self["notify"])
if "whitelist" in self:
self["whitelist"] = ConditionsConfig(
{"whitelist": self["whitelist"]}, rec)
class ActionConfig(BaseConfig):
ACTION_TYPES = {
"add_header": AddHeaderConfig,
"mod_header": ModHeaderConfig,
"del_header": DelHeaderConfig,
"add_disclaimer": AddDisclaimerConfig,
"rewrite_links": RewriteLinksConfig,
"store": StoreConfig,
"notify": NotifyConfig,
"quarantine": QuarantineConfig}
JSON_SCHEMA = {
"type": "object",
"required": ["type", "args"],
"additionalProperties": False,
"properties": {
"name": {"type": "string", "default": "action"},
"loglevel": {"type": "string", "default": "info"},
"pretend": {"type": "boolean", "default": False},
"conditions": {"type": "object"},
"type": {"enum": list(ACTION_TYPES.keys())},
"args": {"type": "object"}}}
def __init__(self, config, rec=True):
super().__init__(config)
if rec:
if "conditions" in self:
self["conditions"] = ConditionsConfig(self["conditions"])
self["action"] = self.ACTION_TYPES[self["type"]](self["args"])
class RuleConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": ["actions"],
"additionalProperties": False,
"properties": {
"name": {"type": "string", "default": "rule"},
"loglevel": {"type": "string", "default": "info"},
"pretend": {"type": "boolean", "default": False},
"conditions": {"type": "object"},
"actions": {"type": "array"}}}
def __init__(self, config, rec=True):
super().__init__(config)
if rec:
if "conditions" in self:
self["conditions"] = ConditionsConfig(self["conditions"])
actions = []
for idx, action in enumerate(self["actions"]):
actions.append(ActionConfig(action, rec))
self["actions"] = actions
class MilterConfig(BaseConfig):
JSON_SCHEMA = {
"type": "object",
"required": ["rules"],
"additionalProperties": False,
"properties": {
"socket": {"type": "string"},
"local_addrs": {"type": "array",
"items": {"type": "string"},
"default": [
"fe80::/64",
"::1/128",
"127.0.0.0/8",
"10.0.0.0/8",
"172.16.0.0/12",
"192.168.0.0/16"]},
"loglevel": {"type": "string", "default": "info"},
"pretend": {"type": "boolean", "default": False},
"rules": {"type": "array"}}}
def __init__(self, config, rec=True):
super().__init__(config)
if rec:
rules = []
for idx, rule in enumerate(self["rules"]):
rules.append(RuleConfig(rule, rec))
self["rules"] = rules
def get_milter_config(cfgfile):
try:
with open(cfgfile, "r") as fh:
# remove lines with leading # (comments), they
# are not allowed in json
cfg = re.sub(r"(?m)^\s*#.*\n?", "", fh.read())
except IOError as e:
raise RuntimeError(f"unable to open/read config file: {e}")
try:
cfg = json.loads(cfg)
except json.JSONDecodeError as e:
cfg_text = [f"{n+1}: {l}" for n, l in enumerate(cfg.splitlines())]
msg = "\n".join(cfg_text)
raise RuntimeError(f"{e}\n{msg}")
return MilterConfig(cfg)

337
pyquarantine/docs/pyquarantine.conf.example
View File

@@ -1,180 +1,221 @@
# This is an example /etc/pyquarantine/pyquarantine.conf file.
# Copy it into place before use.
#
# Comments: use '#' for comment lines and ';' (following a space) for inline comments.
# The file is in JSON format.
#
# If an option is not present in a quarantine section, it will be read from
# the global section.
# The global option 'log' can be overriden per rule or per modification.
#
{
# Section: global
# Notes: Global options.
#
"global": {
# Option: socket
# Type: String
# Notes: The socket used to communicate with the MTA.
#
# Examples:
# unix:/path/to/socket a named pipe
# inet:8899 listen on ANY interface
# inet:8899@localhost listen on a specific interface
# inet6:8899 listen on ANY interface
# inet6:8899@[2001:db8:1234::1] listen on a specific interface
# Value: [ SOCKET ]
"socket": "inet:8898@127.0.0.1",
# Option: local_addrs
# Type: List
# Notes: A list of local hosts and networks.
# Value: [ LIST ]
#
"local_addrs": ["fe80::/64", "::1/128", "127.0.0.0/8", "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16"],
[global]
# Option: loglevel
# Type: String
# Notes: Set loglevel for rules and actions.
# Value: [ error | warning | info | debug ]
#
"loglevel": "info",
# Option: quarantines
# Notes: Set active quarantines (comma-separated).
# Each active quarantine must have a section with the same name below.
# The quarantine name 'global' is forbidden and will be ignored.
# Quarantine names must be unique.
# Values: [ ACTIVE ]
#
quarantines = spam
# Option: pretend
# Type: Bool
# Notes: Just pretend to do the actions, for test purposes.
# Value: [ true | false ]
#
"pretend": true
},
# Option: quarantine_action_precedence
# Notes: Set if the action of the first or the last matching quarantine should
# be used if multiple recipients match multiple quarantines. If an original
# email is delivered to at least one recipient due to whitelisting, the
# email will always be accepted.
# Values: [ first | last ]
#
preferred_quarantine_action = last
# Section: rules
# Notes: Rules and related actions.
#
"rules": [
{
# Option: name
# Type: String
# Notes: Name of the rule.
# Value: [ NAME ]
#
"name": "myrule",
# Section: conditions
# Notes: Optional conditions to process the rule.
# If multiple conditions are set, they all
# have to be true to process the rule.
#
"conditions": {
# Option: local
# Type: Bool
# Notes: Condition wheter the senders host address is listed in local_addrs.
# Value: [ true | false ]
#
"local": false,
[spam]
# Option: host_whitelist
# Notes: Set a list of host and network addresses to be ignored by this quarantine.
# All the common host/network notations are supported, including IPv6.
# Value: [ HOST ]
#
host_whitelist = 10.0.0.0/8,172.16.0.0/12,192.168.0.0/16
# Option: hosts
# Type: String
# Notes: Condition wheter the senders host address is listed in this list.
# Value: [ LIST ]
#
"hosts": [ "127.0.0.1" ],
# Option: regex
# Notes: Set the case insensitive regular expression to match against email headers.
# If the regex matches any of the email headers, the email
# will be processed by this quarantine.
# Values: [ REGEX ]
#
regex = ^X-Spam-Flag: YES
# Option: envfrom
# Type: String
# Notes: Condition wheter the envelop-from address matches this regular expression.
# Value: [ REGEX ]
#
"envfrom": "^.+@mypartner\\.com$",
# Option: smtp_host
# Notes: Set the SMTP host. It will be used to (re-)inject emails.
# Values: [ HOSTNAME | IP_ADDRESS ]
#
smtp_host = 127.0.0.1
# Option: envto
# Type: String
# Notes: Condition wheter the envelop-to address matches this regular expression.
# Value: [ REGEX ]
#
"envto": "^postmaster@.+$"
},
# Option: smtp_port
# Notes: Set the SMTP port.
# Values: [ PORT ]
#
smtp_port = 25
# Section: actions
# Notes: Actions of the rule.
#
"actions": [
{
# Option: name
# Type: String
# Notes: Name of the modification.
# Value: [ NAME ]
#
"name": "add_test_header",
# Option: storage_type
# Notes: Set the storage type.
# Values: [ file | none ]
#
storage_type = file
# Option: type
# Type: String
# Notes: Type of the modification.
# Value: [ add_header | del_header | mod_header ]
#
"type": "add_header",
# Option: storage_directory
# Notes: Set the directory to store quarantined emails.
# This option is needed by quarantine type 'file'.
# Values: [ DIRECTORY ]
#
storage_directory = /var/lib/pyquarantine/spam
# Option: field
# Type: String
# Notes: Name of the header.
# Value: [ NAME ]
#
"field": "X-Test-Header",
# Option: action
# Notes: Set the milter action to perform if email is processed by this quarantine.
# Values: [ accept | discard | reject ]
#
action = discard
# Option: value
# Type: String
# Notes: Value of the header.
# Value: [ VALUE ]
#
"value": "true"
}, {
"name": "modify_subject",
# Option: reject_reason
# Notes: Optionally set the reason to return if action is set to reject.
# Values: [ REASON ]
#
reject_reason = Message rejected
"type": "mod_header",
# Option: notification
# Notes: Set the notification type.
# Values: [ email | none ]
#
notification_type = email
# Option: field
# Type: String
# Notes: Regular expression to match against header lines (e.g. Subject: Test-Subject).
# Value: [ REGEX ]
#
"field": "^Subject$",
# Option: notification_email_smtp_host
# Notes: Set the SMTP host. It will be used to send notification e-mails.
# Values: [ HOSTNAME | IP_ADDRESS ]
#
notification_email_smtp_host = 127.0.0.1
# Option: search
# Type: String
# Notes: Regular expression to match against the headers value.
# Values: [ VALUE ]
#
"search": "(?P<subject>.*)",
# Option: notification_email_smtp_port
# Notes: Set the SMTP port.
# Values: [ PORT ]
#
notification_email_smtp_port = 25
# Option: value
# Type: String
# Notes: New value of the header.
# Values: [ VALUE ]
"value": "[EXTERNAL] \\g<subject>"
}, {
"name": "delete_received_header",
# Option: notification_email_envelope_from
# Notes: Set the envelope-from address used when sending notification emails.
# This option is needed by notification type 'email'.
# Values: [ ENVELOPE_FROM_ADDRESS ]
#
notification_email_envelope_from = notification@domain.tld
"type": "del_header",
# Option: notification_email_from
# Notes: Set the from header used when sending notification emails.
# This option is needed by notification type 'email'.
# Values: [ FROM_HEADER ]
#
notification_email_from = Notification <notification@domain.tld>
# Option: field
# Type: String
# Notes: Regular expression to match against header lines (e.g. Subject: Test-Subject).
# Value: [ REGEX ]
#
"field": "^Received$"
}, {
"name": "add_disclaimer",
# Option: notification_email_usbject
# Notes: Set the subject used when sending notification emails.
# This option is needed by notification type 'email'.
# Values: [ SUBJECT ]
#
notification_email_subject = Spam Quarantine Notification
"type": "add_disclaimer",
# Option: notification_email_template
# Notes: Set the template used when sending notification emails.
# A relative path to this config file can be used.
# This option is needed by notification type 'email'.
# Values: [ TEMPLATE_PATH ]
#
notification_email_template = templates/notification.template
# Option: action
# Type: String
# Notes: Action to perform with the disclaimer.
# Value: [ append | prepend ]
#
"action": "prepend",
# Option: notification_email_strip_images
# Notes: Optionally enable this option to strip img tags from emails.
# Values: [ TRUE | ON | YES | FALSE | OFF | NO ]
#
notification_email_strip_images = False
# Option: html_template
# Type: String
# Notes: Path to a file which contains the html representation of the disclaimer.
# Value: [ FILE_PATH ]
#
"html_template": "/etc/pyquarantine/templates/disclaimer_html.template",
# Option: notification_email_replacement_img
# Notes: Optionally set the path to a replacement image for img tags within emails.
# A relative path to this config file can be used.
# Values: [ IMAGE_PATH ]
#
notification_email_replacement_img = templates/removed.png
# Option: text_template
# Type: String
# Notes: Path to a file which contains the text representation of the disclaimer.
# Value: [ FILE_PATH ]
#
"text_template": "/etc/pyquarantine/templates/disclaimer_text.template",
# Option: notification_email_embedded_imgs
# Notes: Set a list of paths to images to embed in e-mails (comma-separated).
# Relative paths to this config file can be used.
# This option is needed by notification type 'email'.
# Values: [ IMAGE_PATH ]
#
notification_email_embedded_imgs = templates/logo.png
# Option: error_policy
# Type: String
# Notes: Set what should be done if the modification fails (e.g. no message body present).
# Value: [ wrap | ignore | reject ]
#
"error_policy": "wrap"
}, {
"name": "store_message",
# Option: notification_email_parser_lib
# Notes: Optionally set the parser library used to parse
# the text part of emails.
# Values: [ lxml | html.parser ]
#
notification_email_parser_lib = lxml
"type": "store",
# Option: whitelist_type
# Notes: Set the whitelist type.
# Values: [ db | none ]
#
whitelist_type = db
# Option: storage_type
# Type: String
# Notes: The storage type used to store e-mails.
# Value: [ file ]
"storage_type": "file",
# Option: whitelist_db_connection
# Notes: Set the connection string to connect to the database.
# The configured user must have read/write access to
# the whitelist_db_table configured below.
# This option is needed by whitelist type 'db'.
# Values: [ DB_CONNECTION_STRING | none ]
#
whitelist_db_connection = mysql://user:password@localhost/database
# Option: directory
# Type: String
# Notes: Directory used to store e-mails.
# Value: [ file ]
"directory": "/mnt/messages",
# Option: whitelist_db_table
# Notes: Set the database table name.
# This option is needed by whitelist type 'db'.
# Values: [ DATABASE_TABLE]
#
whitelist_db_table = whitelist
# Option: original
# Type: Bool
# Notes: If set to true, store the message as received by the MTA instead of storing the current state
# of the message, that may was modified already by other actions.
# Value: [ true | false ]
"original": true
}
]
}
]
}

9
pyquarantine/docs/templates/disclaimer_html.template vendored Normal file
View File

@@ -0,0 +1,9 @@
<table style="border: 1px solid; background-color: #F8E898; border-color: #885800; font-family: Arial; font-size: 11pt;">
<tr>
<td>
<span style="font-weight: bold; color: #905800;">CAUTION:</span> This email originated from outside the organization.
Do not follow guidance, click links or open attachments unless you recognize the sender and know the content is safe.
</td>
</tr>
</table>
<br/><br/>

4
pyquarantine/docs/templates/disclaimer_text.template vendored Normal file
View File

@@ -0,0 +1,4 @@
CAUTION: This email originated from outside the organization. Do not follow guidance, click links or open attachments unless you recognize the sender and know the content is safe.

16
pyquarantine/docs/templates/notification.template vendored
View File

@@ -4,30 +4,26 @@
<table>
<tr>
<td><b>Envelope-From:</b></td>
<td>{EMAIL_ENVELOPE_FROM}</td>
<td>{ENVELOPE_FROM}</td>
</tr>
<tr>
<td><b>From:</b></td>
<td>{EMAIL_FROM}</td>
<td>{FROM}</td>
</tr>
<tr>
<td><b>Envelope-To:</b></td>
<td>{EMAIL_ENVELOPE_TO}</td>
<td>{ENVELOPE_TO}</td>
</tr>
<tr>
<td><b>To:</b></td>
<td>{EMAIL_TO}</td>
<td>{TO}</td>
</tr>
<tr>
<td><b>Subject:</b></td>
<td>{EMAIL_SUBJECT}</td>
</tr>
<tr>
<td><b>Quarantine ID:</b></td>
<td>{EMAIL_QUARANTINE_ID}</td>
<td>{SUBJECT}</td>
</tr>
</table><br/>
<h2>Preview of the original e-mail</h2>
{EMAIL_HTML_TEXT}
{HTML_TEXT}
</body>
</html>

6
pyquarantine/mailer.py
View File

@@ -1,15 +1,15 @@
# PyQuarantine-Milter is free software: you can redistribute it and/or modify
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# PyQuarantine-Milter is distributed in the hope that it will be useful,
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with PyQuarantineMilter. If not, see <http://www.gnu.org/licenses/>.
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
import logging

13
pyquarantine/misc/openrc/pyquarantine-milter.confd
View File

@@ -1,17 +1,8 @@
# /etc/conf.d/pyquarantine-milter: config file for /etc/init.d/pyquarantine-milter
# Set the socket used to communicate with the MTA.
# Examples:
# unix:/path/to/socket a named pipe
# inet:8899 listen on ANY interface
# inet:8899@localhost listen on a specific interface
# inet6:8899 listen on ANY interface
# inet6:8899@[2001:db8:1234::1] listen on a specific interface
SOCKET="inet:8899@127.0.0.1"
# /etc/conf.d/pyquarantine: config file for /etc/init.d/pyquarantine
# Start the daemon as the user. You can optionally append a group name here also.
# USER="daemon"
# USER="daemon:nobody"
# Optional parameters for pyquarantine-milter
# Optional parameters for pyquarantine
# MILTER_OPTS=""

12
pyquarantine/misc/openrc/pyquarantine-milter.initd
View File

@@ -1,16 +1,15 @@
#!/sbin/openrc-run
user=${USER:-daemon}
socket="${SOCKET:-}"
milter_opts="${MILTER_OPTS:-}"
pidfile="/run/${RC_SVCNAME}.pid"
command="/usr/bin/pyquarantine-milter"
command_args="-s ${socket} ${milter_opts}"
command_args="${milter_opts}"
command_background=true
start_stop_daemon_args="--user ${user}"
command_user="${user}"
extra_commands="configtest"
start_stop_daemon_args="--wait 500"
depend() {
need net
@@ -18,9 +17,6 @@ depend() {
}
checkconfig() {
if [ -z "${socket}" ]; then
eerror "No socket specified in config!"
fi
OUTPUT=$( ${command} ${command_args} -t 2>&1 )
ret=$?
if [ $ret -ne 0 ]; then
@@ -44,7 +40,7 @@ start_pre() {
}
stop_pre() {
if [ "${RC_CMD}" != "restart" ]; then
if [ "${RC_CMD}" == "restart" ]; then
checkconfig || return $?
fi
}

391
pyquarantine/modify.py Normal file
View File

@@ -0,0 +1,391 @@
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = [
"AddHeader",
"ModHeader",
"DelHeader",
"AddDisclaimer",
"RewriteLinks",
"Modify"]
import logging
import re
from base64 import b64encode
from bs4 import BeautifulSoup
from collections import defaultdict
from copy import copy
from email.message import MIMEPart
from email.policy import SMTPUTF8
from pyquarantine import replace_illegal_chars
from pyquarantine.base import CustomLogger
class AddHeader:
"""Add a mail header field."""
_headersonly = True
def __init__(self, field, value, pretend=False):
self.field = field
self.value = value
self.pretend = pretend
def execute(self, milter, logger):
header = f"{self.field}: {self.value}"
if logger.getEffectiveLevel() == logging.DEBUG:
logger.debug(f"add_header: {header}")
else:
logger.info(f"add_header: {header[0:70]}")
milter.msg.add_header(self.field, self.value)
if not self.pretend:
milter.addheader(self.field, self.value)
class ModHeader:
"""Change the value of a mail header field."""
_headersonly = True
def __init__(self, field, value, search=None, pretend=False):
try:
self.field = re.compile(field, re.IGNORECASE)
self.search = search
if self.search is not None:
self.search = re.compile(
self.search, re.MULTILINE + re.DOTALL + re.IGNORECASE)
except re.error as e:
raise RuntimeError(e)
self.value = value
self.pretend = pretend
def execute(self, milter, logger):
idx = defaultdict(int)
for i, (field, value) in enumerate(milter.msg.items()):
field_lower = field.lower()
idx[field_lower] += 1
if not self.field.match(field):
continue
new_value = value
if self.search is not None:
new_value = self.search.sub(self.value, value).strip()
else:
new_value = self.value
if not new_value:
logger.warning(
"mod_header: resulting value is empty, "
"skip modification")
continue
if new_value == value:
continue
header = f"{field}: {value}"
new_header = f"{field}: {new_value}"
if logger.getEffectiveLevel() == logging.DEBUG:
logger.debug(f"mod_header: {header}: {new_header}")
else:
logger.info(
f"mod_header: {header[0:70]}: {new_header[0:70]}")
milter.msg.replace_header(
field, replace_illegal_chars(new_value), idx=idx[field_lower])
if not self.pretend:
milter.chgheader(field, new_value, idx=idx[field_lower])
class DelHeader:
"""Delete a mail header field."""
_headersonly = True
def __init__(self, field, value=None, pretend=False):
try:
self.field = re.compile(field, re.IGNORECASE)
self.value = value
if self.value is not None:
self.value = re.compile(
value, re.MULTILINE + re.DOTALL + re.IGNORECASE)
except re.error as e:
raise RuntimeError(e)
self.pretend = pretend
def execute(self, milter, logger):
idx = defaultdict(int)
for field, value in milter.msg.items():
field_lower = field.lower()
idx[field_lower] += 1
if not self.field.match(field):
continue
if self.value is not None and not self.value.search(value):
continue
header = f"{field}: {value}"
if logger.getEffectiveLevel() == logging.DEBUG:
logger.debug(f"del_header: {header}")
else:
logger.info(f"del_header: {header[0:70]}")
milter.msg.remove_header(field, idx=idx[field_lower])
if not self.pretend:
milter.chgheader(field, "", idx=idx[field_lower])
idx[field_lower] -= 1
def _get_body_content(msg, pref):
part = None
content = None
if not msg.is_multipart() and msg.get_content_type() == f"text/{pref}":
part = msg
else:
part = msg.get_body(preferencelist=(pref))
if part is not None:
content = part.get_content()
return (part, content)
def _has_content_before_body_tag(soup):
s = copy(soup)
for element in s.find_all("head") + s.find_all("body"):
element.extract()
if len(s.text.strip()) > 0:
return True
return False
def _inject_body(milter):
if not milter.msg.is_multipart():
milter.msg.make_mixed()
attachments = []
for attachment in milter.msg.iter_attachments():
if "content-disposition" not in attachment:
attachment["Content-Disposition"] = "attachment"
attachments.append(attachment)
milter.msg.clear_content()
milter.msg.set_content("")
milter.msg.add_alternative("", subtype="html")
milter.msg.make_mixed()
for attachment in attachments:
milter.msg.attach(attachment)
def _wrap_message(milter):
attachment = MIMEPart(policy=SMTPUTF8)
attachment.set_content(milter.msg.as_bytes(),
maintype="plain", subtype="text",
disposition="attachment",
filename=f"{milter.qid}.eml",
params={"name": f"{milter.qid}.eml"})
milter.msg.clear_content()
milter.msg.set_content(
"Please see the original email attached.")
milter.msg.add_alternative(
"<html><body>Please see the original email attached.</body></html>",
subtype="html")
milter.msg.make_mixed()
milter.msg.attach(attachment)
class AddDisclaimer:
"""Append or prepend a disclaimer to the mail body."""
_headersonly = False
def __init__(self, text_template, html_template, action, error_policy,
pretend=False):
self.text_template_path = text_template
self.html_template_path = html_template
try:
with open(text_template, "r") as f:
self.text_template = f.read()
with open(html_template, "r") as f:
html = BeautifulSoup(f.read(), "html.parser")
except IOError as e:
raise RuntimeError(e)
body = html.find('body')
self.html_template = body or html
self.action = action
self.error_policy = error_policy
self.pretend = pretend
def patch_message_body(self, milter, logger):
text_body, text_content = _get_body_content(milter.msg, "plain")
html_body, html_content = _get_body_content(milter.msg, "html")
if text_content is None and html_content is None:
raise RuntimeError("message does not contain any body part")
if text_content is not None:
logger.info(f"{self.action} text disclaimer")
if self.action == "prepend":
content = f"{self.text_template}{text_content}"
else:
content = f"{text_content}{self.text_template}"
text_body.set_content(
content.encode(), maintype="text", subtype="plain")
text_body.set_param("charset", "UTF-8", header="Content-Type")
del text_body["MIME-Version"]
if html_content is not None:
logger.info(f"{self.action} html disclaimer")
soup = BeautifulSoup(html_content, "html.parser")
body = soup.find('body')
if not body:
body = soup
elif _has_content_before_body_tag(soup):
body = soup
if self.action == "prepend":
body.insert(0, copy(self.html_template))
else:
body.append(self.html_template)
html_body.set_content(
str(soup).encode(), maintype="text", subtype="html")
html_body.set_param("charset", "UTF-8", header="Content-Type")
del html_body["MIME-Version"]
def execute(self, milter, logger):
old_headers = milter.msg.items()
try:
try:
self.patch_message_body(milter, logger)
except RuntimeError as e:
logger.info(f"{e}, inject empty plain and html body")
_inject_body(milter)
self.patch_message_body(milter, logger)
except Exception as e:
logger.warning(e)
if self.error_policy == "ignore":
logger.info(
"unable to add disclaimer to message body, "
"ignore error according to policy")
return
elif self.error_policy == "reject":
logger.info(
"unable to add disclaimer to message body, "
"reject message according to policy")
return [
("reject", "Message rejected due to error")]
logger.info("wrap original message in a new message envelope")
try:
_wrap_message(milter)
self.patch_message_body(milter, logger)
except Exception as e:
logger.error(e)
raise Exception(
"unable to wrap message in a new message envelope, "
"give up ...")
if not self.pretend:
milter.update_headers(old_headers)
milter.replacebody()
class RewriteLinks:
"""Rewrite link targets in the mail html body."""
_headersonly = False
def __init__(self, repl, pretend=False):
self.repl = repl
self.pretend = pretend
def execute(self, milter, logger):
html_body, html_content = _get_body_content(milter.msg, "html")
if html_content is not None:
soup = BeautifulSoup(html_content, "html.parser")
rewritten = 0
for link in soup.find_all("a", href=True):
if not link["href"]:
continue
if "{URL_B64}" in self.repl:
url_b64 = b64encode(link["href"].encode()).decode()
target = self.repl.replace("{URL_B64}", url_b64)
else:
target = self.repl
link["href"] = target
rewritten += 1
if rewritten:
logger.info(f"rewrote {rewritten} link(s) in html body")
html_body.set_content(
str(soup).encode(), maintype="text", subtype="html")
html_body.set_param("charset", "UTF-8", header="Content-Type")
del html_body["MIME-Version"]
if not self.pretend:
milter.replacebody()
class Modify:
MODIFICATION_TYPES = {
"add_header": AddHeader,
"mod_header": ModHeader,
"del_header": DelHeader,
"add_disclaimer": AddDisclaimer,
"rewrite_links": RewriteLinks}
def __init__(self, cfg, local_addrs, debug):
self.cfg = cfg
self.logger = logging.getLogger(cfg["name"])
self.logger.setLevel(cfg.get_loglevel(debug))
cfg["args"]["pretend"] = cfg["pretend"]
self._modification = self.MODIFICATION_TYPES[cfg["type"]](
**cfg["args"])
self._headersonly = self._modification._headersonly
def __str__(self):
cfg = []
for key, value in self.cfg["args"].items():
cfg.append(f"{key}={value}")
class_name = type(self._modification).__name__
return f"{class_name}(" + ", ".join(cfg) + ")"
def execute(self, milter):
logger = CustomLogger(
self.logger, {"name": self.cfg["name"], "qid": milter.qid})
self._modification.execute(milter, logger)

354
pyquarantine/notify.py
View File

@@ -1,24 +1,28 @@
# PyQuarantine-Milter is free software: you can redistribute it and/or modify
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# PyQuarantine-Milter is distributed in the hope that it will be useful,
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with PyQuarantineMilter. If not, see <http://www.gnu.org/licenses/>.
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = [
"BaseNotification",
"EMailNotification",
"Notify"]
import email
import logging
import re
from bs4 import BeautifulSoup
from collections import defaultdict
from email.policy import default as default_policy
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from email.mime.image import MIMEImage
@@ -26,26 +30,24 @@ from html import escape
from os.path import basename
from urllib.parse import quote
from pyquarantine.base import CustomLogger
from pyquarantine import mailer
class BaseNotification(object):
class BaseNotification:
"Notification base class"
notification_type = "base"
_headersonly = True
def __init__(self, name, global_cfg, cfg, test=False):
self.name = name
self.logger = logging.getLogger(__name__)
def __init__(self, pretend=False):
self.pretend = pretend
def notify(self, qid, storage_id, mailfrom, recipients, headers,
fp, subgroups=None, named_subgroups=None, synchronous=False):
fp.seek(0)
pass
def execute(self, milter, logger):
return
class EMailNotification(BaseNotification):
"Notification class to send notifications via mail."
notification_type = "email"
_headersonly = False
_bad_tags = [
"applet",
"embed",
@@ -109,122 +111,48 @@ class EMailNotification(BaseNotification):
"width"
]
def __init__(self, name, global_cfg, cfg, test=False):
super(EMailNotification, self).__init__(
name, global_cfg, cfg, test)
defaults = {
"notification_email_replacement_img": "",
"notification_email_strip_images": "false",
"notification_email_parser_lib": "lxml"
}
# check config
for opt in [
"notification_email_smtp_host",
"notification_email_smtp_port",
"notification_email_envelope_from",
"notification_email_from",
"notification_email_subject",
"notification_email_template",
"notification_email_embedded_imgs"] + list(defaults.keys()):
if opt in cfg:
continue
if opt in global_cfg:
cfg[opt] = global_cfg[opt]
elif opt in defaults:
cfg[opt] = defaults[opt]
else:
raise RuntimeError(
f"mandatory option '{opt}' not present in config "
f"section '{self.name}' or 'global'")
self.smtp_host = cfg["notification_email_smtp_host"]
self.smtp_port = cfg["notification_email_smtp_port"]
self.mailfrom = cfg["notification_email_envelope_from"]
self.from_header = cfg["notification_email_from"]
self.subject = cfg["notification_email_subject"]
testvars = defaultdict(str, test="TEST")
# test-parse from header
try:
self.from_header.format_map(testvars)
except ValueError as e:
raise RuntimeError(
f"error parsing notification_email_from: {e}")
# test-parse subject
try:
self.subject.format_map(testvars)
except ValueError as e:
raise RuntimeError(
f"error parsing notification_email_subject: {e}")
# read and parse email notification template
try:
self.template = open(
cfg["notification_email_template"], "r").read()
self.template.format_map(testvars)
except IOError as e:
raise RuntimeError(f"error reading template: {e}")
except ValueError as e:
raise RuntimeError(f"error parsing template: {e}")
strip_images = cfg["notification_email_strip_images"].strip().upper()
if strip_images in ["TRUE", "ON", "YES"]:
self.strip_images = True
elif strip_images in ["FALSE", "OFF", "NO"]:
self.strip_images = False
else:
raise RuntimeError(
"error parsing notification_email_strip_images: unknown value")
self.parser_lib = cfg["notification_email_parser_lib"].strip()
if self.parser_lib not in ["lxml", "html.parser"]:
raise RuntimeError(
"error parsing notification_email_parser_lib: unknown value")
# read email replacement image if specified
replacement_img = cfg["notification_email_replacement_img"].strip()
if not self.strip_images and replacement_img:
try:
self.replacement_img = MIMEImage(
open(replacement_img, "rb").read())
except IOError as e:
raise RuntimeError(
f"error reading replacement image: {e}")
else:
self.replacement_img.add_header(
"Content-ID", "<removed_for_security_reasons>")
else:
self.replacement_img = None
# read images to embed if specified
embedded_img_paths = [
p.strip() for p in cfg["notification_email_embedded_imgs"].split(
",") if p]
self.embedded_imgs = []
for img_path in embedded_img_paths:
# read image
def __init__(self, smtp_host, smtp_port, envelope_from, from_header,
subject, template, embed_imgs=[], repl_img=None,
strip_imgs=False, parser_lib="lxml", pretend=False):
super().__init__(pretend)
self.smtp_host = smtp_host
self.smtp_port = smtp_port
self.mailfrom = envelope_from
self.from_header = from_header
self.subject = subject
try:
self.template = open(template, "r").read()
self.embed_imgs = []
for img_path in embed_imgs:
img = MIMEImage(open(img_path, "rb").read())
except IOError as e:
raise RuntimeError(f"error reading image: {e}")
else:
filename = basename(img_path)
img.add_header("Content-ID", f"<{filename}>")
self.embedded_imgs.append(img)
self.embed_imgs.append(img)
def get_email_body_soup(self, qid, msg):
self.replacement_img = repl_img
self.strip_images = strip_imgs
if not strip_imgs and repl_img:
self.replacement_img = MIMEImage(
open(repl_img, "rb").read())
self.replacement_img.add_header(
"Content-ID", "<removed_for_security_reasons>")
except IOError as e:
raise RuntimeError(e)
self.parser_lib = parser_lib
def get_email_body_soup(self, msg, logger):
"Extract and decode email body and return it as BeautifulSoup object."
# try to find the body part
self.logger.debug(f"{qid}: trying to find email body")
logger.debug("trying to find email body")
try:
body = msg.get_body(preferencelist=("html", "plain"))
except Exception as e:
self.logger.error(
f"{qid}: an error occured in "
f"email.message.EmailMessage.get_body: {e}")
logger.error(
f"an error occured in email.message.EmailMessage.get_body: "
f"{e}")
body = None
if body:
@@ -233,55 +161,51 @@ class EMailNotification(BaseNotification):
try:
content = content.decode(encoding=charset, errors="replace")
except LookupError:
self.logger.info(
f"{qid}: unknown encoding '{charset}', "
f"falling back to UTF-8")
logger.info(
f"unknown encoding '{charset}', falling back to UTF-8")
content = content.decode("utf-8", errors="replace")
content_type = body.get_content_type()
if content_type == "text/plain":
# convert text/plain to text/html
self.logger.debug(
f"{qid}: content type is {content_type}, "
logger.debug(
f"content type is {content_type}, "
f"converting to text/html")
content = re.sub(r"^(.*)$", r"\1<br/>",
escape(content, quote=False),
flags=re.MULTILINE)
else:
self.logger.debug(
f"{qid}: content type is {content_type}")
logger.debug(f"content type is {content_type}")
else:
self.logger.error(
f"{qid}: unable to find email body")
logger.error("unable to find email body")
content = "ERROR: unable to find email body"
# create BeautifulSoup object
length = len(content)
self.logger.debug(
f"{qid}: trying to create BeatufilSoup object with "
logger.debug(
f"trying to create BeatufilSoup object with "
f"parser lib {self.parser_lib}, "
f"text length is {length} bytes")
soup = BeautifulSoup(content, self.parser_lib)
self.logger.debug(
f"{qid}: sucessfully created BeautifulSoup object")
logger.debug("sucessfully created BeautifulSoup object")
return soup
def sanitize(self, qid, soup):
def sanitize(self, soup, logger):
"Sanitize mail html text."
self.logger.debug(f"{qid}: sanitizing email text")
logger.debug("sanitizing email text")
# completly remove bad elements
for element in soup(EMailNotification._bad_tags):
self.logger.debug(
f"{qid}: removing dangerous tag '{element.name}' "
logger.debug(
f"removing dangerous tag '{element.name}' "
f"and its content")
element.extract()
# remove not whitelisted elements, but keep their content
for element in soup.find_all(True):
if element.name not in EMailNotification._good_tags:
self.logger.debug(
f"{qid}: removing tag '{element.name}', keep its content")
logger.debug(
f"removing tag '{element.name}', keep its content")
element.replaceWithChildren()
# remove not whitelisted attributes
@@ -289,126 +213,130 @@ class EMailNotification(BaseNotification):
for attribute in list(element.attrs.keys()):
if attribute not in EMailNotification._good_attributes:
if element.name == "a" and attribute == "href":
self.logger.debug(
f"{qid}: setting attribute href to '#' "
logger.debug(
f"setting attribute href to '#' "
f"on tag '{element.name}'")
element["href"] = "#"
else:
self.logger.debug(
f"{qid}: removing attribute '{attribute}' "
logger.debug(
f"removing attribute '{attribute}' "
f"from tag '{element.name}'")
del(element.attrs[attribute])
return soup
def notify(self, qid, storage_id, mailfrom, recipients, headers, fp,
subgroups=None, named_subgroups=None, synchronous=False):
def notify(self, msg, qid, mailfrom, recipients, logger,
template_vars=defaultdict(str), synchronous=False):
"Notify recipients via email."
super(
EMailNotification,
self).notify(
qid,
storage_id,
mailfrom,
recipients,
headers,
fp,
subgroups,
named_subgroups,
synchronous)
# extract body from email
soup = self.get_email_body_soup(
qid, email.message_from_binary_file(fp, policy=default_policy))
soup = self.get_email_body_soup(msg, logger)
# replace picture sources
image_replaced = False
if self.strip_images:
self.logger.debug(
f"{qid}: looking for images to strip")
logger.debug("looking for images to strip")
for element in soup("img"):
if "src" in element.attrs.keys():
self.logger.debug(
f"{qid}: strip image: {element['src']}")
logger.debug(f"strip image: {element['src']}")
element.extract()
elif self.replacement_img:
self.logger.debug(
f"{qid}: looking for images to replace")
logger.debug("looking for images to replace")
for element in soup("img"):
if "src" in element.attrs.keys():
self.logger.debug(
f"{qid}: replacing image: {element['src']}")
logger.debug(f"replacing image: {element['src']}")
element["src"] = "cid:removed_for_security_reasons"
image_replaced = True
# sanitizing email text of original email
sanitized_text = self.sanitize(qid, soup)
sanitized_text = self.sanitize(soup, logger)
del soup
# sending email notifications
for recipient in recipients:
self.logger.debug(
f"{qid}: generating notification email for '{recipient}'")
self.logger.debug(f"{qid}: parsing email template")
logger.debug(
f"generating notification email for '{recipient}'")
logger.debug("parsing email template")
# generate dict containing all template variables
variables = defaultdict(
str,
EMAIL_HTML_TEXT=sanitized_text,
EMAIL_FROM=escape(headers["from"], quote=False),
EMAIL_ENVELOPE_FROM=escape(mailfrom, quote=False),
EMAIL_ENVELOPE_FROM_URL=escape(quote(mailfrom),
variables = defaultdict(str, template_vars)
variables.update({
"HTML_TEXT": sanitized_text,
"FROM": escape(msg["from"], quote=False),
"ENVELOPE_FROM": escape(mailfrom, quote=False),
"ENVELOPE_FROM_URL": escape(quote(mailfrom),
quote=False),
EMAIL_TO=escape(headers["to"], quote=False),
EMAIL_ENVELOPE_TO=escape(recipient, quote=False),
EMAIL_ENVELOPE_TO_URL=escape(quote(recipient)),
EMAIL_SUBJECT=escape(headers["subject"], quote=False),
EMAIL_QUARANTINE_ID=storage_id)
if subgroups:
number = 0
for subgroup in subgroups:
variables[f"SUBGROUP_{number}"] = escape(subgroup,
quote=False)
if named_subgroups:
for key, value in named_subgroups.items():
named_subgroups[key] = escape(value, quote=False)
variables.update(named_subgroups)
"TO": escape(msg["to"], quote=False),
"ENVELOPE_TO": escape(recipient, quote=False),
"ENVELOPE_TO_URL": escape(quote(recipient)),
"SUBJECT": escape(msg["subject"], quote=False)})
# parse template
htmltext = self.template.format_map(variables)
msg = MIMEMultipart('related')
msg["From"] = self.from_header.format_map(
defaultdict(str, EMAIL_FROM=headers["from"]))
msg["To"] = headers["to"]
msg["Subject"] = self.subject.format_map(
defaultdict(str, EMAIL_SUBJECT=headers["subject"]))
msg["Date"] = email.utils.formatdate()
msg.attach(MIMEText(htmltext, "html", 'UTF-8'))
newmsg = MIMEMultipart('related')
newmsg["From"] = self.from_header.format_map(
defaultdict(str, FROM=msg["from"]))
newmsg["To"] = msg["to"]
newmsg["Subject"] = self.subject.format_map(
defaultdict(str, SUBJECT=msg["subject"]))
newmsg["Date"] = email.utils.formatdate()
newmsg.attach(MIMEText(htmltext, "html", 'UTF-8'))
if image_replaced:
self.logger.debug(
f"{qid}: attaching notification_replacement_img")
msg.attach(self.replacement_img)
logger.debug("attaching notification_replacement_img")
newmsg.attach(self.replacement_img)
for img in self.embedded_imgs:
self.logger.debug(f"{qid}: attaching imgage")
msg.attach(img)
for img in self.embed_imgs:
logger.debug("attaching imgage")
newmsg.attach(img)
self.logger.debug(
f"{qid}: sending notification email to: {recipient}")
logger.debug(f"sending notification email to: {recipient}")
if synchronous:
try:
mailer.smtp_send(self.smtp_host, self.smtp_port,
self.mailfrom, recipient, msg.as_string())
self.mailfrom, recipient,
newmsg.as_string())
except Exception as e:
raise RuntimeError(
f"error while sending email to '{recipient}': {e}")
else:
mailer.sendmail(self.smtp_host, self.smtp_port, qid,
self.mailfrom, recipient, msg.as_string(),
self.mailfrom, recipient, newmsg.as_string(),
"notification email")
def execute(self, milter, logger):
super().execute(milter, logger)
# list of notification types and their related notification classes
TYPES = {"email": EMailNotification}
self.notify(msg=milter.msg, qid=milter.qid,
mailfrom=milter.msginfo["mailfrom"],
recipients=milter.msginfo["rcpts"],
template_vars=milter.msginfo["vars"],
logger=logger)
class Notify:
NOTIFICATION_TYPES = {
"email": EMailNotification}
def __init__(self, cfg, local_addrs, debug):
self.cfg = cfg
self.logger = logging.getLogger(cfg["name"])
self.logger.setLevel(cfg.get_loglevel(debug))
nodification_type = cfg["args"]["type"]
del cfg["args"]["type"]
cfg["args"]["pretend"] = cfg["pretend"]
self._notification = self.NOTIFICATION_TYPES[nodification_type](
**cfg["args"])
self._headersonly = self._notification._headersonly
def __str__(self):
cfg = []
for key, value in self.cfg["args"].items():
cfg.append(f"{key}={value}")
class_name = type(self._notification).__name__
return f"{class_name}(" + ", ".join(cfg) + ")"
def execute(self, milter):
logger = CustomLogger(
self.logger, {"name": self.cfg["name"], "qid": milter.qid})
self._notification.execute(milter, logger)

66
pyquarantine/rule.py Normal file
View File

@@ -0,0 +1,66 @@
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = ["Rule"]
from pyquarantine.action import Action
from pyquarantine.conditions import Conditions
class Rule:
"""
Rule to implement multiple actions on emails.
"""
def __init__(self, cfg, local_addrs, debug):
self.cfg = cfg
self.conditions = cfg["conditions"] if "conditions" in cfg else None
if self.conditions is not None:
self.conditions["name"] = f"{cfg['name']}: condition"
self.conditions["loglevel"] = cfg["loglevel"]
self.conditions = Conditions(self.conditions, local_addrs, debug)
self.actions = []
for idx, action_cfg in enumerate(cfg["actions"]):
if "name" in action_cfg:
action_cfg["name"] = f"{cfg['name']}: {action_cfg['name']}"
else:
action_cfg["name"] = f"action#{idx}"
if "loglevel" not in action_cfg:
action_cfg["loglevel"] = cfg["loglevel"]
if "pretend" not in action_cfg:
action_cfg["pretend"] = cfg["pretend"]
self.actions.append(Action(action_cfg, local_addrs, debug))
def __str__(self):
cfg = []
for key in ["name", "loglevel", "pretend"]:
value = self.cfg[key]
cfg.append(f"{key}={value}")
if self.conditions is not None:
cfg.append(f"conditions={self.conditions}")
actions = []
for action in self.actions:
actions.append(str(action))
cfg.append("actions=[" + ", ".join(actions) + "]")
return "Rule(" + ", ".join(cfg) + ")"
def execute(self, milter):
"""Execute all actions of this rule."""
if self.conditions is None or \
self.conditions.match(milter):
for action in self.actions:
milter_action = action.execute(milter)
if milter_action is not None:
return milter_action

141
pyquarantine/run.py
View File

@@ -1,127 +1,152 @@
#!/usr/bin/env python
#
# PyQuarantine-Milter is free software: you can redistribute it and/or modify
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# PyQuarantine-Milter is distributed in the hope that it will be useful,
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with PyQuarantineMilter. If not, see <http://www.gnu.org/licenses/>.
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = ["main"]
import Milter
import argparse
import logging
import logging.handlers
import sys
import pyquarantine
from pyquarantine import mailer
from pyquarantine import ModifyMilter
from pyquarantine import __version__ as version
from pyquarantine.config import get_milter_config
def main():
"Run PyQuarantine-Milter."
# parse command line
python_version = ".".join([str(v) for v in sys.version_info[0:3]])
python_version = f"{python_version}-{sys.version_info[3]}"
"Run pyquarantine."
parser = argparse.ArgumentParser(
description="PyQuarantine milter daemon",
description="pyquarantine-milter daemon",
formatter_class=lambda prog: argparse.HelpFormatter(
prog, max_help_position=45, width=140))
parser.add_argument(
"-c", "--config",
help="List of config files to read.",
nargs="+",
default=pyquarantine.QuarantineMilter.get_cfg_files())
"-c", "--config", help="Config file to read.",
default="/etc/pyquarantine/pyquarantine.conf")
parser.add_argument(
"-s", "--socket",
"-s",
"--socket",
help="Socket used to communicate with the MTA.",
default="inet:8899@127.0.0.1")
default="")
parser.add_argument(
"-d", "--debug",
"-d",
"--debug",
help="Log debugging messages.",
action="store_true")
parser.add_argument(
"-t", "--test",
"-t",
"--test",
help="Check configuration.",
action="store_true")
parser.add_argument(
"-v", "--version",
help="Print version.",
action="version",
version=f"%(prog)s ({version})")
version=f"%(prog)s {version} (python {python_version})")
args = parser.parse_args()
# setup logging
loglevel = logging.INFO
logname = "pyquarantine-milter"
syslog_name = logname
if args.debug:
loglevel = logging.DEBUG
logname = f"{logname}[%(name)s]"
syslog_name = f"{syslog_name}: [%(name)s] %(levelname)s"
# set config files for milter class
pyquarantine.QuarantineMilter.set_cfg_files(args.config)
root_logger = logging.getLogger()
root_logger.setLevel(loglevel)
root_logger.setLevel(logging.DEBUG)
# setup console log
stdouthandler = logging.StreamHandler(sys.stdout)
stdouthandler.setLevel(logging.DEBUG)
formatter = logging.Formatter("%(message)s")
formatter = logging.Formatter("%(levelname)s: %(message)s")
stdouthandler.setFormatter(formatter)
root_logger.addHandler(stdouthandler)
logger = logging.getLogger(__name__)
if args.test:
if not args.debug:
logger.setLevel(logging.INFO)
try:
pyquarantine.setup_milter(test=args.test)
print("Configuration ok")
logger.debug("read milter configuration")
cfg = get_milter_config(args.config)
logger.setLevel(cfg.get_loglevel(args.debug))
if args.socket:
socket = args.socket
elif "socket" in cfg:
socket = cfg["socket"]
else:
raise RuntimeError(
"listening socket is neither specified on the command line "
"nor in the configuration file")
if not cfg["rules"]:
raise RuntimeError("no rules configured")
for rule in cfg["rules"]:
if not rule["actions"]:
raise RuntimeError(
f"{rule['name']}: no actions configured")
except (RuntimeError, AssertionError) as e:
logger.error(f"config error: {e}")
sys.exit(255)
try:
ModifyMilter.set_config(cfg, args.debug)
except RuntimeError as e:
logger.error(e)
sys.exit(255)
else:
sys.exit(254)
if args.test:
print("Configuration OK")
sys.exit(0)
formatter = logging.Formatter(
f"%(asctime)s {logname}: [%(levelname)s] %(message)s",
datefmt="%Y-%m-%d %H:%M:%S")
# setup console log for runtime
formatter = logging.Formatter("%(asctime)s - %(levelname)s: %(message)s")
stdouthandler.setFormatter(formatter)
stdouthandler.setLevel(logging.DEBUG)
# setup syslog
sysloghandler = logging.handlers.SysLogHandler(
address="/dev/log", facility=logging.handlers.SysLogHandler.LOG_MAIL)
sysloghandler.setLevel(loglevel)
formatter = logging.Formatter(f"{syslog_name}: %(message)s")
sysloghandler.setFormatter(formatter)
sysloghandler.setFormatter(
logging.Formatter("pyquarantine: %(message)s"))
root_logger.addHandler(sysloghandler)
logger.info("PyQuarantine-Milter starting")
try:
# generate milter config
pyquarantine.setup_milter()
except RuntimeError as e:
logger.error(e)
sys.exit(255)
logger.info("pyquarantine-milter starting")
# register to have the Milter factory create instances of your class:
Milter.factory = pyquarantine.QuarantineMilter
# register milter factory class
Milter.factory = ModifyMilter
Milter.set_exception_policy(Milter.TEMPFAIL)
# run milter
if args.debug:
Milter.setdbg(1)
rc = 0
try:
Milter.runmilter("pyquarantine-milter", socketname=args.socket,
timeout=300)
Milter.runmilter("pyquarantine", socketname=socket, timeout=600)
except Milter.milter.error as e:
logger.error(e)
rc = 255
pyquarantine.mailer.queue.put(None)
logger.info("PyQuarantine-Milter terminated")
mailer.queue.put(None)
logger.info("pyquarantine-milter stopped")
sys.exit(rc)

355
pyquarantine/storage.py
View File

@@ -1,17 +1,23 @@
# PyQuarantine-Milter is free software: you can redistribute it and/or modify
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# PyQuarantine-Milter is distributed in the hope that it will be useful,
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with PyQuarantineMilter. If not, see <http://www.gnu.org/licenses/>.
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = [
"BaseMailStorage",
"FileMailStorage",
"Store",
"Quarantine"]
import json
import logging
import os
@@ -19,23 +25,31 @@ import os
from calendar import timegm
from datetime import datetime
from glob import glob
from shutil import copyfileobj
from time import gmtime
from pyquarantine.base import CustomLogger
from pyquarantine.conditions import Conditions
from pyquarantine.config import ActionConfig
from pyquarantine.notify import Notify
class BaseMailStorage(object):
class BaseMailStorage:
"Mail storage base class"
storage_type = "base"
_headersonly = True
def __init__(self, name, global_cfg, cfg, test=False):
self.name = name
self.logger = logging.getLogger(__name__)
def __init__(self, original=False, metadata=False, metavar=None,
pretend=False):
self.original = original
self.metadata = metadata
self.metavar = metavar
self.pretend = False
def add(self, qid, mailfrom, recipients, headers,
fp, subgroups=None, named_subgroups=None):
def add(self, data, qid, mailfrom="", recipients=[]):
"Add email to storage."
fp.seek(0)
return ""
return ("", "")
def execute(self, milter, logger):
return
def find(self, mailfrom=None, recipients=None, older_than=None):
"Find emails in storage."
@@ -50,116 +64,158 @@ class BaseMailStorage(object):
return
def get_mail(self, storage_id):
"Return a file pointer to the email and metadata."
"Return email and metadata."
return
class FileMailStorage(BaseMailStorage):
"Storage class to store mails on filesystem."
storage_type = "file"
_headersonly = False
def __init__(self, name, global_cfg, cfg, test=False):
super(FileMailStorage, self).__init__(name, global_cfg, cfg, test)
defaults = {}
# check config
for opt in ["storage_directory"] + list(defaults.keys()):
if opt in cfg:
continue
if opt in global_cfg:
cfg[opt] = global_cfg[opt]
elif opt in defaults:
cfg[opt] = defaults[opt]
else:
def __init__(self, directory, original=False, metadata=False, metavar=None,
mode=None, pretend=False):
super().__init__(original, metadata, metavar, pretend)
# check if directory exists and is writable
if not os.path.isdir(directory) or \
not os.access(directory, os.W_OK):
raise RuntimeError(
f"mandatory option '{opt}' not present in config "
f"section '{self.name}' or 'global'")
self.directory = cfg["storage_directory"]
f"directory '{directory}' does not exist or is "
f"not writable")
self.directory = directory
try:
self.mode = int(mode, 8) if mode is not None else None
if self.mode is not None and self.mode > 511:
raise ValueError
except ValueError:
raise RuntimeError(f"invalid mode '{mode}'")
# check if quarantine directory exists and is writable
if not os.path.isdir(self.directory) or not os.access(
self.directory, os.W_OK):
raise RuntimeError(
f"file quarantine directory '{self.directory}' does "
f"not exist or is not writable")
self._metadata_suffix = ".metadata"
def _save_datafile(self, storage_id, fp):
def __str__(self):
cfg = []
cfg.append(f"metadata={self.metadata}")
cfg.append(f"metavar={self.metavar}")
cfg.append(f"pretend={self.pretend}")
cfg.append(f"directory={self.directory}")
cfg.append(f"original={self.original}")
return "FileMailStorage(" + ", ".join(cfg) + ")"
def get_storageid(self, qid):
timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
return f"{timestamp}_{qid}"
def _get_file_paths(self, storage_id):
datafile = os.path.join(self.directory, storage_id)
metafile = f"{datafile}{self._metadata_suffix}"
return metafile, datafile
def _save_datafile(self, datafile, data):
try:
if self.mode is None:
with open(datafile, "wb") as f:
copyfileobj(fp, f)
f.write(data)
else:
umask = os.umask(0)
with open(
os.open(datafile, os.O_CREAT | os.O_WRONLY, self.mode),
"wb") as f:
f.write(data)
os.umask(umask)
except IOError as e:
raise RuntimeError(f"unable save data file: {e}")
def _save_metafile(self, storage_id, metadata):
metafile = os.path.join(
self.directory, f"{storage_id}{self._metadata_suffix}")
def _save_metafile(self, metafile, metadata):
try:
if self.mode is None:
with open(metafile, "w") as f:
json.dump(metadata, f, indent=2)
else:
umask = os.umask(0)
with open(
os.open(metafile, os.O_CREAT | os.O_WRONLY, self.mode),
"w") as f:
json.dump(metadata, f, indent=2)
os.umask(umask)
except IOError as e:
raise RuntimeError(f"unable to save metadata file: {e}")
def _remove(self, storage_id):
datafile = os.path.join(self.directory, storage_id)
metafile = f"{datafile}{self._metadata_suffix}"
metafile, datafile = self._get_file_paths(storage_id)
try:
if self.metadata:
os.remove(metafile)
except IOError as e:
raise RuntimeError(f"unable to remove metadata file: {e}")
try:
os.remove(datafile)
except IOError as e:
raise RuntimeError(f"unable to remove data file: {e}")
raise RuntimeError(f"unable to remove file: {e}")
def add(self, qid, mailfrom, recipients, headers,
fp, subgroups=None, named_subgroups=None):
def add(self, data, qid, mailfrom="", recipients=[], subject=""):
"Add email to file storage and return storage id."
super(
FileMailStorage,
self).add(
qid,
mailfrom,
recipients,
headers,
fp,
subgroups,
named_subgroups)
timestamp = datetime.now().strftime("%Y%m%d%H%M%S")
storage_id = f"{timestamp}_{qid}"
super().add(data, qid, mailfrom, recipients)
storage_id = self.get_storageid(qid)
metafile, datafile = self._get_file_paths(storage_id)
# save mail
self._save_datafile(storage_id, fp)
self._save_datafile(datafile, data)
if not self.metadata:
return storage_id, None, datafile
# save metadata
metadata = {
"mailfrom": mailfrom,
"recipients": recipients,
"headers": headers,
"date": timegm(gmtime()),
"queue_id": qid,
"subgroups": subgroups,
"named_subgroups": named_subgroups
}
"subject": subject,
"timestamp": timegm(gmtime()),
"queue_id": qid}
try:
self._save_metafile(storage_id, metadata)
self._save_metafile(metafile, metadata)
except RuntimeError as e:
datafile = os.path.join(self.directory, storage_id)
os.remove(datafile)
raise e
return storage_id
return storage_id, metafile, datafile
def execute(self, milter, logger):
if self.original:
milter.fp.seek(0)
data = milter.fp.read
mailfrom = milter.mailfrom
recipients = list(milter.rcpts)
subject = ""
else:
data = milter.msg.as_bytes
mailfrom = milter.msginfo["mailfrom"]
recipients = list(milter.msginfo["rcpts"])
subject = milter.msg["subject"] or ""
if not self.pretend:
storage_id, metafile, datafile = self.add(
data(), milter.qid, mailfrom, recipients, subject)
logger.info(f"stored message in file {datafile}")
else:
storage_id = self.get_storageid(milter.qid)
metafile, datafile = self._get_file_paths(storage_id)
if self.metavar:
milter.msginfo["vars"][f"{self.metavar}_ID"] = storage_id
milter.msginfo["vars"][f"{self.metavar}_DATAFILE"] = datafile
if self.metadata:
milter.msginfo["vars"][f"{self.metavar}_METAFILE"] = metafile
def get_metadata(self, storage_id):
"Return metadata of email in storage."
super(FileMailStorage, self).get_metadata(storage_id)
super().get_metadata(storage_id)
metafile = os.path.join(
self.directory, f"{storage_id}{self._metadata_suffix}")
if not self.metadata:
return None
metafile, _ = self._get_file_paths(storage_id)
if not os.path.isfile(metafile):
raise RuntimeError(
f"invalid storage id '{storage_id}'")
@@ -177,12 +233,15 @@ class FileMailStorage(BaseMailStorage):
def find(self, mailfrom=None, recipients=None, older_than=None):
"Find emails in storage."
super(FileMailStorage, self).find(mailfrom, recipients, older_than)
super().find(mailfrom, recipients, older_than)
if isinstance(mailfrom, str):
mailfrom = [mailfrom]
if isinstance(recipients, str):
recipients = [recipients]
if not self.metadata:
return {}
emails = {}
metafiles = glob(os.path.join(
self.directory, f"*{self._metadata_suffix}"))
@@ -215,18 +274,22 @@ class FileMailStorage(BaseMailStorage):
def delete(self, storage_id, recipients=None):
"Delete email from storage."
super(FileMailStorage, self).delete(storage_id, recipients)
super().delete(storage_id, recipients)
if not recipients or not self.metadata:
self._remove(storage_id)
return
try:
metadata = self.get_metadata(storage_id)
except RuntimeError as e:
raise RuntimeError(f"unable to delete email: {e}")
if not recipients:
self._remove(storage_id)
else:
metafile, _ = self._get_file_paths(storage_id)
if type(recipients) == str:
recipients = [recipients]
for recipient in recipients:
if recipient not in metadata["recipients"]:
raise RuntimeError(f"invalid recipient '{recipient}'")
@@ -234,19 +297,135 @@ class FileMailStorage(BaseMailStorage):
if not metadata["recipients"]:
self._remove(storage_id)
else:
self._save_metafile(storage_id, metadata)
self._save_metafile(metafile, metadata)
def get_mail(self, storage_id):
super(FileMailStorage, self).get_mail(storage_id)
super().get_mail(storage_id)
metadata = self.get_metadata(storage_id)
datafile = os.path.join(self.directory, storage_id)
_, datafile = self._get_file_paths(storage_id)
try:
fp = open(datafile, "rb")
data = open(datafile, "rb").read()
except IOError as e:
raise RuntimeError(f"unable to open email data file: {e}")
return (fp, metadata)
return (metadata, data)
# list of storage types and their related storage classes
TYPES = {"file": FileMailStorage}
class Store:
STORAGE_TYPES = {
"file": FileMailStorage}
def __init__(self, cfg, local_addrs, debug):
self.cfg = cfg
self.logger = logging.getLogger(cfg["name"])
self.logger.setLevel(cfg.get_loglevel(debug))
storage_type = cfg["args"]["type"]
del cfg["args"]["type"]
cfg["args"]["pretend"] = cfg["pretend"]
self._storage = self.STORAGE_TYPES[storage_type](
**cfg["args"])
self._headersonly = self._storage._headersonly
def __str__(self):
cfg = []
for key, value in self.cfg["args"].items():
cfg.append(f"{key}={value}")
class_name = type(self._storage).__name__
return f"{class_name}(" + ", ".join(cfg) + ")"
def execute(self, milter):
logger = CustomLogger(
self.logger, {"name": self.cfg["name"], "qid": milter.qid})
self._storage.execute(milter, logger)
class Quarantine:
"Quarantine class."
_headersonly = False
def __init__(self, cfg, local_addrs, debug):
self.cfg = cfg
self.logger = logging.getLogger(cfg["name"])
self.logger.setLevel(cfg.get_loglevel(debug))
store_cfg = ActionConfig({
"name": cfg["name"],
"loglevel": cfg["loglevel"],
"pretend": cfg["pretend"],
"type": "store",
"args": cfg["args"]["store"].get_config()})
store_cfg["args"]["metadata"] = True
self.store = Store(store_cfg, local_addrs, debug)
self.notify = None
if "notify" in cfg["args"]:
notify_cfg = ActionConfig({
"name": cfg["name"],
"loglevel": cfg["loglevel"],
"pretend": cfg["pretend"],
"type": "notify",
"args": cfg["args"]["notify"].get_config()})
self.notify = Notify(notify_cfg, local_addrs, debug)
self.whitelist = None
if "whitelist" in cfg["args"]:
whitelist_cfg = cfg["args"]["whitelist"]
whitelist_cfg["name"] = cfg["name"]
whitelist_cfg["loglevel"] = cfg["loglevel"]
self.whitelist = Conditions(
whitelist_cfg,
local_addrs=[],
debug=debug)
self.milter_action = None
if "milter_action" in cfg["args"]:
self.milter_action = cfg["args"]["milter_action"]
self.reject_reason = None
if "reject_reason" in cfg["args"]:
self.reject_reason = cfg["args"]["reject_reason"]
def __str__(self):
cfg = []
cfg.append(f"store={str(self.store)}")
if self.notify is not None:
cfg.append(f"notify={str(self.notify)}")
if self.whitelist is not None:
cfg.append(f"whitelist={str(self.whitelist)}")
for key in ["milter_action", "reject_reason"]:
if key not in self.cfg["args"]:
continue
value = self.cfg["args"][key]
cfg.append(f"{key}={value}")
class_name = type(self).__name__
return f"{class_name}(" + ", ".join(cfg) + ")"
def execute(self, milter):
logger = CustomLogger(
self.logger, {"name": self.cfg["name"], "qid": milter.qid})
wl_rcpts = []
if self.whitelist:
wl_rcpts = self.whitelist.get_wl_rcpts(
milter.msginfo["mailfrom"], milter.msginfo["rcpts"], logger)
logger.info(f"whitelisted recipients: {wl_rcpts}")
rcpts = [
rcpt for rcpt in milter.msginfo["rcpts"] if rcpt not in wl_rcpts]
if not rcpts:
# all recipients whitelisted
return
logger.info(f"add to quarantine for recipients: {rcpts}")
milter.msginfo["rcpts"] = rcpts
self.store.execute(milter)
if self.notify is not None:
self.notify.execute(milter)
milter.msginfo["rcpts"].extend(wl_rcpts)
milter.delrcpt(rcpts)
if self.milter_action is not None and not milter.msginfo["rcpts"]:
return (self.milter_action, self.reject_reason)

122
pyquarantine/whitelist.py
View File

@@ -1,17 +1,21 @@
# PyQuarantine-Milter is free software: you can redistribute it and/or modify
# pyquarantine is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# PyQuarantine-Milter is distributed in the hope that it will be useful,
# pyquarantine is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with PyQuarantineMilter. If not, see <http://www.gnu.org/licenses/>.
# along with pyquarantine. If not, see <http://www.gnu.org/licenses/>.
#
__all__ = [
"DatabaseWhitelist",
"WhitelistBase"]
import logging
import peewee
import re
@@ -20,17 +24,23 @@ from datetime import datetime
from playhouse.db_url import connect
class WhitelistBase(object):
class WhitelistBase:
"Whitelist base class"
def __init__(self, cfg, debug):
self.cfg = cfg
self.logger = logging.getLogger(cfg["name"])
self.logger.setLevel(cfg.get_loglevel(debug))
whitelist_type = "base"
peewee_logger = logging.getLogger("peewee")
peewee_logger.setLevel(cfg.get_loglevel(debug))
def __init__(self, name, global_cfg, cfg, test=False):
self.name = name
self.test = test
self.logger = logging.getLogger(__name__)
self.valid_entry_regex = re.compile(
r"^[a-zA-Z0-9_.=+-]*?(@[a-zA-Z0-9-]+\.[a-zA-Z0-9-.]+)?$")
self.batv_regex = re.compile(
r"^prvs=[0-9]{4}[0-9A-Fa-f]{6}=(?P<LEFT_PART>.+?)@")
def remove_batv(self, addr):
return self.batv_regex.sub(r"\g<LEFT_PART>", addr, count=1)
def check(self, mailfrom, recipient):
"Check if mailfrom/recipient combination is whitelisted."
@@ -63,7 +73,7 @@ class WhitelistModel(peewee.Model):
permanent = peewee.BooleanField(default=False)
class Meta(object):
class Meta:
indexes = (
# trailing comma is mandatory if only one index should be created
(('mailfrom', 'recipient'), True),
@@ -76,32 +86,11 @@ class DatabaseWhitelist(WhitelistBase):
_db_connections = {}
_db_tables = {}
def __init__(self, name, global_cfg, cfg, test=False):
super(
DatabaseWhitelist,
self).__init__(
global_cfg,
cfg,
test)
def __init__(self, cfg, debug):
super().__init__(cfg, debug)
defaults = {}
# check config
for opt in ["whitelist_db_connection",
"whitelist_db_table"] + list(defaults.keys()):
if opt in cfg:
continue
if opt in global_cfg:
cfg[opt] = global_cfg[opt]
elif opt in defaults:
cfg[opt] = defaults[opt]
else:
raise RuntimeError(
f"mandatory option '{opt}' not present in config "
f"section '{self.name}' or 'global'")
tablename = cfg["whitelist_db_table"]
connection_string = cfg["whitelist_db_connection"]
tablename = cfg["table"]
connection_string = cfg["connection"]
if connection_string in DatabaseWhitelist._db_connections.keys():
db = DatabaseWhitelist._db_connections[connection_string]
@@ -125,22 +114,29 @@ class DatabaseWhitelist(WhitelistBase):
self.meta = Meta
self.meta.database = db
self.meta.table_name = tablename
self.model = type(f"WhitelistModel_{self.name}", (WhitelistModel,), {
"Meta": self.meta
})
self.model = type(
f"WhitelistModel_{self.cfg['name']}",
(WhitelistModel,),
{"Meta": self.meta})
if connection_string not in DatabaseWhitelist._db_tables.keys():
DatabaseWhitelist._db_tables[connection_string] = []
if tablename not in DatabaseWhitelist._db_tables[connection_string]:
DatabaseWhitelist._db_tables[connection_string].append(tablename)
if not self.test:
try:
db.create_tables([self.model])
except Exception as e:
raise RuntimeError(
f"unable to initialize table '{tablename}': {e}")
def __str__(self):
cfg = []
for arg in ("connection", "table"):
if arg in self.cfg:
cfg.append(f"{arg}={self.cfg[arg]}")
return "DatabaseWhitelist(" + ", ".join(cfg) + ")"
def _entry_to_dict(self, entry):
result = {}
result[entry.id] = {
@@ -163,12 +159,14 @@ class DatabaseWhitelist(WhitelistBase):
value += 1
return value
def check(self, mailfrom, recipient):
def check(self, mailfrom, recipient, logger):
# check if mailfrom/recipient combination is whitelisted
super(DatabaseWhitelist, self).check(mailfrom, recipient)
super().check(mailfrom, recipient)
mailfrom = self.remove_batv(mailfrom)
recipient = self.remove_batv(recipient)
# generate list of possible mailfroms
self.logger.debug(
logger.debug(
f"query database for whitelist entries from <{mailfrom}> "
f"to <{recipient}>")
mailfroms = [""]
@@ -212,7 +210,7 @@ class DatabaseWhitelist(WhitelistBase):
def find(self, mailfrom=None, recipients=None, older_than=None):
"Find whitelist entries."
super(DatabaseWhitelist, self).find(mailfrom, recipients, older_than)
super().find(mailfrom, recipients, older_than)
if isinstance(mailfrom, str):
mailfrom = [mailfrom]
@@ -243,14 +241,15 @@ class DatabaseWhitelist(WhitelistBase):
def add(self, mailfrom, recipient, comment, permanent):
"Add entry to whitelist."
super(
DatabaseWhitelist,
self).add(
super().add(
mailfrom,
recipient,
comment,
permanent)
mailfrom = self.remove_batv(mailfrom)
recipient = self.remove_batv(recipient)
try:
self.model.create(
mailfrom=mailfrom,
@@ -262,7 +261,7 @@ class DatabaseWhitelist(WhitelistBase):
def delete(self, whitelist_id):
"Delete entry from whitelist."
super(DatabaseWhitelist, self).delete(whitelist_id)
super().delete(whitelist_id)
try:
query = self.model.delete().where(self.model.id == whitelist_id)
@@ -273,32 +272,3 @@ class DatabaseWhitelist(WhitelistBase):
if deleted == 0:
raise RuntimeError("invalid whitelist id")
class WhitelistCache(object):
def __init__(self):
self.cache = {}
def load(self, whitelist, mailfrom, recipients):
for recipient in recipients:
self.check(whitelist, mailfrom, recipient)
def check(self, whitelist, mailfrom, recipient):
if whitelist not in self.cache.keys():
self.cache[whitelist] = {}
if recipient not in self.cache[whitelist].keys():
self.cache[whitelist][recipient] = None
if self.cache[whitelist][recipient] is None:
self.cache[whitelist][recipient] = whitelist.check(
mailfrom, recipient)
return self.cache[whitelist][recipient]
def get_recipients(self, whitelist, mailfrom, recipients):
self.load(whitelist, mailfrom, recipients)
return list(filter(
lambda x: self.cache[whitelist][x],
self.cache[whitelist].keys()))
# list of whitelist types and their related whitelist classes
TYPES = {"db": DatabaseWhitelist}

30
setup.py
View File

@@ -4,17 +4,16 @@ def read_file(fname):
with open(fname, 'r') as f:
return f.read()
setup(name = "pyquarantine",
author = "Thomas Oettli",
author_email = "spacefreak@noop.ch",
description = "A pymilter based sendmail/postfix pre-queue filter.",
license = "GPL 3",
keywords = "quarantine milter",
url = "https://github.com/spacefreak86/pyquarantine-milter",
keywords = "header quarantine milter",
url = "https://github.com/spacefreak86/pyquarantine",
packages = ["pyquarantine"],
long_description = read_file("README.md"),
long_description_content_type="text/markdown",
long_description_content_type = "text/markdown",
classifiers = [
# 3 - Alpha
# 4 - Beta
@@ -26,12 +25,29 @@ setup(name = "pyquarantine",
"Programming Language :: Python :: 3",
"Topic :: Communications :: Email :: Filters"
],
include_package_data = True,
entry_points = {
"console_scripts": [
"pyquarantine-milter=pyquarantine.run:main",
"pyquarantine=pyquarantine.cli:main"
"pyquarantine=pyquarantine.run:main",
]
},
install_requires = ["pymilter", "netaddr", "beautifulsoup4[lxml]", "peewee"],
python_requires = ">=3.6"
data_files = [
(
"/etc/pyquarantine",
[
"pyquarantine/docs/pyquarantine.conf.example"
]
), (
"/etc/pyquarantine/templates",
[
"pyquarantine/docs/templates/disclaimer_html.template",
"pyquarantine/docs/templates/disclaimer_text.template",
"pyquarantine/docs/templates/notification.template",
"pyquarantine/docs/templates/removed.png"
]
)
],
install_requires = ["pymilter", "jsonschema", "netaddr", "beautifulsoup4[lxml]", "peewee"],
python_requires = ">=3.8"
)