How to re-apply Sieve filters to a Dovecot IMAP inbox

Mario Steinitz — Sun, 09 Oct 2022 05:15:58 +0000

Introduction

Our Dovecot IMAP email service user defined mail filter rules using the Managesieve protocol. His Sieve rules automatically classify the hundreds of emails flooding his inbox every day. They delete unwanted messages, store others in distinct folders, set flags and prioritize them for later reading. A typo in his sieve scripts caused the filters to break. Emails were sent directly to his inbox where now thousands of unclassified newsletters and spam mails linger, making him unable to determine the important communication. We were asked to help him refilter the messages using his fixed Sieve filters.

Example environment

For the sake of this article, we use `user.name@example.com` as email inbox. We advice our IMAP user to create a subfolder `Refilter` in his email inbox and move all unclassified emails into this folder.

Our example Dovecot email server uses `/var/vmail/[domain]/[user]` as storage location. User-defined custom Sieve scripts are located in a `/var/vmail/[domain]/[user]/.sieve` symbolic link that targets the enabled Sieve filter rules file. The inbox folder root is `INBOX`.

Proceedings

In order to re-apply Sieve rules on a Dovecot IMAP server, newer versions of Dovecot Pigeonhole come with the `sieve-filter` tool. It takes the target inbox, the sieve rules to apply, and the inbox folder to apply the rules to as arguments:

sieve-fil­ter -u "user.name@exam­ple.com" /var/vmail/exam­ple.com/user.name/.sieve INBOX.Refilter

By default, the `sieve-filter` command runs in simulation mode. To apply any changes, we have to explicitly enable execution mode by adding the `-e` option, and allow write and delete access to the mailbox by adding the `-W` option. We also want the Sieve scripts to be recompiled to apply any last changes by the user. So we add the `-C` option to the command (a complete reference of available options can be found at the Pigeonhole Dovecot sieve-filter documentation page):

sieve-fil­ter -e -W -C -u "user.name@exam­ple.com" /var/vmail/exam­ple.com/user.name/.sieve INBOX.Refilter

Considering that inboxes can grow rather large, and processing thousands of messages at a time may put a heavy strain on our live mail server, we may want to run the filtering with lower priority, only when the system isn't too busy. Therefore, we additionally use the `ionice` command with `-c2` for best-effort mode and `-n7` for lowest priority:

ion­ice -c2 -n7 sieve-fil­ter -e -W -C -u "user.name@exam­ple.com" /var/vmail/exam­ple.com/user.name/.sieve INBOX.Refilter

Wrap it into a Bash script

We don't always want to return to our knowledge base when having to help a mail user to refilter its messages. Therefore, we created a simple Bash script that just requires the mail user name as argument and does the job for us.

It is using the `doveadm` user lookup command to determine the users' Sieve script location and mailbox folders:

doveadm user "user.name@exam­ple.com"

Which, for existing mailbox users, returns an answer like this:

field   value

uid     *****

gid     *****

home    /var/vmail/exam­ple.com/user.name

mail    maildir:/var/vmail/exam­ple.com/user.name/Maildir

quo­ta_rule      *:stor­age=0B

sieve   /var/vmail/exam­ple.com/user.name/.sieve

We are using the `maildir` information to check for existing messages to process, as well as the `sieve` information to find the user-defined Sieve rules.

And here is the script as reference for our valued readers `sieve-refilter.sh`:

#!/bin/bash

#------------------------------------------------------------------------------

# Reap­ply user-defined Sieve fil­ters to a Dove­cot mail­box folder.

#------------------------------------------------------------------------------

# REQUIRE­MENTS:

# awk, cat, doveadm, echo, exit, find, grep, ion­ice, printf, read, shift,

# sieve-fil­ter, tr, wc, non-POSIX sup­port for `< <()` syn­tax

#------------------------------------------------------------------------------

# USAGE:

# ./sieve-refilter.sh <user> [<folder>]

#

# Options:

#   <user>             The Dove­cot user to reap­ply Sieve fil­ters for. E.g.,

#                      "user.name@exam­ple.com".

#   <folder>           Optional: The mail­box folder to process.

#                      Defaults to 'INBOX.Refilter'.

#------------------------------------------------------------------------------

# Author: SHORELESS Limited <con­tact@shore­less.lim­ited>

# See https://shore­less.ltd/kb20224071315

#------------------------------------------------------------------------------

# User option.

EMAIL_USER="$1"

if [[ -z "${EMAIL_USER}" ]]; then

  printf "\n\e[1;31mAn argu­ment error occurred.\e[0m\n\n"

  (>&2 printf 'Argu­ment error: The user para­me­ter "<user>" is miss­ing.')

  printf "\n\nUsage: ./sieve-refilter.sh <user> [<folder>]\n\n"

  printf "Options:\n\n"

  printf "  <user>             The Dove­cot user to reap­ply Sieve fil­ters for. E.g.,\n"

  printf "                     \"user.name@exam­ple.com\"."

  printf "  <folder>           Optional: The mail­box folder to process.\n"

  printf "                     Defaults to 'INBOX.Refilter'.\n\n"

  exit 22

fi

FOLDER="$2"

if [[ -z "${FOLDER}" ]]; then

  FOLDER="INBOX.Refilter"

fi

# Exe­cutes a com­mand while writ­ing STD­OUT and STDERR to vari­ables.

#

# @param STD­OUT

#   The vari­able to write STD­OUT to.

# @param STDERR

#   The vari­able to write STDERR to.

# @param COM­MAND

#   The com­mand to run.

# @param [ARG1[ ARG2[ ...[ ARGN]]]]

#   Any addi­tional argu­ments for the com­mand to run.

#

# @return

#   The com­mand exit code.

#

# @see https://stack­over­flow.com/ques­tions/11027679/#answer-59592881

catch() {

  {

    IFS=$'\n' read -r -d '' "${1}";

    IFS=$'\n' read -r -d '' "${2}";

    (IFS=$'\n' read -r -d '' _ER­RNO_; return ${_ER­RNO_});

  } < <((printf '\0%s\0%d\0' "$(((({ shift 2; ${@}; echo "${?}" 1>&3-; } | tr -d '\0' 1>&4-) 4>&2- 2>&1- | tr -d '\0' 1>&4-) 3>&1- | exit "$(cat)") 4>&1-)" "${?}" 1>&2) 2>&1)

}

echo "Dove­cot Sieve refilter"

echo "----------------------"

echo "Check­ing pro­vided options and mail­box:"

# Check, whether the user exists in the local Dove­cot instance.

printf '\e[0m- Check­ing, whether the user exists... '

catch STD­OUT STDERR doveadm user "${EMAIL_USER}"

if [ ${?} -eq 0 ]; then

  printf "\e[1;32m[ok]\e[0m\n"

else

  printf "\e[1;31m[error]\e[0m\n"

  (>&2 printf '%s\n%s\n\n' "  User lookup failed." "  ${STDERR}")

  exit 22;

fi

# Deter­mine mail folder.

printf '\e[0m- Deter­mine mail folder... '

MAIL_­FOLDER=$(echo "${STD­OUT}" | grep '^mail' | awk '{print $2}')

MAIL_­FOLDER="${MAIL_­FOLDER#maildir:}"

if [[ -z "${MAIL_­FOLDER}" ]] || [ ! \( -d "${MAIL_­FOLDER}" \) ]; then

  printf "\e[1;31m[error]\e[0m\n"

  (>&2 printf '%s\n%s\n\n' "  Dove­cot Maildir could not be found." "  ${MAIL_­FOLDER}")

  exit 2;

else

  printf "\e[1;32m[ok]\e[0m\n"

  printf "  ${MAIL_­FOLDER}\n"

fi

# Deter­mine sieve script.

printf '\e[0m- Check for Sieve script... '

SIEVE=$(echo "${STD­OUT}" | grep '^sieve' | awk '{print $2}')

if [[ -z "${SIEVE}" ]] || [ ! \( -e "${SIEVE}" \) ] || [ ! \( -f "${SIEVE}" \) ] || [ ! \( -r "${SIEVE}" \) ] || [ ! \( -s "${SIEVE}" \) ]; then

  printf "\e[1;31m[error]\e[0m\n"

  (>&2 printf '%s\n%s\n\n' "  Sieve script does not exist or is empty." "  ${SIEVE}")

  exit 2;

else

  printf "\e[1;32m[ok]\e[0m\n"

  printf "  ${SIEVE}\n"

fi

# Count the num­ber of unprocessed mes­sages.

printf '\e[0m- Check for mes­sages to refilter... '

MAIL_­FOLDER="${MAIL_­FOLDER}/.${FOLDER}"

if [ ! \( -d "${MAIL_­FOLDER}/cur" \) ] || [ ! \( -d "${MAIL_­FOLDER}/new" \) ]; then

  printf "\e[1;31m[error]\e[0m\n"

  (>&2 printf "  User inbox doesn't have a '${FOLDER}' folder.\n\n")

  exit 2;

fi

# Count the num­ber of files in the tar­get inbox folder.

FILES=`find "${MAIL_­FOLDER}/cur/" "${MAIL_­FOLDER}/new/" -type f -name '*' | wc -l`

if [ ${FILES} -eq 0 ]; then

  printf "\e[1;33m[warn­ing]\e[0m\n"

  (>&2 printf "  No mes­sages to process. Abort­ing.\n\n")

  exit 61;

fi

printf "\e[1;32m[ok]\e[0m\n"

printf "  Found ${FILES} mes­sages to process.\n\n"

echo "Run­ning ${FILES} mes­sages in ${FOLDER} through the Sieve fil­ter."

ion­ice -c2 -n7 sieve-fil­ter -e -W -C -u "${EMAIL_USER}" "${SIEVE}" "${FOLDER}"

Make the script executable:

chmod +x sieve-refilter.sh

Now we can run the script as administrator or vmail shell user:

sudo ./sieve-refilter.sh "user.name@exam­ple.com"

Conclusion

The Pigeonhole `sieve-filter` tool is great for re-running Sieve filter rules on a Dovecot IMAP inbox folder. We created a simple script to perform this operation for a single user.

How to run headless Chrome as systemd service

Mario Steinitz — Tue, 04 Oct 2022 23:35:04 +0000

How to run headless Chrome as systemd service

Introduction

We operate an application on a Linux based remote server. This application runs several tasks which require headless Chrome for generating formatted output from HTML pages (PDF documents, screenshots of web pages, ...). Each completion of such tasks includes starting up a headless Chrome instance, generating the output, and closing Chrome.

During high usage times, a lot of system resources are required for parallel working tasks. We wish to optimize the generation of documents via Chrome by using a single Chrome instance only that runs as system service and serves all tasks of our application . In order to ensure this Chrome instance doesn't use up too many resources in the long term, we also want it to be restarted on a regular base.

To avoid malicious JavaScripts on the processed HTML pages to harm our server, we also want to restrict Chrome access to the system as much as possible.

Proceedings

We create a dedicated user chrome, which can't login nor has a home folder. It serves as a non-elevated user to run the Chrome instance:

sudo adduser chrome --shell=/bin/false --no-cre­ate-home

We want to exchange files with the Chrome instance. Chrome will save the generated output files to an exchange folder, where our application can pick them up. This folder shall be the only writable folder for Chrome:

# Cre­ate a file exchange folder.

sudo mkdir /var/file­ex­change

# Set folder own­er­ship.

sudo chown chrome:chrome /var/file­ex­change

# Set appro­pri­ate file exchange folder per­mis­sions.

sudo chmod 775 /var/file­ex­change

For running our headless Chrome in the background, we create a system-wide systemd file /etc/systemd/system/chrome.service:

[Unit]

Descrip­tion=Head­less chrome

[Ser­vice]

Type=sim­ple

Restart=always

Run­timeMaxSec=1d

Dynam­i­cUser=true

User=chrome

Pro­tect­Sys­tem=strict

Read­WritePaths=/var/file­ex­change

Envi­ron­ment=HOME=/tmp

Envi­ron­ment=DIS­PLAY=:0

Exec­Start=/opt/chrome --head­less --no-first-run --no-default-browser-check --no-sand­box --dis­able-setuid-sand­box --dis­able-gpu --dis­able-dev-shm-usage --remote-debug­ging-port=9222 --win­dow-size="1920,1200" --force-device-scale-fac­tor=0.5

[Install]

Want­edBy=multi-user.tar­get

Key aspects of this service unit:

Type=simple: There's only one main process, which is started with ExecStart.
Restart=always: Restart the service in case of any failures or interruptions.
RuntimeMaxSec=1d: Limit the service to a one day runtime. The service will automatically be stopped and because of above Restart directive restarted.
DynamicUser=true: Hardens file and process access to the system. It implies private temporary folders and strict system protection.
User=chrome: The system user that will run the Chrome instance. We use our previously created non-elevated user, rather than a random user which DynamicUser=true would create without this directive.
ProtectSystem=strict: This option is redundant. It is implied already by DynamicUser=true. We kept as reminder of a very strict read-only file system.
ReadWritePaths=/var/fileexchange: Space separated whitelist for writable folders. Adding our previously created file exchange folder here ensures Chrome can write to this folder.
Environment=HOME=/tmp: Set the home directory to the private temp files path.
Environment=DISPLAY=:0: Explicitly use the first display by defining the DISPLAY environment variable.
ExecStart: The path to and start options for our headless Chrome instance. You may want to replace the path with your actual Chrome path and also adjust the Chrome startup flags according to your requirements.
WantedBy=multi-user.target: Start the service after all network services are available and when the system is ready for login.

Set proper file permissions for the systemd service file:

sudo chmod 644 /etc/sys­temd/sys­tem/chrome.ser­vice

Reload the unit files to make systemd know about the new service:

sudo sys­tem­ctl dae­mon-reload

Start the service and observe any error messages (e.g., when the path to Chrome isn't correct, the chosen remote debugging port already used, with a typo in Chrome startup flags, and so on):

sudo sys­tem­ctl start chrome.ser­vice

If no errors occurred and the service is running fine, enable it for automatic start on machine boot:

sudo sys­tem­ctl enable chrome.ser­vice

Conclusion

We successfully created a systemd service for headless Chrome that

restarts daily,
starts with a non-elevated user,
is mostly isolated from the remaining system, but
can create files for processed output in a specifically defined location.