KB20224071315

How to re-apply Sieve fil­ters to a Dove­cot IMAP inbox

using the Pigeon­hole sieve-fil­ter tool
Published on: 09.10.2022
Categories
Dovecot
Sieve Mail Filters
IMAP

Sieve scripts are a great tool to fil­ter incom­ing email mes­sages before they are stored in your mail­box. The Dove­cot IMAP server sup­ports Sieve scripts via the Pigeon­hole plu­gin. How­ever, those scripts are applied to your incom­ing emails before they are deliv­ered to your inbox.

Some­times, you may wish to fil­ter mes­sages that are already stored in your mail­box. For instance, when a bug in a Sieve script caused many mes­sages to be deliv­ered incor­rectly. As long as the Sieve IMAP fil­ter is exper­i­men­tal and its sup­port lim­ited, the `sieve-fil­ter` tool can come to your aid. This arti­cle explains how it is done.

The steps shown in this arti­cle are intended for being con­ducted on the com­mand line inter­face (CLI) of your mail server. If you don't have access to the CLI, please ask your mail servers' sys­tems admin­is­tra­tor, whether he can help you.

 

Introduction

Our Dove­cot IMAP email ser­vice user defined mail fil­ter rules using the Man­a­ge­sieve pro­to­col. His Sieve rules auto­mat­i­cally clas­sify the hun­dreds of emails flood­ing his inbox every day. They delete unwanted mes­sages, store oth­ers in dis­tinct fold­ers, set flags and pri­or­i­tize them for later read­ing. A typo in his sieve scripts caused the fil­ters to break. Emails were sent directly to his inbox where now thou­sands of unclas­si­fied newslet­ters and spam mails linger, mak­ing him unable to deter­mine the impor­tant com­mu­ni­ca­tion. We were asked to help him refilter the mes­sages using his fixed Sieve fil­ters.

 

Example environment

For the sake of this arti­cle, we use `user.name@exam­ple.com` as email inbox. We advice our IMAP user to cre­ate a sub­folder `Refilter` in his email inbox and move all unclas­si­fied emails into this folder.

Our exam­ple Dove­cot email server uses `/var/vmail/[domain]/[user]` as stor­age loca­tion. User-defined cus­tom Sieve scripts are located in a `/var/vmail/[domain]/[user]/.sieve` sym­bolic link that tar­gets the enabled Sieve fil­ter rules file. The inbox folder root is `INBOX`.

 

Proceedings

In order to re-apply Sieve rules on a Dove­cot IMAP server, newer ver­sions of Dove­cot Pigeon­hole come with the `sieve-fil­ter` tool. It takes the tar­get inbox, the sieve rules to apply, and the inbox folder to apply the rules to as argu­ments:

sieve-fil­ter -u "user.name@exam­ple.com" /var/vmail/exam­ple.com/user.name/.sieve INBOX.Refilter

 

By default, the `sieve-fil­ter` com­mand runs in sim­u­la­tion mode. To apply any changes, we have to explic­itly enable exe­cu­tion mode by adding the `-e` option, and allow write and delete access to the mail­box by adding the `-W` option. We also want the Sieve scripts to be recom­piled to apply any last changes by the user. So we add the `-C` option to the com­mand (a com­plete ref­er­ence of avail­able options can be found at the Pigeon­hole Dove­cot sieve-fil­ter doc­u­men­ta­tion page):

sieve-fil­ter -e -W -C -u "user.name@exam­ple.com" /var/vmail/exam­ple.com/user.name/.sieve INBOX.Refilter

 

Con­sid­er­ing that inboxes can grow rather large, and pro­cess­ing thou­sands of mes­sages at a time may put a heavy strain on our live mail server, we may want to run the fil­ter­ing with lower pri­or­ity, only when the sys­tem isn't too busy. There­fore, we addi­tion­ally use the `ion­ice` com­mand with `-c2` for best-effort mode and `-n7` for low­est pri­or­ity:

ion­ice -c2 -n7 sieve-fil­ter -e -W -C -u "user.name@exam­ple.com" /var/vmail/exam­ple.com/user.name/.sieve INBOX.Refilter

 

Wrap it into a Bash script

We don't always want to return to our knowl­edge base when hav­ing to help a mail user to refilter its mes­sages. There­fore, we cre­ated a sim­ple Bash script that just requires the mail user name as argu­ment and does the job for us.

It is using the `doveadm` user lookup com­mand to deter­mine the users' Sieve script loca­tion and mail­box fold­ers:

doveadm user "user.name@exam­ple.com"

Which, for exist­ing mail­box users, returns an answer like this:

field   value
uid     *****
gid     *****
home    /var/vmail/exam­ple.com/user.name
mail    maildir:/var/vmail/exam­ple.com/user.name/Maildir
quo­ta_rule      *:stor­age=0B
sieve   /var/vmail/exam­ple.com/user.name/.sieve

We are using the `maildir` infor­ma­tion to check for exist­ing mes­sages to process, as well as the `sieve` infor­ma­tion to find the user-defined Sieve rules.

 

And here is the script as ref­er­ence for our val­ued read­ers `sieve-refilter.sh`: 

#!/bin/bash
#------------------------------------------------------------------------------
# Reap­ply user-defined Sieve fil­ters to a Dove­cot mail­box folder.
#------------------------------------------------------------------------------
# REQUIRE­MENTS:
# awk, cat, doveadm, echo, exit, find, grep, ion­ice, printf, read, shift,
# sieve-fil­ter, tr, wc, non-POSIX sup­port for `< <()` syn­tax
#------------------------------------------------------------------------------
# USAGE:
# ./sieve-refilter.sh <user> [<folder>]
#
# Options:
#   <user>             The Dove­cot user to reap­ply Sieve fil­ters for. E.g.,
#                      "user.name@exam­ple.com".
#   <folder>           Optional: The mail­box folder to process.
#                      Defaults to 'INBOX.Refilter'.
#------------------------------------------------------------------------------
# Author: SHORELESS Limited <con­tact@shore­less.lim­ited>
# See https://shore­less.ltd/kb20224071315
#------------------------------------------------------------------------------

# User option.
EMAIL_USER="$1"
if [[ -z "${EMAIL_USER}" ]]; then
  printf "\n\e[1;31mAn argu­ment error occurred.\e[0m\n\n"
  (>&2 printf 'Argu­ment error: The user para­me­ter "<user>" is miss­ing.')
  printf "\n\nUsage: ./sieve-refilter.sh <user> [<folder>]\n\n"
  printf "Options:\n\n"
  printf "  <user>             The Dove­cot user to reap­ply Sieve fil­ters for. E.g.,\n"
  printf "                     \"user.name@exam­ple.com\"."
  printf "  <folder>           Optional: The mail­box folder to process.\n"
  printf "                     Defaults to 'INBOX.Refilter'.\n\n"
  exit 22
fi

FOLDER="$2"
if [[ -z "${FOLDER}" ]]; then
  FOLDER="INBOX.Refilter"
fi

# Exe­cutes a com­mand while writ­ing STD­OUT and STDERR to vari­ables.
#
# @param STD­OUT
#   The vari­able to write STD­OUT to.
# @param STDERR
#   The vari­able to write STDERR to.
# @param COM­MAND
#   The com­mand to run.
# @param [ARG1[ ARG2[ ...[ ARGN]]]]
#   Any addi­tional argu­ments for the com­mand to run.
#
# @return
#   The com­mand exit code.
#
# @see https://stack­over­flow.com/ques­tions/11027679/#answer-59592881
catch() {
  {
    IFS=$'\n' read -r -d '' "${1}";
    IFS=$'\n' read -r -d '' "${2}";
    (IFS=$'\n' read -r -d '' _ER­RNO_; return ${_ER­RNO_});
  } < <((printf '\0%s\0%d\0' "$(((({ shift 2; ${@}; echo "${?}" 1>&3-; } | tr -d '\0' 1>&4-) 4>&2- 2>&1- | tr -d '\0' 1>&4-) 3>&1- | exit "$(cat)") 4>&1-)" "${?}" 1>&2) 2>&1)
}

echo "Dove­cot Sieve refilter"
echo "----------------------"
echo "Check­ing pro­vided options and mail­box:"

# Check, whether the user exists in the local Dove­cot instance.
printf '\e[0m- Check­ing, whether the user exists... '
catch STD­OUT STDERR doveadm user "${EMAIL_USER}"
if [ ${?} -eq 0 ]; then
  printf "\e[1;32m[ok]\e[0m\n"
else
  printf "\e[1;31m[error]\e[0m\n"
  (>&2 printf '%s\n%s\n\n' "  User lookup failed." "  ${STDERR}")
  exit 22;
fi

# Deter­mine mail folder.
printf '\e[0m- Deter­mine mail folder... '
MAIL_­FOLDER=$(echo "${STD­OUT}" | grep '^mail' | awk '{print $2}')
MAIL_­FOLDER="${MAIL_­FOLDER#maildir:}"
if [[ -z "${MAIL_­FOLDER}" ]] || [ ! \( -d "${MAIL_­FOLDER}" \) ]; then
  printf "\e[1;31m[error]\e[0m\n"
  (>&2 printf '%s\n%s\n\n' "  Dove­cot Maildir could not be found." "  ${MAIL_­FOLDER}")
  exit 2;
else
  printf "\e[1;32m[ok]\e[0m\n"
  printf "  ${MAIL_­FOLDER}\n"
fi

# Deter­mine sieve script.
printf '\e[0m- Check for Sieve script... '
SIEVE=$(echo "${STD­OUT}" | grep '^sieve' | awk '{print $2}')
if [[ -z "${SIEVE}" ]] || [ ! \( -e "${SIEVE}" \) ] || [ ! \( -f "${SIEVE}" \) ] || [ ! \( -r "${SIEVE}" \) ] || [ ! \( -s "${SIEVE}" \) ]; then
  printf "\e[1;31m[error]\e[0m\n"
  (>&2 printf '%s\n%s\n\n' "  Sieve script does not exist or is empty." "  ${SIEVE}")
  exit 2;
else
  printf "\e[1;32m[ok]\e[0m\n"
  printf "  ${SIEVE}\n"
fi

# Count the num­ber of unprocessed mes­sages.
printf '\e[0m- Check for mes­sages to refilter... '
MAIL_­FOLDER="${MAIL_­FOLDER}/.${FOLDER}"
if [ ! \( -d "${MAIL_­FOLDER}/cur" \) ] || [ ! \( -d "${MAIL_­FOLDER}/new" \) ]; then
  printf "\e[1;31m[error]\e[0m\n"
  (>&2 printf "  User inbox doesn't have a '${FOLDER}' folder.\n\n")
  exit 2;
fi
# Count the num­ber of files in the tar­get inbox folder.
FILES=`find "${MAIL_­FOLDER}/cur/" "${MAIL_­FOLDER}/new/" -type f -name '*' | wc -l`
if [ ${FILES} -eq 0 ]; then
  printf "\e[1;33m[warn­ing]\e[0m\n"
  (>&2 printf "  No mes­sages to process. Abort­ing.\n\n")
  exit 61;
fi

printf "\e[1;32m[ok]\e[0m\n"
printf "  Found ${FILES} mes­sages to process.\n\n"

echo "Run­ning ${FILES} mes­sages in ${FOLDER} through the Sieve fil­ter."
ion­ice -c2 -n7 sieve-fil­ter -e -W -C -u "${EMAIL_USER}" "${SIEVE}" "${FOLDER}"

 

Make the script exe­cutable:

chmod +x sieve-refilter.sh

 

Now we can run the script as admin­is­tra­tor or vmail shell user:

sudo ./sieve-refilter.sh "user.name@exam­ple.com"

 

Conclusion

The Pigeon­hole `sieve-fil­ter` tool is great for re-run­ning Sieve fil­ter rules on a Dove­cot IMAP inbox folder. We cre­ated a sim­ple script to per­form this oper­a­tion for a sin­gle user.