Re: [hypermail] dir option weirdness from kent landfield on 2003-04-09 (Hypermail Development List)

From: kent landfield <kent_at_hypermail.org_at_hypermail-project.org>
Date: Wed, 9 Apr 2003 18:08:48 -0500 (CDT)
Message-Id: <20030409230848.28F7E2701A8_at_dev.hypermail.org>

misiek_at_pld.ORG.PL (Arkadiusz Miskiewicz) writes:

>          For example, if you wished to have Hypermail archive files  by  year
>          and month, you might use
>
>                    dir = /lists/somelist/%y/%M
>
>          When  a  message  was  received  it  would  be  put into a directory
>          /lists/somelist/1998/Jun (if the message arrived in June  of  1998).
>          If  the message arrives and there is no storage directory, Hypermail
>          will automatically create it and store the message in the new direc-
>          tory.
>
>
>,,When  a  message  was  received'' - well right now that isn't the case.
>It should be rather ,,When a message was feeded to hypermail'' - at least for
>me hypermail doesn't use received field from mbox nor date field and
>just checks current system date which is bad because it seems that I
>can't create web archive using quite old mbox archive.

Peter C. McCluskey writes:
# This seems strange. Does anyone know why it is using the system date?
# The code that does this (in the dirpath function) is clearly designed to
# do that, but I can't think of a good reason for it. If I don't hear any
# objections within the next few days, I will change the code to use the
# received date.
#
# Note that there is a newer option called folder_by_date which uses
# the received date and produces results which are better for many purposes
# (for example, it creates links between the directories).

There was some initial discussion about this along time ago though I don't see it in the archives... If I remember right, the concerns were based on

trust of the Date: header generated on a foreign site, allowing an external person from messing up your archives by putting in bogus date information,
Use of received headers not always consistent between sites, certain received headers can be disabled locally for concealment of local information, multiple received headers due to hops, listmanager usage, etc..
Use of received headers not always consistent MTAs (postfix, sendmail, etc.).

My biggy was the first one as someone did just that to an archive I managed more than once. To eliminate all of the above, the local time on the system was used. If the man page is wrong, let's change it. Note however, it does not say the date in the Received header is used. But point taken, the wording can be confusing.

As for creating a "web archive using a quite old mbox archive..." I do it all the time with no problem. I have separate mboxes for the individual period, in my case by month and I use the script below to regenerate the entire archive or any section of it at will. A version something like it is in the archive directory of the distribution.

-- 
Kent Landfield             |  HYPERMAIL: http://www.hypermail.org/ 
Email: kent_at_hypermail.org  |  RFCS: http://www.faqs.org/rfcs/


#!/bin/sh
#


# set -x  # bug squisher

#


# The purpose of this script is to allow you to re-generate the

# hypermail archives at will.   What you need to do is have a Unix

# mailbox copy of the archive that you want to rebuild.  

#


# Paths to things on disk

#


# ARCHIVE_DIR = The base directory of the archive 

# MAILBOX_DIR = Mailbox version of the archive

# RDMSG       = Location of mbox2hypermail utility

#
ARCHIVE_DIR=/www/mail-archive
MAILBOX_DIR=${ARCHIVE_DIR}/mailbox
LISTNAME=hypermail
CONFIGFILE=/usr/local/lib/hypermail/hypermail.rc
HYPERMAIL=/usr/local/bin/hypermail

#


# Ownership of the archives

#


# Beware: Hypermail needs to be able to potentially 

#         write these directories. Know who your mailer

#         runs as.

#
OWNER=lists
GROUP=daemon

#


# Create_archive YYYY MON path-to-mailbox 

#


#      YYYY - 4 digit year

#      MON  - 3 Letter month abreviation

#      

create_archive()
{
    year=$1
    month=$2
    mailbox=$3

    if [ ! -d ${ARCHIVE_DIR}/$year ]; then
        mkdir ${ARCHIVE_DIR}/$year
    fi

    if [ ! -d ${ARCHIVE_DIR}/$year/$month ]; then
        mkdir ${ARCHIVE_DIR}/$year/$month
    else
        rm -r ${ARCHIVE_DIR}/$year/$month
        mkdir ${ARCHIVE_DIR}/$year/$month
    fi

    echo 
    echo "$HYPERMAIL -p -m $mailbox -x -c $CONFIGFILE -d ${ARCHIVE_DIR}/$year/$month -l 'Hypermail Development List'"

    cd ${ARCHIVE_DIR}/$year/$month
    $HYPERMAIL -p -m "$mailbox" -x -c $CONFIGFILE -d ${ARCHIVE_DIR}/$year/$month -l 'Hypermail Development List'

    chown -R ${OWNER}.${GROUP} ${ARCHIVE_DIR}/$year/$month
}

#########################


# Ye Olde Main

#########################


# 

# 1998 archives

#
create_archive 1998 Apr ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9804
create_archive 1998 May ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9805
create_archive 1998 Jun ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9806
create_archive 1998 Jul ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9807
create_archive 1998 Aug ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9808
create_archive 1998 Sep ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9809
create_archive 1998 Oct ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9810
create_archive 1998 Nov ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9811
create_archive 1998 Dec ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9812


# 

# 1998 archives

#
create_archive 1999 Jan ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9901
create_archive 1999 Feb ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9902
create_archive 1999 Mar ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9903
create_archive 1999 Apr ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9904
create_archive 1999 May ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9905
create_archive 1999 Jun ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9906
create_archive 1999 Jul ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9907
create_archive 1999 Aug ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9908
create_archive 1999 Sep ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9909
create_archive 1999 Oct ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9910
create_archive 1999 Nov ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9911
create_archive 1999 Dec ${ARCHIVE_DIR}/mailbox/${LISTNAME}.9912
#


# 2000 archives

#
create_archive 2000 Jan ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200001
create_archive 2000 Feb ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200002
create_archive 2000 Mar ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200003
create_archive 2000 Apr ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200004
create_archive 2000 May ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200005
create_archive 2000 Jun ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200006
create_archive 2000 Jul ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200007
create_archive 2000 Aug ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200008
create_archive 2000 Sep ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200009
create_archive 2000 Oct ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200010
create_archive 2000 Nov ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200011
create_archive 2000 Dec ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200012
#


# 2001 archives

#
create_archive 2001 Jan ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200101
create_archive 2001 Feb ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200102
create_archive 2001 Mar ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200103
create_archive 2001 Apr ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200104
create_archive 2001 May ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200105
create_archive 2001 Jun ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200106
create_archive 2001 Jul ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200107
create_archive 2001 Aug ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200108
create_archive 2001 Sep ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200109
create_archive 2001 Oct ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200110
create_archive 2001 Nov ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200111
create_archive 2001 Dec ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200112
#


# 2002 archives

#
create_archive 2002 Jan ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200201
create_archive 2002 Feb ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200202
create_archive 2002 Mar ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200203
create_archive 2002 Apr ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200204
create_archive 2002 May ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200205
create_archive 2002 Jun ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200206
create_archive 2002 Jul ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200207
create_archive 2002 Aug ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200208
create_archive 2002 Sep ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200209
create_archive 2002 Oct ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200210
create_archive 2002 Nov ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200211
create_archive 2002 Dec ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200212
#


# 2003 archives

#
create_archive 2003 Jan ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200301
create_archive 2003 Feb ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200302
create_archive 2003 Mar ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200303
create_archive 2003 Apr ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200304
#create_archive 2003 May ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200305
#create_archive 2003 Jun ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200306
#create_archive 2003 Jul ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200307
#create_archive 2003 Aug ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200308
#create_archive 2003 Sep ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200309
#create_archive 2003 Oct ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200310
#create_archive 2003 Nov ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200311
#create_archive 2003 Dec ${ARCHIVE_DIR}/mailbox/${LISTNAME}.200312
#


# C'ya

#
exit 0

Received on Thu 10 Apr 2003 01:08:48 AM GMT

This archive was generated by hypermail 2.3.0 : Sat 13 Mar 2010 03:46:12 AM GMT GMT