Re: Hypermail incremental mode

Daniel Stenberg wrote:

> J
>
> > 1. added archive locking
>
> This exists in v2. In the recent 2a18 you're now able to specify how long
> to wait for a lock before deliberatly breaking it.
>

How do multiple hypermail processes coordinate? Wait for archive to become unlocked or pass work onto the active process? For incremental mode this is the biggest problem area since the buildup of processes during heavy email sessions can bring a server to its knees. During testing I hit the hypermail archive with 100000 messages at 10 messages per second. The server (Sun Ultra1) still gets hit pretty hard with all the forking but it still keeps going. Of course the other fixes are necessary to make this happen.

>
> > 2. added an optional file that contained the summary info of the archive
> > to avoid opening all the archive files during an update. [this removed a
> > big performance hit on large archives]
>
> This does not exist, and I have not added any such on purpose. I believe
> this is left for discussions whether we should add one or not. I'm in
> favour of a solution where such a file is used if existent, but where the
> HTML files are scanned and that info is used if the "cache" is for some
> reason not there.

This is how my implementation worked. The cache file would be rebuilt if it didn't exist. The nice thing about a cache file is that the time to add and index each individual message is the same no matter what size the archive is. Opening 10000 or more files for each message is a big performance problem.

> > 7. eliminated the on-the-fly sorting of messages by using qsort on the
> > tables. [big performance enhancement for large archives]
>
> I surely wouldn't mind if you added that functionality to the 2a18
> package and sent me a patch for it! ;-)
>

This is one of the biggest problems with large archives. I define large archives as those larger than 10000 messages. When I was testing my performance fixes I created an archive with 1000000 messages.

>
> > 8. fixed up the date routines which seemed to broken in a number of ways
>
> The date routines are pretty much modified for 2a18 too. Paul Haldane has
> been friendly enough to shoulder the date stuff for now so I'm hoping
> you can coordinate your ideas with his new stuff.
>
> I'm awaiting some further fixes in that area.
>

I'll take a look.

John Received on Mon 12 Apr 1999 07:53:52 PM GMT