Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 704940 - archives.gentoo.org: no mails since 2020-01-03 are listed on archives.gentoo.org
Summary: archives.gentoo.org: no mails since 2020-01-03 are listed on archives.gentoo.org
Status: RESOLVED FIXED
Alias: None
Product: Websites
Classification: Unclassified
Component: Archives (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Infrastructure
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-07 17:18 UTC by Franz Trischberger
Modified: 2021-09-27 18:21 UTC (History)
5 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Franz Trischberger 2020-01-07 17:18:51 UTC
Usually when I send a mail to gentoo-user it is visible on archives.gentoo.org instantly.
Now the last messages are from January 3rd. I am subscribed to gentoo-user and gentoo-dev and both lists had traffic since then. I think something is dead...
Comment 1 Alec Warner (RETIRED) archtester gentoo-dev Security 2020-01-15 02:50:25 UTC
So my brief glance:

We have the mails on the machine, but they are not being sorted into their associated lists. I'm basically reverse engineering how this thing works, let me drop some notes:

 - Mail gets delivered to the mailing list server as normal (lists.gentoo.org)
 - garchives receives a copy of each ML and stores it in a garchives homedir; 1 maildir per mailing list.
 - The garchives backend (on a different machine) rsyncs over the garchives maildirs to the garchives backend machine.
 - Every 10m we run a cronjob that examines the maildir archives, lookings for MLs with unprocessed messages, then kicks off the indexer to index the new messages.

My investigation has concluded that our setup for (1) has failed because it was consuming procmail-lib; a package recently removed from Gentoo. We then unmerged it and it caused procmail to fail.

This is not a data loss issue (mail was delivered to the main maildir, not a list specific maildir.) We need to produce a fix for new messages, then run a backlog job that processes mail from Jan 3 to $PRESENT.
Comment 2 Alec Warner (RETIRED) archtester gentoo-dev Security 2020-01-15 03:36:02 UTC
So in progress:

I've added procmail-lib to our infra overlay.
Waiting 1h for all that jazz to sync up.
Then merge procmail-lib on the box.

Then in theory archives will be fixed.

-A
Comment 3 Alec Warner (RETIRED) archtester gentoo-dev Security 2020-01-15 07:33:26 UTC
Ok the archives system has returned to service. We have a gap from Jan 3 to Jan 15 that we need to reprocess; I will handle that at a later date.

-A
Comment 4 Franz Trischberger 2020-01-28 04:53:28 UTC
Seems the system is down again.
Comment 5 Alec Warner (RETIRED) archtester gentoo-dev Security 2020-01-28 07:10:54 UTC
(In reply to Franz Trischberger from comment #4)
> Seems the system is down again.

What evidence leads you to that conclusion?

-A
Comment 6 Franz Trischberger 2020-01-28 07:21:35 UTC
(In reply to Alec Warner from comment #5)
> (In reply to Franz Trischberger from comment #4)
> > Seems the system is down again.
> 
> What evidence leads you to that conclusion?
> 
> -A

No updates since January 26th on gentoo-user, although plenty of mails arrived here. I even wrote one myself (with a different mail account than this one I use to login on b.g.o).
Since the fix mails were listed on archives.gentoo.org immediately, so I assume something is wrong, again.
Comment 7 Nuno Silva 2020-01-28 11:28:18 UTC
At least now I can see messages from January 27 and 28. Any chance whatever wasn't working is now working again?
Comment 8 Franz Trischberger 2020-01-29 05:47:19 UTC
Sorry, I was wrong, all seems to work fine.
For some reason the archive website now defaults to "threads" for me, previously it loaded just "messages" (did that feature come with a recent update?). The thread view shows the date (and sender) of the first message in the thread instead of the most recent one which made me believe something was wrong. Probably that can be changed?
Comment 9 John Helmert III archtester Gentoo Infrastructure gentoo-dev Security 2021-09-27 18:21:27 UTC
Seems to be working again