Hi, I've modified the -r2 ebuild somewhat to imho improve a few minor things. Specifically: 1. Optional vhost support via vhosts flag. It's based in part on a combination of the cpanel patch and another that I found on the mailman mailing list (lost the url, post #41830), which in turn was based on a patch in sf: http://sourceforge.net/tracker/index.php?func=detail&aid=943827&group_id=103&atid=300103 - but with a few extra fixes. 2. Alternative support for /icons/, essentially create a new /mailman-icons/ alias and patch mailman to try and get it's icons from there instead. This negates the need for a new user to copy files around on his system, creating orphan files. There may be other mods too, but those look essentially like what I changed. I'll attached now. Reproducible: Always Steps to Reproduce:
Created attachment 142602 [details] mailman-2.1.9-r3.ebuild
Created attachment 142603 [details] 50_mailman.conf
Created attachment 142604 [details] README.gentoo-r3 I removed the copy instructions for /icons/ and added a section on exim handling.
Created attachment 142606 [details, diff] mailman-2.1.9-icons.patch
Created attachment 142607 [details, diff] mailman-2.1.9-vhosts.patch Custom, I'd like to be notified if any modifications to this file is made at any given point - should I simply add a header at the top of the file (ignored by patch)?
Created attachment 142608 [details] net-mail.mailman-2.1.9-r3.tgz Archive that can be extracted into /usr/portage (or /usr/local/portage) to get everything you need to merge this.
Created attachment 142634 [details] mailman-2.1.9-r3.ebuild Fixed a small error (dosed doesn't seem to understand regular expressions, or extended regular expressions).
Created attachment 142635 [details] net-mail.mailman-2.1.9-r3.tgz
Created attachment 142925 [details] README.gentoo-r3 Fix the sample config for exim, and mention the page for why the double colon.
Created attachment 142927 [details, diff] mailman-2.1.9-vhosts.patch fixed whitespace bug in the mailmanctl startup file.
Created attachment 142929 [details] mailman-2.1.9-r3.ebuild This one is going to be controversial. But I don't see much other choice, except perhaps adding apache to the mailman group: --- net-mail/mailman/mailman-2.1.9-r3.ebuild (revision 22) +++ net-mail/mailman/mailman-2.1.9-r3.ebuild (revision 24) @@ -102,7 +102,7 @@ chown -R ${MAILUSR}:${MAILGRP} "${D}/${VAR_PREFIX}" "${D}/${INSTALLDIR}" "${D}"/etc/mailman/* chmod 2775 "${D}/${INSTALLDIR}" "${D}/${INSTALLDIR}"/templates/* \ "${D}/${INSTALLDIR}"/messages/* "${D}/${VAR_PREFIX}" "${D}/${VAR_PREFIX}"/{logs,lists,spam,locks,archives/public} - chmod 2750 "${D}/${VAR_PREFIX}/archives/private" + chmod 2771 "${D}/${VAR_PREFIX}/archives/private" chmod 2770 "${D}/${VAR_PREFIX}/qfiles" chmod 2755 "${D}/${INSTALLDIR}"/cgi-bin/* "${D}/${INSTALLDIR}/mail/mailman"
Created attachment 142931 [details] net-mail.mailman-2.1.9-r3.tgz And an updated archive containing everything.
Hi, Thanks for updating mailman, I think some of your changes are great and well needed! :-) However, one thing I would have liked is a little notification somewhere that upgrading from 2.1.9 to 2.1.9-r3+ would totally break your current list setup. Please don't get me wrong here. I think the changes are fantastic and required. However, we need to know when changes that impact an existing and functional setup. Perhaps writing a "Mailman Upgrade" howto would be an idea? This way we can all fix our current setups that are now broken without having to resort to downgrading to 2.1.9 as I have done. Many thanks for your hard work, Rich
> This one is going to be controversial. But I don't see much other choice, > except perhaps adding apache to the mailman group: Isn't adding Apache to the mailman group the obvious and more secure choice!? I don't know now if it was an ebuild decision or a personal one, but my archives directory is owned by apache:mailman. Perhaps we could do that and change the user ownership to be that of the CGIID?
Could the ebuild be modified to detect the sendmail USE flag and submit the corresponding mail_gid flag? I know the earlier ebuild had conditional code for this but the sendmail entry was wrong which caused all my upgrades to break. Maybe the old code should be fixed and put back in. The current ebuild will still break my system but now it is just a bit easier to fix. If the code is not restored could you modify the ebuild warning message to give an example like this: ewarn "Configuration changes can be done on the command line." ewarn "For example:" ewarn "MAILMAN_MAILGID=12 emerge mailman" Thanks.
> However, one thing I would have liked is a little notification somewhere that > upgrading from 2.1.9 to 2.1.9-r3+ would totally break your current list setup. I can easily add that. So far I'm aware of the following things that break: * /usr/local/mailman/{lists,archives} needs to be moved, as well as archives. * mailman's home directory needs to be updated * crontab needs to be re-imported. * You probably want to copy your mm_cfg.py file to /etc/mailman/ I've added stuff to the ewarn to this effect. Anything else that I'm not aware of? (My install was a clean one so I didn't have most of these issues, I just picked them up from repeated re-merging). > Perhaps writing a "Mailman Upgrade" howto would be an idea? This way we > can all fix our current setups that are now broken without having to > resort to downgrading to 2.1.9 as I have done. No problem. I don't have official access anywhere but I can put something on my personal site, but I reckon that isn't where it belongs, Wiki perhaps? Also, whilst I'm perfectly capable of doing it I don't have an older list setup to migrate, so there may be issues I'm not aware of. I'd prefer to rather assist someone (over IRC perhaps) that needs to do the migration, and then co-auth the howto. I reckon that should result in a higher quality document for others to follow. > Isn't adding Apache to the mailman group the obvious and more secure choice!? > > I don't know now if it was an ebuild decision or a personal one, but my > archives directory is owned by apache:mailman. I actually like your solution with setting it to apache:mailman. This is simply because I don't want to give all the other permissions that goes with being added to the mailman group (which is why I opted for the chmod) Ok, I've just added a MAILMAN_CGIUID variable. I've also changed the CGI??? variables to default to apache instead of the number 81 (./configure can handle that), and then to chown the archives directory (base only) to $CGIUID:$MAILGRP. I've reverted the perms to 2770 (I first though about using 2570 but then realised since it's sgid anyway it makes no difference). > Could the ebuild be modified to detect the sendmail USE flag and submit the > corresponding mail_gid flag? Is there no way to tell sendmail to deliver as the user mailman:mailman? If not you can already use MAILMAN_MAILUSR and MAILMAN_MAILGRP variables to change this behaviour. Something like: MAILMAN_MAILUSR=mail MAILMAN_MAILGRP=mail emerge -av mailman However, I'd prefer to not have to put that from the command line, however, I'm not sure at all how to configure this in portage, I reckon adding lines like: MAILMAN_MAILUSR=mail MAILMAN_MAILGRP=mail _might_ work, but I haven't tested this. Also, from what I can tell, make _very_ sure that this user and group actually exists _before_ merging mailman or they will be created with uid and gid values that might be unexpected. I can probably pick up on the sendmail USE flag and _default_ to using mail:mail as default values. I'd however prefer to not do this as (as far as my understanding goes) those USE flag based decision has been removed for a reason.
Created attachment 142977 [details, diff] mailman-2.1.9-vhosts.patch Alter bin/list_lists to give the names as would be required by other tools in order to find the list. Eg, instead of just "mailman", give "mailman-lists.uls.co.za".
Created attachment 142979 [details] mailman-2.1.9-r3.ebuild * Add MAILMAN_CGIUID config variable. * Change MAILMAN_CGIGID variable to default to apache. * Add extra info regarding migration. This probably belongs in a Gentoo howto. * Install archives/private folder with ${CGIUID}:{MAILGRP} ownership * Install archives/private with 2770 perms (revert previous modification)
Created attachment 142980 [details] net-mail.mailman-2.1.9-r3.tgz
Please MASK this update! It made mailman stop working on a productive system and even after hours I'm still trying to make it work properly again - to no avail. There's absolutely no reason why an update like this should slip into a regular 'emerge -uD world' if it breaks things that badly and that unnecessarily like this one. So, either keep full backwards compatibility of binaries and configuration, even if it's a suboptimal setup OR make sure the update process is smooth, well documented and automated wherever possible. I'm sorry for being that explicit, but this is really a total NO GO!!
Chill man. This is NOT yet in the mainline tree. It's based on the -r2 that was in the tree at the time I wrote these, and I probably need to go check out what changed between -r2 and -r3 in the tree and incorporate that into these ebuilds as a suggested -r4. Please do check the comments if you have trouble migrating, there is also instructions in the ebuild on how to migrate from older installations, as well as in the README files in /usr/share/doc/mailman-??? - please do check that out. Whilst I'm not the maintainer of the mailman ebuilds in the tree, I am willing to help you out if you have trouble, because I'd really like to see this move into mainline ASAP, and from the feedback I've been receiving (and from what I can see in this thread) so are a lot of other people. Please do rather work _with_ us, and explain exactly what broke instead of just bitching (which will get you nowhere). PS: The update to -r3 looks like a security update wrt XSS, so if you were running some variation of 2.1.9 then I highly suggest you do work through this and upgrade, and during the process we could even write the suggested "howto". You're welcome to contact me directly for assistance, but then you're going to need to be willing to help us to help you, and not simply "I'm sorry for being that explicit, but this is really a total NO GO!!". We're all looking to move forward and to continually improve the system for everyone. That unfortunately means big changes from time to time that cannot be fully automated. The warnings in the ebuild is rather explicit about requiring intervention. However, I repeat, this -r3 ebuild is NOT the one in the tree, this will hopefully become -r4.
Created attachment 145019 [details, diff] mailman-2.1.9-fix-XSS.patch This files comes from -r3, placed here just to have it here explicitly in case somebody is downloading into an overlay.
Created attachment 145021 [details] mailman-2.1.9-r4.ebuild Updated to -r4 in lue of the -r3 in mainline tree. Imported the security fix from -r3 in mainline. Change the vhosts USE flag to mailvhosts because vhosts on my amd64 is masked. Hope this is ok, if not, please advise. And then specifically, please note these notes that was already in the ebuild: * Messages for package net-mail/mailman-2.1.9-r4: * * Please read /usr/share/doc/mailman-2.1.9-r4/README.gentoo.bz2 for additional * Setup information, mailman will NOT run unless you follow * those instructions! * * An example Mailman configuration file for Apache has been installed into: * /50_mailman.conf * * To enable, you will need to add "-D MAILMAN" to * /etc/conf.d/apache2. * * Default-Configuration has changed deeply in 2.1.9-r2. You can configure * mailman with the following variables: * MAILMAN_PREFIX (default: /usr/lib64/mailman) * MAILMAN_VAR_PREFIX (default: /var/lib/mailman) * MAILMAN_CGIUID (default: apache) * MAILMAN_CGIGID (default: apache) * MAILMAN_CGIEXT (default: empty) * MAILMAN_MAILUSR (default: mailman) * MAILMAN_MAILUID (default: 280) * MAILMAN_MAILGRP (default: mailman) * MAILMAN_MAILGID (default: 280) * * Config file is now symlinked in /etc/mailman, so etc-update works. * * If you're upgrading from below 2.1.9-r2 or changed MAILMAN_PREFIX, you * NEED to make a few manual updates to your system: * * 1. Update your mailman users's home directory: usermod -d /usr/lib64/mailman mailman * 2. Re-import the crontab: su - mailman -c 'crontab cron/crontab.in' * 3. Copy your old mm_cfg.py file to /etc/mailman/mm_cfg.py * * Additionally if you've modified MAILMAN_VAR_PREFIX (or upgraded from * a pre 2.1.9-r2 installation), you should move your old lists/ and * archives/ directory to the new location, ensuring that the * permissions is correct. See bug #208789 for a discussion. * * The mailvhosts use flag is experimental and not supported upstream. * Instead contact Jaco Kroon <jaco@uls.co.za> if you need support.
First of all, I'd like to emphasize that I'm not working against anything Gentoo at all (I am a proud long-time user and used to contribute portage code and ebuilds in Gentoo's early days myself. Unfortunately I lacked the time for that for quite while, but I hope this will change in the near future again). So, like all of us, it's in my very best interest to help Gentoo prosper. Very seldomn, that regrettably means pulling the emergency handle as an immediate measure to hopefully prevent other Gentoo users from running into the same troubles. Having a fix at hand would have been better, sure, but at the point of my comment all I knew was that something with the mailman update has obviously gone wrong. The least I could do was issuing a warning/feedback as soon as possible. My comment applied to mailman-2.1.9-r3.ebuild like it is currently in mainline. I erroneously assumed it would be the same one as the mailman-2.1.9-r3.ebuild discussed here, due to identical ebuild names. My point however, stays the same: mailman-2.1.9-r3 (from mainline) made my productive mailman service stop running and thus caused an unplanned downtime for hours - until I found the reasons for the problem and finally a solution. (Thanks for offering your help, Jaco, I appreciate this, though I found a solution myself after a while). The thing that disturbs me about this problem is that I don't see a compulsory reason for it to be a problem at all. I'm talking about a) the location of the mailman directory containing the lists/archive/data etc. being changed without caring about moving already existing lists to that new location and other backwards-incompatible changes and b) without appropriate information of the user, so he could at least do it himself within reasonable time. Here are my recommendations for Gentoo's mailman maintainer (Hanno) and Gentoo maintainers in general: Regarding a): Major changes like this should be strictly restricted to major Gentoo distribution release upgrades, particularly if they silently break things and backwards compatibility and require major user intervention. It's at major Gentoo distro upgrades where you can expect users (i.e. sysadmins) to be appreciative for the need of (major) manual intervention during the upgrade. You can't expect people to anticipate and appreciate such a thing when all they do is a minor package update. So please make sure such ebuilds are masked appropriately so they don't slip in accidentally. Backwards compatibility in general should have a higher priority in Gentoo, I can't emphasize this enough. Particularly for things like mailman which is clearly software used in a server environment, likely being mission critical for many users. Backwards compatibility, service availability and reliability are crucial for having success in a server environment, and we all want Gentoo to be successful in a server environment, too. Still, even for major changes in updates, the update process should be made as smooth as possible, preferably not requiring manual intervention by admins. That may require 5 hours (or less or more) of additional effort for the ebuild maintainer, but that's still preferable to requiring 1 hour of additional effort for each and every of many thousands of Gentoo users/admins. In mailman's case, please consider moving already existing lists, archives, data etc. to the new location and migrating/adjusting the configuration automatically. Regarding b): If manual intervention can not be avoided at all, make sure the user/admin gets appropriately informed about relevant changes (and their consequences) and required steps to make the transition as smooth and quick as possible. In the case of mailman, the information Jaco added as postinst ewarn messages to -r4 is very important for users/admins upgrading from older versions and should have been there in -r3 (mainline) already. I hope you take these suggestions as suggestions for improvement and not as an offense. I really don't want to offense or blame anybody as I'm fully aware of your all efforts and the fact that it's all voluntary work (that I highly appreciate). Thank you Dani
(In reply to comment #24) > My comment applied to mailman-2.1.9-r3.ebuild like it is currently in mainline. > I erroneously assumed it would be the same one as the mailman-2.1.9-r3.ebuild > discussed here, due to identical ebuild names. No problem. I did some further investigation for you, -r2 already contained these changes but was ~x86, presumably due to these changes. -r3 was security release which then unmasked those changes which caught you off guard. > The thing that disturbs me about this problem is that I don't see a compulsory > reason for it to be a problem at all. I'm talking about a) the location of the > mailman directory containing the lists/archive/data etc. being changed without > caring about moving already existing lists to that new location and other > backwards-incompatible changes and b) without appropriate information of the > user, so he could at least do it himself within reasonable time. Would a mechanism that prevents merging in /usr/local/mailman still exists and the MAILMAN_* variables NOT pointing there satisfy you? In other words, refuse to merge if the lists still exist in the old location. Personally I don't like this as it means I've got downtime during the merge process whereas I'd rather have a couple of seconds downtime after merging. > Here are my recommendations for Gentoo's mailman maintainer (Hanno) and Gentoo > maintainers in general: > > Regarding a): Major changes like this should be strictly restricted to major > Gentoo distribution release upgrades, particularly if they silently break Unlike other distributions I don't really think Gentoo has the concept of "major Gentoo distribution release upgrades". I've still got systems coming from the days of 1.4 :). Other than a Laptop I've just installed from a 2007.0 LiveCD the majority of my other systems are from 2006.0. Once installed I don't ever re-install (One of the major reasons why I like Gentoo). > things and backwards compatibility and require major user intervention. It's at > major Gentoo distro upgrades where you can expect users (i.e. sysadmins) to be > appreciative for the need of (major) manual intervention during the upgrade. > You can't expect people to anticipate and appreciate such a thing when all they > do is a minor package update. So please make sure such ebuilds are masked > appropriately so they don't slip in accidentally. > Backwards compatibility in general should have a higher priority in Gentoo, To an extent I have to agree with this. > I can't emphasize this enough. Particularly for things like mailman which is > clearly software used in a server environment, likely being mission critical > for many users. Backwards compatibility, service availability and reliability > are crucial for having success in a server environment, and we all want Gentoo > to be successful in a server environment, too. Another idea just popped into my head ... if we set the MAILMAN_HOME vars to /usr/local/mailman by default if this location already exists (or rather, extract the MAILMAN_HOME from getent passwd mailman) then we pretty much become backwards compatible, even though new installs will use the considerably more sane defaults. > In mailman's case, please consider moving already existing lists, archives, > data etc. to the new location and migrating/adjusting the configuration > automatically. pkg_postinst? I'll take a look and see what can be done. > Regarding b): If manual intervention can not be avoided at all, make sure the > user/admin gets appropriately informed about relevant changes (and their > consequences) and required steps to make the transition as smooth and quick as > possible. > In the case of mailman, the information Jaco added as postinst ewarn messages > to -r4 is very important for users/admins upgrading from older versions and > should have been there in -r3 (mainline) already. Those warnings were (for the most part) already in -r2. > I hope you take these suggestions as suggestions for improvement and not as an > offense. I really don't want to offense or blame anybody as I'm fully aware of > your all efforts and the fact that it's all voluntary work (that I highly > appreciate). No offence taken from my side. I've been plenty hacked off in the past and fully understand the frustration of a production system "breaking for no reason at all". J
Jaco, sorry for the delay. Regarding your question: Refusing to merge in the conditions you mentioned would IMHO be an acceptable alternative indeed. Personally, I'd prefer this compared to ending up with a silently broken system. Usually, if things don't merge properly, I try to find and fix the problem (so the ebuild could say sth like "either set your MAILMAN_* vars to point to the old locations abc, then update again OR stop mailman, move the old lists to the new locations xyz, emerge update, start mailman again" or whatever is appropriate). I'm not up-to-date regarding portage's current features. Maybe there's a feature to not abort the whole emerge process but rather just skip a particular update, issuing a warning message? Like that, other updates would still be merged. If there isn't any, the problem should probably be handled the same way as in similar cases in Gentoo. ATM, I don't have any similar example at hand though. Dani P.S. Alternatively, as another workaround, one could perhaps just make the new location a symlink pointing to the old location if there are any old lists. I'm not sure whether this would work, however.
In my opinion it should refuse to update like net-nds/openldap or dev-db/postgresql-8.3.1 does when you have data already in the directories. Juan.
Version bump - 2.1.10rc1 was just announced... Any chance for a new ebuild for this? It has two new features I'd really like to take advantage of (umbrella lists, and the ability to specify a list in any of the 'accept_these_nonmembers_* fields)... Thanks!!!
2.1.10 was just released... Any chance for an ebuild for it? "If I could, I would..."
Created attachment 151713 [details] mailman-2.1.10.ebuild Updated mailman ebuild for mailman 2.1.10, the updated vhosts patch will be uploaded in a second. This ebuild contains three changes: Firstly, a check for whether /usr/local/mailman exists, and if it does and VAR_PREFIX doesn't point to it, refuse to merge. + if test -d "${ROOT}/usr/local/mailman" \ + && [ "${MAILMAN_VAR_PREFIX}" != "/usr/local/mailman" ] \ + && [ ! "$MAILMAN_USE_THE_FORCE" ]; then + eerror "You have mailman var data /usr/local/mailman, however" + eerror "MAILMAN_VAR_PREFIX is not pointing there." + eerror "The layout structure of Mailman has changed, please see" + eerror "Bug #208789 for an explanation." + eerror "If you'd like to proceed with the merge in any case," + eerror "do so with MAILMAN_USE_THE_FORCE=1 emerge -av mailman." + + die "Invalid state for Mailman to merge from." + fi Secondly the XSS patch is no longer part of the patch set as it's now mainline. And thirdly, and updated vhosts patch which I'll attach in a second.
Created attachment 151714 [details, diff] mailman-2.1.10-vhosts.patch Minor changes from the old patch to make it apply again. The code paths has not been tested yet. So if you're needing this on a production system - PLEASE TEST IT ON A DEVELOPMENT BOX FIRST.
Created attachment 151715 [details] net-mail.mailman-2.1.10.tgz
*** Bug 217779 has been marked as a duplicate of this bug. ***
I've added 2.1.11 with most of your changes. I left out vhost support, but I intend to make a -r1 version soon with some additional features (beside vhost also ssls)
