Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 245984

Summary: app-misc/ca-certificates ebuild should recommend `update-ca-certificates --fresh'
Product: Gentoo Linux Reporter: Lindsay Haisley <fmouse-gentoo>
Component: New packagesAssignee: Gentoo's Team for Core System packages <base-system>
Status: RESOLVED FIXED    
Severity: enhancement    
Priority: High    
Version: unspecified   
Hardware: All   
OS: Linux   
See Also: https://bugs.gentoo.org/show_bug.cgi?id=475352
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on: 475352    
Bug Blocks:    
Attachments: Patch to update-ca-certificates to allow optional removal of broken symlinks in /etc/ssl/certs

Description Lindsay Haisley 2008-11-07 18:02:46 UTC
Updating app-misc/ca-certificates frequently results in broken symlinks in /etc/ssl/certs.  These broken symlinks will cause any application which requires SSL certificates to fail, often with implicit errors and error message that require a bit of research.  This has happened to me several times, and I always seem to forget to fix the links until one of my customers complains that STARTTLS isn't working, or some such.

The ebuild for ca-certificates senses such broken symlinks and reports them in output, listing them and suggesting that they be removed.  Instead of simply saying "You MUST remove the above broken symlinks" it would be much more helpful to suggest the use of update-ca-certificates --fresh which batch-removes all the symlinks and rebuilds them.  Many ebuilds suggest the use of revdep-rebuild or python-updater.  Suggesting the use, by name, of a gentoo utility to fix a problem sensed during an emerge makes good sense.
Comment 1 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2008-11-07 21:11:29 UTC
You failed to read the comment block in the ebuild.

--fresh is overzealous in removing files, and wipes out any custom certs that the user has in place.

I discussed this in another bug previously, that if somebody really wanted to make an impact, we need to have a file that tracked every cert path that had ever been in the ca-certificate package, and consider ONLY those symlinks for deletion when they were broken. I'll write the list if somebody else writes the patch for update-ca-certificates.
Comment 2 Lindsay Haisley 2008-11-07 21:43:18 UTC
Robin, this is rather easily done.  See the google cache at http://74.125.45.104/search?q=cache:s0OfJ-X-020J:www.openldap.org/lists/openldap-bugs/200808/msg00089.html+%22find+/etc/ssl/certs.  (openldap.org's website isn't up right now).

find /etc/ssl/certs -type l ! -xtype f ! -xtype d -ok rm -f {} \;

will rather surgically remove _only_ those symlinks that are broken.  This could be done either in the ca-certificates ebuild or in update-ca-certificates.

As far as this being resolved upstream, it doesn't appear to be resolved at all.  I've tripped over this particular rock several times.
Comment 3 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2008-11-07 21:56:23 UTC
I specifically said to only remove only the broken symlinks that were installed by the ca-certificates package. Not all of them. It should NOT touch any user created files or symlinks, regardless of them being broken or not.

And really, this is duplicate, just like I noted of the previous report, that find invocation sucks, and there's a much better one in the ebuild already.
# find -L "${ROOT}"etc/ssl/certs/ -type l
(add "-exec /bin/rm -f \{} \; if you want).

Comment 4 Lindsay Haisley 2008-11-07 22:20:32 UTC
A broken symlink is a broken symlink here, and will cause any SSL-dependent application to fail implicitly if it trips on it.  You're right in that it's not the responsibility of the ca-certificates ebuild to fix broken symlinks (or anything!) if the problem wasn't the result of the action of the ebuild.

The proper place for this is in the update-ca-certificates script which needs an option to remove all broken symlinks in /etc/ssl/certs, regardless of their origin.  I'm no expert on find, and my time is rather limited right now, but the invocation I quoted fixed the problem here on 2 different servers.  I'm putting it into update-ca-certificates and will submit a patch with a different bug report.

Thanks
Comment 5 Lindsay Haisley 2008-11-08 00:36:57 UTC
(In reply to comment #1) 
> I discussed this in another bug previously, that if somebody really wanted to
> make an impact, we need to have a file that tracked every cert path that had
> ever been in the ca-certificate package, and consider ONLY those symlinks for
> deletion when they were broken.

NO.

> I'll write the list if somebody else writes the
> patch for update-ca-certificates.

Robin, this is overkill!  ANY broken symlink in /etc/ssl/certs will cause SSL processing to fail.  Yes?  This list doesn't belong in update-ca-certificates.  The maintenance program should, or should be able to, along with whatever else it does, remove ALL broken symlinks in this dir, whether they were installed by the ca-certificates package or created by accident by the user.  Think about it.
Comment 6 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2008-11-08 00:49:26 UTC
The real problem you have is that the SSL programs fail when they shouldn't because of broken symlinks. It was reported to upstream openldap, and they fixed it in the 2.4.x series. I believe (but I'm not 100% certain) that Apache fixed it some years ago.

OpenSSL shouldn't break just because it can't read a file. I do have a valid use-case for the certificate stuff anyway, and that's having a symlink that points to storage that isn't always present (for security reasons). Removing the apparently broken symlink would then break the application in question.
Comment 7 Lindsay Haisley 2008-11-08 07:41:21 UTC
Good point.  The error I saw was a rather obtuse error/failure message from a customer out of Eudora about a "negotiated exchange key with a length of -1".  I tracked it down to the failure of POP3 over SSL when connecting with one of my courier installations (Courier pop3d).  Courier was also failing with STARTTLS in SMTP.  The openldap reference I posted was just one I found with a suggestion for removing the broken symlinks.

So I have 2 questions here.  First, where does the responsibility for this problem lie?  If it's not primarily with the existence of the broken links, is the isssue that application authors are misusing the openssl API, and if so is there a deficiciency in the documentation?  This is way outside of my area of expertise.

Second, given the practical necessity of keeping things running, would it not be advisable to have an option in update-ca-certificates to take down broken symlinks?  Probably a lot of us have run into this problem.  It's one thing to get upstream maintainers to modify their code to resolve the basic problem.  It's quite another to have customers calling and emailing because application x, for which they're paying, isn't running as expected, and they want it fixed NOW.  So I have to go into /etc/ssl/certs and manually rm any broken symlinks one by one, or google for a batch solution.  I quite understand your case re. symlinks to intermittently present storage, and while this case should be ALLOWED, it's probably far more likely that an SSL-dependent app will fail because of this.

I'm going to post my mods to update-ca-certificates in a separate bug report.  I'm not a world-class shell programmer but the mod will do the job, which is what counts.
Comment 8 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2008-11-08 07:49:44 UTC
The request I made for a patch isn't complex, just run the find command for broken symlinks and rather than blindly deleting them, also check them against a static list before deleting.
Comment 9 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2008-11-08 07:54:49 UTC
Sorry, hit commit too early.

The OpenLDAP one was filed here, then upstream OpenLDAP also fixed the actual bug in their app. I'd say that courier-pop3d probably needs to be fixed by the sounds of it.

- Apps shouldn't fail with broken links and CADirectory style directives.
- OpenSSL should maybe ignore the broken symlinks in the first place (or check them before using).
- more than 99% of the broken symlinks are there because of brokenness rather than the special case I mentioned.

The problem is why we presently print out a nice warning at the end of every ca-certificates emerge. If you critically depend on services (for your own use or paying customers), then reading the output of emerge is strongly in your own best interest. Even if you filter it down to only the warnings and errors, it's not a lot to read - and portage can email it to you quite fine.
Comment 10 Lindsay Haisley 2008-11-08 08:09:46 UTC
(In reply to comment #8)
> The request I made for a patch isn't complex, just run the find command for
> broken symlinks and rather than blindly deleting them, also check them against
> a static list before deleting.

OK, I'm posting a proposed patch to update-ca-certifiates here for your review.  This patch does "blindly delete" the symlinks, but does so _optionally_.  As you point out, probably 99% of the time the special case you validly allow doesn't apply, and there's no easy way to clean up the directory.  Shouldn't a more focused, list-based cleanup of broken symlinks be done within the context of the ca-certificates ebuild?

Comment 11 Lindsay Haisley 2008-11-08 08:13:53 UTC
Created attachment 171079 [details, diff]
Patch to update-ca-certificates to allow optional removal of broken symlinks in /etc/ssl/certs

This patch introdudes an additional option, --fix-broken, or just -b, to update-ca-certificates to batch-remove broken symlinks in /etc/ssl/certs.
Comment 12 Lindsay Haisley 2008-11-08 08:25:04 UTC
(In reply to comment #9)
> The problem is why we presently print out a nice warning at the end of every
> ca-certificates emerge. If you critically depend on services (for your own use
> or paying customers), then reading the output of emerge is strongly in your own
> best interest. Even if you filter it down to only the warnings and errors, it's
> not a lot to read - and portage can email it to you quite fine.

Keeping up with the output of ebuilds isn't trivial, which is why I worked on the enotice program a few years ago which a number of gentoo users picked up on and used.  Things have improved since then but it's still awkward to keep up with it all, and my original point in filing this bug was that the package, and the ebuild output offers no way or suggestion to batch-remove broken symlinks in the 99% chance that they're 100% problematic.  We're left doing it manually.  And yes, I rather religiously try to keep up with ebuild output on important packages.  I didn't just fall off the turnip truck (I'm 67 years old) and people do actually _pay_ me to do system administration sometimes ;-)
Comment 13 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2008-11-08 09:55:20 UTC
I stuck an extra bit of ewarn into the ebuilds for the moment.

IIRC there is way with PORTAGE_ELOG to have it log the info+log levels, but email the warning+error levels. That makes the review/tracking much easier, esp when you manage 100+ heterogeneous boxes like I do.

For the patch you posted, it's a really small jump to check against a file.
(this is rough puesdo-code, it's late now, past bedtime).
BAD_LIST="$(find ...)"
for i in $BAD_LIST ; do
  echo "Bad symlink $i"
  fgrep "$i" /foo/bar/file-of-all-certs-installed-by-pkg
  rc=$?
  if [ $rc -eq 0 ]; then
   rm "$i"
  else
   echo "Warn user about manual bad link or fix depending on flags"
  fi
done
Comment 14 Lindsay Haisley 2008-11-08 15:34:09 UTC
This is a bit of extra work but would satisfy me.  If you can assemble a such a list I'll flesh out your pseudocode and put it in update-ca-certificates and the man page for same, and you, or someone, can get it into the next ebuild.  This would, of course, be a maintenance issue going forward since the list would have to be updated every time changes are made to the cert list.  I don't know anything about the dynamics or politics of cert issuance but I know this particular issue has bitten me in the butt before, and I keep forgetting about it the next time it happens and getting bitten again.
Comment 15 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2008-11-08 18:28:31 UTC
I've got a list of certs ready to go.

The log/info/warn/error bit I mentioned with only emailing the important stuff:
PORTAGE_ELOG_SYSTEM="mail:warn,error save:info,log"
in your make.conf.
Comment 16 Lindsay Haisley 2008-11-08 18:44:38 UTC
Good.  Either send it to me in email, or post it as an attachment to this bug and I'll pick it up.  This weekend's big project is re-roofing a building before we get rain, but I'll give the project some quality attention very soon.

Thanks for the nudge on the ELOG stuff.  I've not used it before, but I'm playing with it on my current desktop update and will fine-tune it as necessary, and already it's sending me useful and informative email.  The comments in the current make.conf.example are quite adequate for setting it up.
Comment 17 SpanKY gentoo-dev 2015-03-22 00:01:35 UTC
the ca-certificates package automatically deletes & logs any broken links it finds.  i guess that covers this bug.