Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 130039 - check gpg signature when using emerge-webrsync
Summary: check gpg signature when using emerge-webrsync
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: High major (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
: 152977 (view as bug list)
Depends on:
Blocks: 216231
  Show dependency tree
 
Reported: 2006-04-15 02:26 UTC by Alon Bar-Lev (RETIRED)
Modified: 2010-08-28 05:22 UTC (History)
13 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge-webrsync - check gpg signature (emerge-webrsync.diff,2.53 KB, patch)
2006-04-15 02:33 UTC, Alon Bar-Lev (RETIRED)
Details | Diff
emerge-webrsync.diff - gpg + timestamp (emerge-webrsync.diff,4.83 KB, patch)
2006-04-15 16:46 UTC, Alon Bar-Lev (RETIRED)
Details | Diff
my-emerge-webrsync (my-emerge-webrsync,8.32 KB, text/plain)
2006-04-16 12:18 UTC, Alon Bar-Lev (RETIRED)
Details
s/PORTAGE_GPG_HOME/PORTAGE_GPG_DIR/ (emerge-websync,8.31 KB, text/plain)
2006-10-28 17:34 UTC, CPUShare
Details
don't call tarsync with -v if --verbose has not been specified (emerge-websync,8.34 KB, text/plain)
2006-11-09 03:27 UTC, CPUShare
Details
added call to post_sync (emerge-websync,8.40 KB, text/plain)
2007-04-07 12:02 UTC, CPUShare
Details
my-emerge-webrsync (my-emerge-webrsync,8.06 KB, text/plain)
2007-06-14 20:27 UTC, Alon Bar-Lev (RETIRED)
Details
my-emerge-webrsync (my-emerge-webrsync,8.04 KB, text/plain)
2007-06-17 05:23 UTC, Alon Bar-Lev (RETIRED)
Details
base-emerge-webrsync (emerge-webrsync,4.75 KB, text/plain)
2007-06-17 05:24 UTC, Alon Bar-Lev (RETIRED)
Details
Patch to make.conf.5 (make.conf.5.patch,403 bytes, patch)
2007-08-06 11:33 UTC, Arfrever Frehtes Taifersar Arahesis (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Alon Bar-Lev (RETIRED) gentoo-dev 2006-04-15 02:26:57 UTC
Hello,

I know you had gone through this issue again and again.
But I could not find any relevant formal resource regarding this.

Current portage sync is totally vulnerable to man-in-the-middle attack, files and digests can be replaced to make the end-user to compile a different sources.

All it takes is one successful attach in order to compromise a system!!!

I know there is a signature mechanism on individual ebuilds, but it is not enough...

I read about an option of signing ebuilds using a master key after a developer signs with his key, this is much better approach, providing that *ALL* ebuilds are signed! But I could not find any formal documentation regarding this, and not all ebuilds are signed.

Also, a key usage policy should be published... Where are the developers' keys stored (software, hardware), how do you deal with key lost, how do the master key is handled/accessed, how is the master key protected, where can it be retrive, where does it reside on local system, how can user make sure it is not replaced  by automatic procedures.

In the mean time, there is a gpg signature for portage snapshot... but I could not find any formal tool that verify this signature.

If there are formal documents that discuss this vulnerability and the solution, please forward me to... And accept my apologies.

Best Regards,
Alon Bar-Lev.
Comment 1 Alon Bar-Lev (RETIRED) gentoo-dev 2006-04-15 02:33:07 UTC
Created attachment 84701 [details, diff]
emerge-webrsync - check gpg signature

Again, I've seen people do this before... Just wanted to know why this kind a temperary solution was not merged.

In order to enforce signature validation, you need to put the following in make.conf:

PORTAGE_GPG_HOME="/etc/portage/gnupg"
WEBSYNC_VERIFY_SIGNATURE=1
SYNC="" # so emerge --sync will not work
Comment 2 Alon Bar-Lev (RETIRED) gentoo-dev 2006-04-15 14:57:10 UTC
Hello,

While we are discussing emerge-webrsync -- a few more thoughts:

1. We need to check file signature against time... So snapshot of 2006-02-03 was signed at 2006-02-03 -> 2006-02-04. This is required so attacker will not be able to place old snapshot with a new name (with signature valid) to rollback user.

2. The emerge-webrsync should be updated so if a newer portage snapshot was synced it will not sync an older snapshot. Current implementation goes from now into the past and sync the first one it finds... A rollback can be performed even if signature is valid only by returning not found for all recent ones.

3. The gpg signature of snapshot should be upgraded to DSA-2048/SHA-512. SHA-1 is weak.

Issues 1+2, can be easily solved by placing a file in the snapshot's archive that specifies some meta data (Including date of snapshot, date of signature). Maybe there is one, but I could not find it. It must be in the archive since it should be signed as well. This file can be even rsynced to the portage directory so a snapshot mark will be available and persistant even for users who do mixed rsync/webrsync.

Issue 3, can be solved even while we are speaking... :)

Best Regards,
Alon Bar-Lev.
Comment 3 SpanKY gentoo-dev 2006-04-15 15:10:39 UTC
> 1. We need to check file signature against time...

or just sign a digest of the file which includes the filename

timestamps can also be spoofed

> 2. The emerge-webrsync should be updated so if a newer portage snapshot was
> synced it will not sync an older snapshot.

i dont see how this helps anything
Comment 4 Alon Bar-Lev (RETIRED) gentoo-dev 2006-04-15 16:46:28 UTC
Created attachment 84746 [details, diff]
emerge-webrsync.diff - gpg + timestamp

>> 1. We need to check file signature against time...
> or just sign a digest of the file which includes the filename
But then users will not be able to use a simple gpg command in order to verify it...

> timestamps can also be spoofed
True. But I don't see the difference between the timestamp in the file name and a file within the archive.

>> 2. The emerge-webrsync should be updated so if a newer portage snapshot was
>> synced it will not sync an older snapshot.
> i dont see how this helps anything
Maybe I don't understand rsync very well... But it seems that if I rsync a snapshot from 2006-01-01 and then sync 2005-01-01 the result is portage at 2005-01-01, so rollback can be performed by an attacker.

---

I've found a file in portage/metadata/timestamp.x which have field 1 %s timestamp. I hope it is correct and usable.

I modified the script so it will not sync an older snapshot by comparing field 1 of timestamp.x in snapshot and in active portage. It also stops attempts after date is older than active portage timestamp.x. I hope I got it right... Especially for BSD which I cannot test.

The result:
portage snapshot is synced only if md5 matches, gpg signature is valid and snapshot is newer than active portage (snapshot's timestamp.x is newer than the active timestamp.x).

[[[ also fixing $FILE, "$FILE", ${FILE} to "${FILE}" ]]]
Comment 5 SpanKY gentoo-dev 2006-04-15 18:50:56 UTC
> >> 1. We need to check file signature against time...
> > or just sign a digest of the file which includes the filename
> But then users will not be able to use a simple gpg command in order to verify
> it...

so ?  users dont generally verify the snapshots, emerge-webrsync does ... plus, if a user is grabbing/verifying the archive, then it's up to the user to verify the metadata timestamp matches the filename of the archive and really, what average user is going to even know to do that ?  they're going to check the signature, see that it's correct, and assume the filename is also correct

> > timestamps can also be spoofed
> True. But I don't see the difference between the timestamp in the file name
> and a file within the archive.

i was referring to timestamps on the file itself, not in the names ... adding a sanity check to see if the filename matches the portage/metadata/timestamp.x would be a good idea in general

> >> 2. The emerge-webrsync should be updated so if a newer portage snapshot was
> >> synced it will not sync an older snapshot.
> > i dont see how this helps anything
> Maybe I don't understand rsync very well... But it seems that if I rsync a
> snapshot from 2006-01-01 and then sync 2005-01-01 the result is portage at
> 2005-01-01, so rollback can be performed by an attacker.

if the gpg signature includes the filename then an attacker cannot roll back the snapshot ... users may have a reason they want to roll back the snapshot so preventing valid usage "for the sake of security" is ridiculous
Comment 6 Alon Bar-Lev (RETIRED) gentoo-dev 2006-04-16 02:26:24 UTC
>> But then users will not be able to use a simple gpg command in order to verify
>> it...
> so ?  users dont generally verify the snapshots, emerge-webrsync does ... 
You are kidding, right? Maybe will... But it should not make things complex.

> if a user is grabbing/verifying the archive, then it's up to the user to verify
> the metadata timestamp matches the filename of the archive and really, what
> average user is going to even know to do that ?  they're going to check the
> signature, see that it's correct, and assume the filename is also correct
First of all, please let's not underestimate users! There is wide range of type of users, most of them are quite smart...

> i was referring to timestamps on the file itself, not in the names ... adding a
> sanity check to see if the filename matches the portage/metadata/timestamp.x
> would be a good idea in general
Why do you care what is the file name... Provided a file name XXX which passes verification using Gentoo key, and within this file there is portage/metadata/timestamp.x from 2006-03-02 -- This is enough to proceed.

I still think the filename is irrelevant... I don't understand why you keep mention it... Can you please explain?

> if the gpg signature includes the filename then an attacker cannot roll back
> the snapshot ... users may have a reason they want to roll back the snapshot so
Why does the signature is relevant here?
You don't provide a way to revert snapshot now in any of your current tools!
If we want this feature we can add it by a simple argument for emerge-webrsync:

emerge-webrsync --revert=2006-04-12

Then emerge-webrsync will automatically download snapshot 2006-04-12, check its signature, and sync portage ignoring current portage timestamp.

I can send you a patch for this too :)

> preventing valid usage "for the sake of security" is ridiculous

1. You don't allow this now... So it has nothing to do with security. If you thought it is important it had already been implemented.
2. Preventing usage for sake of security is valid! Current implementation doesn't take security in mind at all... So something should be corrected in the approach.
3. Nobody preventing... When a user tries to do something that may compromise its system, he should be warn... Usually a special force parameter solves this for advanced users, simple users will pick up the phone and call an advanced user... Hence protecting their system.
4. I would have replaced "for the sake of security" to "for the sake of the users that trust us". I believe you don't think that users' trust in Gentoo is ridiculous.
Comment 7 Alon Bar-Lev (RETIRED) gentoo-dev 2006-04-16 12:18:18 UTC
Created attachment 84790 [details]
my-emerge-webrsync

Hello,

A major change in original script.
Code is now cleaner and is a better base for discussion.

Changes:
Fixed file cleanup issues.
Fixed tarsync issues.
Fixed invalid operation of first downloaded md5 is invalid.
Default behavior is not to rollback portage to an older version.
Added revert options in order to rollback to an older version. 
Added snapshot signature check.
Revert will check that timestamp is in resnable period or requested revert.
Download is restartable now.

Known issues:
I did not check it on BSD environment.
I guess many more problems caused by rewrite.

When ready, this is going to be the main interface of syncing for many people, it would be great if you,  the long CC, will find the time to test it.

Best Regards,
Alon Bar-Lev.
Comment 8 SpanKY gentoo-dev 2006-04-16 13:18:04 UTC
> > so ?  users dont generally verify the snapshots, emerge-webrsync does ... 
>
> You are kidding, right? Maybe will... But it should not make things complex.

people are lazy ... the signature checked out thus the file is valid ... we already do the samething with our release media (sign the manifests, not the files)

> First of all, please let's not underestimate users!

i'm not saying users are stupid, people in general are lazy ... and why would someone even really think of checking the filename versus some internal metadata file ?  do a survey and see how many people even know about PORTDIR/metadata/

> Why do you care what is the file name...

because that's how we index the snapshots and that's how people find the latest snapshot release ... and if they want an older snapshot, then they look at the filename to find a version they want

> Why does the signature is relevant here?

.asc -> signature

> You don't provide a way to revert snapshot now in any of your current tools!

because we generally dont have any sources that are timestamped ... only our snapshots have timestamps

> 2. Preventing usage for sake of security is valid!

no it isnt
Comment 9 Alon Bar-Lev (RETIRED) gentoo-dev 2006-04-16 14:10:00 UTC
> i'm not saying users are stupid, people in general are lazy ... and why would
> someone even really think of checking the filename versus some internal
> metadata file ?  do a survey and see how many people even know about
> PORTDIR/metadata/

People? Gentoo developers should know about PORTDIR/metadata, people need to know that Gentoo developers make the best in order to secure their systems.

That's true that most people are lazy... But not ignorant... And exactly this is why we should provide an automatic secure package system. You (Gentoo community) had done a great deal in securing the portage once it is synced. There is one more major milestone to complete the work. The gpg snapshot signature validation is only 1/2 forward... Signing each ebuild (metadata) by a developer and by a trusted master key is the best solution.

---

Now let's cut to the chase.
1. From all this corresponding I did not understand if you acknowledged the need to add the signature validation to the emerge-webrsync.
2. Are the scripts that I post here help you? or would you rather do it your-self?
3. Do you need anything else in order to resolve this issue?
4. Can I help further in resolving this issue?

*There is a long CC list for this bug... I expected more people share their thoughts with us.*

Best Regards,
Alon Bar-Lev.

Comment 10 Thierry Carrez (RETIRED) gentoo-dev 2006-04-17 09:45:50 UTC
Changing this to "Default configs" as there are ways to secure the update but the default usage is indeed insecure.

IIRC the problem was more to decide on key policy (so that everything is really signed) rather than the sigchecking frontend...
Comment 11 Alon Bar-Lev (RETIRED) gentoo-dev 2006-04-17 12:12:19 UTC
> IIRC the problem was more to decide on key policy (so that everything is really
> signed) rather than the sigchecking frontend...

True!
Here is a proposal that seems a very good place to continue with... It is from may 2004!!!
Maybe the low-level already infrastructure exists, but all portage metadata should be updated. If there is no developer signature, an automatic signature may be provided for none critical packages. So portage will not emerge none trusted signed ebuilds.

From:  Robin H. Johnson - view profile
  Date:  Thurs, Mar 25 2004 5:00 am 
  Email:   "Robin H. Johnson" <robb...@gentoo.org>
  Subject: 2004.1 will not include a secure portage. 
  Groups:   linux.gentoo.dev
 
OK, after reading this entire thread, I've been thinking about a usable 
 implementation from both the administrative and developer perspective. 
 One of the most important things to remember in designing this, is that while 
 you can prevent damage from most individual attacks, no system in existence can 
 withstand a multi-faceted all-out assault. 
 
Goals: 
 ------ 
 - protect against compromised developer box / rogue developer 
 - protect against compromised rsync server 
 
Required operations: 
 -------------------- 
 1. add a key to trusted list 
 2. remove a key from trusted list 
 3. verify that a package has not been tampered with 
 4. sign a package 
 
General idea: 
 ------------- 
 we have a list (single file) distributed via rsync with all of keyids known 
 good for signing manifests. this file is must also signed, with at least one 
 (two or three required for more security) master key that has it's public part 
 widely published and it's private part kept secure. the verifiable signatures 
 on the trust list make it trusted. The signature for the list should be 
 available individually so that it can be checked frequently by portage for 
 changes (and if it has changed, download the list again). Some other users have 
 suggested using expiry times in various ways but that ignores the fact that a 
 portage tree snapshot taken in a known good state remains in that known good 
 state until altered. 
 
Signing a package: 
 ------------------ 
 manifests are signed as clear-text INSIDE the Manifest file (my prototype 
 already did this). this is important not to bloat the tree with more files. 
 This also allows multiple signatures on a Manifest (basically it just nests). 
 add a command to repoman, 'repoman sign' that signs a manifest and have repoman 
 commit check that a valid signature is present before committing (and 
 automatically does the sign stage on commit if required). 
 
to add a key to the trusted list: 
 --------------------------------- 
 1. add keyid into list 
 2. resign modified list with all required master keys 
 
to remove a key from the trusted list: 
 -------------------------------------- 
 1. remove keyid from list 
 2. resign modified list with all required master keys 
 
to verified that a package has not been tampered with: 
 ------------------------------------------------------ 
 1. read in manifest file, check that the signature(s) are valid. 
 2. grab the keyid(s) that it was signed with, and compare them against the 
 trusted list (and optionally a LOCAL list for users to have their own trusted 
 key structure) 
 
users can specify how many signatures a package needs to have for them to trust 
 it, eg a really paranoid user may require each manifest has at least 3 
 signatures. 
 
problem case examples: 
 ---------------------- 
 case 1: an rsync server is compromised 
 ====================================== 
 if intruder modifies an ebuild/patch it doesn't pass the manifest check 
 anymore, and if they modify the manifest, they need a key to sign it which 
 won't be in the trusted list. if they modify the list it won't pass the 
 external signature checks anymore. this case is made mostly airtight for the 
 users that enable security checks. 
 
case 2: a developers box is compromised 
 ========================================================= 
 this is a weakness in all of schemes floating around already. on discovery, the 
 keyid is removed from the trust list and an advisory issued. users requiring at 
 least N signatures on a package are protected unless the intruder hacks N 
 developer boxes. Assuming that developers keep their gpg keys with a pass-phrase 
 as they should (just like their ssh keys), there is minimal danger until 
 ssh/gpg are trojaned to capture the unencrypted key/pass-phrase. 
 
case 3: rogue developer 
 ======================= 
 this is one of the hardest things to catch in any system. just like in case 2, 
 users requiring at least N signatures are safe until N developers are 
 rogue/compromised. 
 
possible problems with this approach: 
 ------------------------------------- 
 for users that specify more than 1 signature must be on a manifest, it's very 
 hard to keep manifests in this state if the package changes often, since any 
 package change requires redoing the manifest which removes the old signatures. 
 this could be fixed by making manifest building incremental (eg it is NOT 
 regenerated from scratch but instead appended until an old set of signed data 
 no longer applies to any files [as they have all been changed]), but that would 
 require a re-write of the manifest generation code. 
 
Plan for incremental implementation: 
 ------------------------------------ 
 1. roll out portage with a cleaned up version of my prototype for repoman and 
    simple verification code for manifests ONLY. 
 2. create the trusted list and roll out a portage with support for it. 
 
-- 
 Robin Hugh Johnson 
 E-Mail     : robb...@orbis-terrarum.net 
Comment 12 Alon Bar-Lev (RETIRED) gentoo-dev 2006-05-09 10:42:31 UTC
Hello,
How can we make a progress?
Comment 13 Alon Bar-Lev (RETIRED) gentoo-dev 2006-09-04 09:06:40 UTC
Ping....
Comment 14 Jakub Moc (RETIRED) gentoo-dev 2006-10-27 08:07:19 UTC
*** Bug 152977 has been marked as a duplicate of this bug. ***
Comment 15 CPUShare 2006-10-27 09:17:32 UTC
I was redirected here after posting a similar bugreport.

Your my-emerge-webrsync sounds a good start, but wouldn't it be better to be able to keep using rsync?

See my suggestion in bug #152977. A simple signed checksum of all ordered files  in the portage tree would do the trick, it should reduce the network bandwidth required compared to the websync approach.

Perhaps my suggested `find|sort|xargs tar|md5sum|gpg --clearsign` isn't the best implementation but it gives you the idea of how simple that is.

Leaving the rsync servers going in a completely unverifiable manner as now sounds bad, not just for this distro but for linux as a whole, regardless of the websync. All other notable distro based on rpm are already robust against hacked rpm on the mirrors no matter how you fetch them (ftp/http/rsync/whatever).
Comment 16 CPUShare 2006-10-27 09:22:46 UTC
About your websync script, wouldn't it be better to use PORTAGE_GPG_DIR, that's what the "gpg strict" (mostly useless because imcomplete and too slow) feature already uses.
Comment 17 Alon Bar-Lev (RETIRED) gentoo-dev 2006-10-27 12:35:30 UTC
(In reply to comment #15)
> Your my-emerge-webrsync sounds a good start, but wouldn't it be better to be
> able to keep using rsync?

There is a problem with incremental update and signatures, I think keeping rsync with central signing will lead into some nasty edge conditions. Also, I've found that rsync works *MUCH* slower than getting the whole tree... So I am very happy with webrsync.

> Leaving the rsync servers going in a completely unverifiable manner as now
> sounds bad, not just for this distro but for linux as a whole, regardless of
> the websync. All other notable distro based on rpm are already robust against
> hacked rpm on the mirrors no matter how you fetch them
> (ftp/http/rsync/whatever).

I read bug#152977, I don't agree with you that validating signature when emerging takes time, on the contrary, it is the prefferable way, so at the time of emerge you know the integrity of the package. The validation should not access network, except of downloading CRL/OCSP Request or similar data structure one time until it expires.

There is work in progress regarding signing the whole tree, I don't know what the status is. I think this is *THE MAJOR* issue of Gentoo distribution.

This script was meant to give users a solution until portage will offer full scale integrity. I think it is simple enough, but it was not merged.

(In reply to comment #16)
> About your websync script, wouldn't it be better to use PORTAGE_GPG_DIR, that's
> what the "gpg strict" (mostly useless because imcomplete and too slow) feature
> already uses.

I can probably change this... But I don't see anyone wishes to merge this script in-place of current webrsync. So people I work with use this instead of the formal sync.
Comment 18 CPUShare 2006-10-27 12:58:42 UTC
(In reply to comment #17)
> There is a problem with incremental update and signatures, I think keeping
> rsync with central signing will lead into some nasty edge conditions. Also,
> I've found that rsync works *MUCH* slower than getting the whole tree... So I
> am very happy with webrsync.

Those edge conditions would imply that rsync has failed to do the job as good as websync would have done. So you better catch them as errors rather than risking an not coherent view of the portage tree. There's a potential issue with the md5sum on tar checking the permissions as well, but it's a minor issue and I suggested tar in my example but that's just an implementation detail, a couple of lines of python could be used to eliminate from the checksum all irrelevant information (like for example directory times, like they should be removed from the rsync as well with --omit-dir-times).

> I read bug#152977, I don't agree with you that validating signature when
> emerging takes time, on the contrary, it is the prefferable way, so at the
> time

To me it looked like it was taking time on "emerge -s". My point is that I don't want any emerge command (except --sync) to take any time checking any signature *all*, regardless if it's measurable or not.

If something's wrong with the mirror, it should be noticeable immediately during the --sync, and then no check is needed anymore. This has the advantage of the slow work happening overnight and not when you need a new package.

Furthermore it sounds good to have a check on the profiles and other metadata.

> There is work in progress regarding signing the whole tree, I don't know what
> the status is. I think this is *THE MAJOR* issue of Gentoo distribution.

Major weakness indeed.

> This script was meant to give users a solution until portage will offer full
> scale integrity. I think it is simple enough, but it was not merged.

Bad that it was not merged. I'll be an user of your scrit too, even though for me websync looks much slower than rsync. Perhaps I've less network bandwidth than you have so downloading 32M of the whole portage tree ever and ever again isn't noticeable slowdown for you.

For me rsync generates a total 4.8MByte of network transfer (both direction included), and in turn it's 6 times faster than the websync.

> I can probably change this... But I don't see anyone wishes to merge this

I understand what you mean.

> script in-place of current webrsync. So people I work with use this instead
> of the formal sync.

They're very right doing so. I'll be using it too even if it's slower than rsync for me, because it greatly decreases the risk to be trivially hacked (like it could have happened already btw, this isn't a theoretical matter we're talking about).
Comment 19 CPUShare 2006-10-28 17:34:51 UTC
Created attachment 100677 [details]
s/PORTAGE_GPG_HOME/PORTAGE_GPG_DIR/

changed to _DIR.

Script seem to work for me, hope it'll be merged and that there will be interest to sign the --sync tree as well in a similar secure way.

Thanks.
Comment 20 CPUShare 2006-11-09 03:27:31 UTC
Created attachment 101532 [details]
don't call tarsync with -v if --verbose has not been specified

minor change not to be verbose unless asked.

script works great here (it generates a lot more network traffic than rsync, but at least tarsync seems to reduce the local I/O compared to a full untar)

thanks.
Comment 21 CPUShare 2006-11-09 03:57:40 UTC
On a related topic, is there any valid reason why emerge --metadata doesn't call /etc/portage/bin/post_sync?
Comment 22 CPUShare 2006-11-09 05:05:21 UTC
I now setup "crontab -e" like this:

20 4 * * *      /root/bin/emerge-websync | grep -v '>>> Updating Portage cache'; /etc/portage/bin/post_sync
40 4 * * *      glsa-check -d affected

Then optionally with procmail I catch the email from cron and I forward it through sms to my cellphone.

:0 cw
* ^From:.*root@example\.com
* ^Subject:.*Cron.*glsa-check
* !^X-Loop: glsa-check@example\.com
| (formail \
  -i"From: glsa-check@example.com"; \
  -A"X-Loop: glsa-check@example.com" ; \
  ) | $SENDMAIL -oi -fglsa-check@example.com mycellphone@example.it 

I guess a --quiet option to pass to emerge-websync (to forward it to emerge --metadata) would be a plus to avoid the grep -v.
Comment 23 CPUShare 2006-11-18 02:19:24 UTC
It works perfectly, this morning I received an sms telling me to upgrade media-libs/libpng and the description of the bug, cve etc... Pretty cool.

The last version of the crontab is like this:

20 4 * * *      /root/bin/emerge-websync | grep -v '>>> Updating Portage cache'; /etc/portage/bin/post_sync; emerge -uDNvp --nocolor --nospinner world
40 4 * * *      glsa-check -d affected
Comment 24 Sune Kloppenborg Jeppesen (RETIRED) gentoo-dev 2006-11-24 12:15:25 UTC
Any news on this one?
Comment 25 Alon Bar-Lev (RETIRED) gentoo-dev 2006-11-24 12:21:46 UTC
(In reply to comment #24)
> Any news on this one?

Not that I am ware of...
But I don't understand why at least not replace the current emerge-webrsync with the modified one.
Comment 26 Tavis Ormandy (RETIRED) gentoo-dev 2006-11-24 12:26:24 UTC
reassigning to portage team as this is not an issue for the security team.
Comment 27 Zac Medico gentoo-dev 2006-12-05 13:15:37 UTC
(In reply to comment #25)
> But I don't understand why at least not replace the current emerge-webrsync
> with the modified one.

I'd like to make the signature verification conditional on FEATURES="gpg" being enabled.  I've never enabled that feature before, so I still need to try that before it's integrated into emerge-webrsync.
Comment 28 Alon Bar-Lev (RETIRED) gentoo-dev 2006-12-06 11:30:50 UTC
(In reply to comment #27)
> I'd like to make the signature verification conditional on FEATURES="gpg" being
> enabled.  I've never enabled that feature before, so I still need to try that
> before it's integrated into emerge-webrsync.

I don't think they are related...
The new script check the integrity of the download, and the gpg check the integrity of a specific package.

The gpg portage feature is far from being complete, until it will checking the integrity of the portage snapshot should be sufficient.
Comment 29 CPUShare 2007-02-25 20:53:04 UTC
I think the gpg portage feature is somewhat overkill (not just to maintain but it's quite overkill at runtime too!), furthermore it's not even close to be functional and it's certainly totally orthogonal with this web fetching script.

Please add the last attached secure script in replacement of the current web fetching script. It works fine and everyone should _stop_ to depend on the mirrors to be secure. This way people can use safely even the most local of the mirrors without worries improving the global bandwidth.
Comment 30 CPUShare 2007-04-02 22:14:22 UTC
reading on bugtraq the suggestion to run:

    # emerge --sync
    # emerge --ask --oneshot --verbose net-misc/asterisk

in order to fix a recent security vulnerability reminded me of this bug.

I think it's bad to recommend people to depend on a totally untrusted cleartext download to update their system.

Now if you followed my suggestion to add a signed checksum of the whole portage tree, we could still use cleartext rsync fully securely, until that will be implemented rsync should not be recommended and only the attached update script should be used. all IMHO ;)
Comment 31 CPUShare 2007-04-07 12:02:06 UTC
Created attachment 115656 [details]
added call to post_sync

same as before but it calls post_sync too
Comment 32 CPUShare 2007-05-05 16:01:09 UTC
today there was a wrong Changelog checksum on some vim package, so I tried to run "emerge --sync" to see if the more uptodate version was fixed.

While doing so I noticed that rsync deleted quite some stuff that was left there by tarsync, example:

deleting net-www/mod_watch/files/digest-mod_watch-3.18-r2
deleting net-www/mod_xslt/files/digest-mod_xslt-2.0.4
deleting net-www/mozplugger/files/digest-mozplugger-1.7.3-r1
deleting net-www/mplayerplug-in/files/digest-mplayerplug-in-3.35
deleting net-www/mplayerplug-in/files/digest-mplayerplug-in-3.31-r1
deleting net-www/mplayerplug-in/files/digest-mplayerplug-in-3.21
deleting net-www/netscape-flash/files/digest-netscape-flash-9.0.31.0
deleting net-www/netscape-flash/files/digest-netscape-flash-7.0.68
deleting net-www/nspluginwrapper/files/digest-nspluginwrapper-0.9.91.4
deleting net-www/nspluginwrapper/files/digest-nspluginwrapper-0.9.91.3
deleting net-www/nspluginwrapper/files/digest-nspluginwrapper-0.9.91.2
deleting net-www/pears/files/digest-pears-1.7
deleting net-www/pwauth/files/digest-pwauth-2.3.1-r3
deleting net-www/pwauth/files/digest-pwauth-2.3.1-r2
deleting net-www/vdradmin-am/files/digest-vdradmin-am-3.5.3
deleting net-www/vdradmin-am/files/digest-vdradmin-am-3.4.7-r1
deleting net-zope/abracadabraobject/files/digest-abracadabraobject-1.5.1
deleting net-zope/annotations/files/digest-annotations-0.4.3
deleting net-zope/archetypes/files/digest-archetypes-1.3.4
deleting net-zope/archetypes/files/digest-archetypes-1.3.2
deleting net-zope/archetypes/files/digest-archetypes-1.3.1
deleting net-zope/archetypes/files/digest-archetypes-1.2.5_rc5
deleting net-zope/atcontenttypes/files/digest-atcontenttypes-0.2.0
deleting net-zope/btreefolder2/files/digest-btreefolder2-1.0.1
deleting net-zope/btreefolder2/files/digest-btreefolder2-1.0
deleting net-zope/btreefolder2/files/digest-btreefolder2-0.5.0
deleting net-zope/calendarx/files/digest-calendarx-0.6.0
deleting net-zope/calendarx/files/digest-calendarx-0.4.15
deleting net-zope/cmf/files/digest-cmf-2.0.0
deleting net-zope/cmf/files/digest-cmf-1.6.0
deleting net-zope/cmf/files/digest-cmf-1.5.6
deleting net-zope/cmf/files/digest-cmf-1.5.1
deleting net-zope/cmf/files/digest-cmf-1.4.8
deleting net-zope/cmf/files/digest-cmf-1.4.7
deleting net-zope/cmf/files/digest-cmf-1.3.2
deleting net-zope/cmfactionicons/files/digest-cmfactionicons-0.9
deleting net-zope/cmfboard/files/digest-cmfboard-2.1.4
deleting net-zope/cmfboard/files/digest-cmfboard-2.1.2
deleting net-zope/cmfboard/files/digest-cmfboard-1.4.3
deleting net-zope/cmfcollectorng/files/digest-cmfcollectorng-0.20
deleting net-zope/cmfformcontroller/files/digest-cmfformcontroller-1.0.3
deleting net-zope/cmfformcontroller/files/digest-cmfformcontroller-1.0.2
deleting net-zope/cmfformcontroller/files/digest-cmfformcontroller-1.0.1
deleting net-zope/cmfforum/files/digest-cmfforum-1.0
deleting net-zope/cmfmember/files/digest-cmfmember-1.0_rc3
deleting net-zope/cmfmember/files/digest-cmfmember-1.0
deleting net-zope/cmfoodocument/files/digest-cmfoodocument-0.2.1
deleting net-zope/cmfopenflow/files/digest-cmfopenflow-1.0.99
deleting net-zope/cmfphoto/files/digest-cmfphoto-0.5.0
deleting net-zope/cmfphotoalbum/files/digest-cmfphotoalbum-0.5.0
deleting net-zope/cmfquickinstallertool/files/digest-cmfquickinstallertool-1.5.0
deleting net-zope/cmfquickinstallertool/files/digest-cmfquickinstallertool-1.4
deleting net-zope/coreblog/files/digest-coreblog-1.24
deleting net-zope/coreblog/files/digest-coreblog-1.20
deleting net-zope/coreblog/files/digest-coreblog-1.11
deleting net-zope/coreblog/files/digest-coreblog-1.0
deleting net-zope/coreblog/files/digest-coreblog-0.6
deleting net-zope/coreblog2/files/digest-coreblog2-0.90b
deleting net-zope/coreblog2/files/digest-coreblog2-0.81b
deleting net-zope/cvsfile/files/digest-cvsfile-0.9.0
deleting net-zope/epoz/files/digest-epoz-0.8.4
deleting net-zope/epoz/files/digest-epoz-0.8.0
deleting net-zope/externaleditor/files/digest-externaleditor-0.9.1
deleting net-zope/externaleditor/files/digest-externaleditor-0.8
deleting net-zope/externaleditor/files/digest-externaleditor-0.7.2
deleting net-zope/externaleditor/files/digest-externaleditor-0.7-r1
deleting net-zope/externalfile/files/digest-externalfile-1.2.0
deleting net-zope/externalstorage/files/digest-externalstorage-0.6
deleting net-zope/extfile/files/digest-extfile-1.4.2
deleting net-zope/extfile/files/digest-extfile-1.2.0_beta2
deleting net-zope/exuserfolder/files/digest-exuserfolder-0.20.0-r1
deleting net-zope/exuserfolder/files/digest-exuserfolder-0.20.0
deleting net-zope/filesystemsite/files/digest-filesystemsite-1.4.2
deleting net-zope/filesystemsite/files/digest-filesystemsite-1.4.1
deleting net-zope/fle3/files/digest-fle3-1.4.5
deleting net-zope/fle3/files/digest-fle3-1.4.2
deleting net-zope/formulator/files/digest-formulator-1.8.0
deleting net-zope/formulator/files/digest-formulator-1.6.2
deleting net-zope/formulator/files/digest-formulator-1.6.1
deleting net-zope/formulator/files/digest-formulator-1.4.2
deleting net-zope/formulator/files/digest-formulator-1.3.1
deleting net-zope/generator/files/digest-generator-1.3.0.13
deleting net-zope/generator/files/digest-generator-1.3.0.11
deleting net-zope/generator/files/digest-generator-1.3.0
deleting net-zope/genericuserfolder/files/digest-genericuserfolder-1.2.4
deleting net-zope/groups/files/digest-groups-0.3.2
deleting net-zope/groups/files/digest-groups-0.3.1
deleting net-zope/groupuserfolder/files/digest-groupuserfolder-3.2
deleting net-zope/groupuserfolder/files/digest-groupuserfolder-2.0.1
deleting net-zope/groupuserfolder/files/digest-groupuserfolder-2.0
deleting net-zope/i18ndude/files/digest-i18ndude-2.0
deleting net-zope/indexedcatalog/files/digest-indexedcatalog-0.6.0
deleting net-zope/issuetrackerproduct/files/digest-issuetrackerproduct-0.6.5
deleting net-zope/issuetrackerproduct/files/digest-issuetrackerproduct-0.5.2
deleting net-zope/issuetrackerproduct/files/digest-issuetrackerproduct-0.5.0b
deleting net-zope/kupu/files/digest-kupu-1.3.5
deleting net-zope/kupu/files/digest-kupu-1.3.1
deleting net-zope/ldapuserfolder/files/digest-ldapuserfolder-2.4
deleting net-zope/localfs/files/digest-localfs-1.7_rc1
deleting net-zope/localfs/files/digest-localfs-1.6
deleting net-zope/localfs/files/digest-localfs-1.0.0
deleting net-zope/localizer/files/digest-localizer-1.1.0_alpha1
deleting net-zope/localizer/files/digest-localizer-1.0.1
deleting net-zope/localizer/files/digest-localizer-1.0.0-r1
deleting net-zope/mimetypesregistry/files/digest-mimetypesregistry-1.3.3
deleting net-zope/mimetypesregistry/files/digest-mimetypesregistry-1.3.2.7
deleting net-zope/mimetypesregistry/files/digest-mimetypesregistry-1.3.2
deleting net-zope/mpoll/files/digest-mpoll-0.3.1
deleting net-zope/mymediamanager/files/digest-mymediamanager-1.3.1
deleting net-zope/neoboard/files/digest-neoboard-1.1
deleting net-zope/neoportallibrary/files/digest-neoportallibrary-0.9b
deleting net-zope/parsedxml/files/digest-parsedxml-1.4
deleting net-zope/passwordresettool/files/digest-passwordresettool-0.4.1


Is this normal tarsync behavior? I'd rather prefer rsync and tarsync to do the same thing. Otherwise I've to run --sync first to delete the stuff, and then emerge-websync to get the signature on it.

Still it's pretty bad that there's no way to get a signature verification with rsync, that eventually needs fixing too, there are many ways to do it, and it's fairly easy.
Comment 33 Alon Bar-Lev (RETIRED) gentoo-dev 2007-05-05 17:17:04 UTC
portage: Any chance including this someday? It simple enough, and best if maintained synchronous.
Comment 34 Alec Warner archtester Gentoo Infrastructure gentoo-dev Security 2007-05-05 22:58:27 UTC
You call portageq like a dozen times, call it once, ENVVAR takes multiple args and you can then use bash (mucho cheaper) to get the values out.  I think we do it in another of our bash scripts somewhere.

-Alec
Comment 35 Alon Bar-Lev (RETIRED) gentoo-dev 2007-06-14 20:27:13 UTC
Created attachment 122073 [details]
my-emerge-webrsync

Fixed script. Resync with emerge-webrsync.

I could not test this on BSD, I separated date related into functions: get_utc_date_in_seconds, get_date_part, get_utc_from_string.

If someone will be kind to test/modify these for BSD it would be great.

portage: If we start maintain only one script it will help everybody. This new script is modular and should be easier to maintain... Not mentioning it support signature validation :)

Thanks!
Comment 36 CPUShare 2007-06-14 21:16:05 UTC
Resynced script verified, and deployed.

Please use this one so everyone is secure (well with the exception of the poor rsync users that may not notice the difference).
Comment 37 Alon Bar-Lev (RETIRED) gentoo-dev 2007-06-17 05:23:56 UTC
Created attachment 122289 [details]
my-emerge-webrsync

Sync with latest portage.
Comment 38 Alon Bar-Lev (RETIRED) gentoo-dev 2007-06-17 05:24:39 UTC
Created attachment 122290 [details]
base-emerge-webrsync

For me to rebase in future.
Comment 39 Alon Bar-Lev (RETIRED) gentoo-dev 2007-08-05 10:36:42 UTC
Hello,
Is there any more we can do in order for you to merge this?
Alon.
Comment 40 Zac Medico gentoo-dev 2007-08-06 05:00:08 UTC
I've merged this into trunk (svn r7579). The only change that I made to your code was to trigger gpg verification with FEATURES="webrsync-gpg" instead of WEBSYNC_VERIFY_SIGNATURE.
Comment 41 Arfrever Frehtes Taifersar Arahesis (RETIRED) gentoo-dev 2007-08-06 11:33:40 UTC
Created attachment 127051 [details, diff]
Patch to make.conf.5

It should be documented before it will be forgotten.
Comment 42 Zac Medico gentoo-dev 2007-08-06 19:55:57 UTC
(In reply to comment #41)
> Created an attachment (id=127051) [edit]
> Patch to make.conf.5

Thanks, that's in svn r7582.
Comment 43 Alon Bar-Lev (RETIRED) gentoo-dev 2007-11-03 05:44:47 UTC
Hello,
Can you please release this in next release?
There is no reason to wait major.
Thanks!
Comment 44 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2007-11-25 03:09:03 UTC
Anybody using this, please note the new key is 0x239C75C4. Fetch it from the PGP keyservers.
Comment 45 Alon Bar-Lev (RETIRED) gentoo-dev 2007-11-25 11:07:11 UTC
Thanks!
This script continues to evolves as portage developer modify it in their source control.
Latest script is:
http://sources.gentoo.org/viewcvs.py/portage/main/trunk/bin/emerge-webrsync?view=markup

Don't forget adding webrsync-gpg to FEATURES at /etc/make.conf
Comment 46 Marius Mauch (RETIRED) gentoo-dev 2008-03-20 18:14:43 UTC
This is supposed to be fixed in portage-2.2_pre5 or earlier.
Comment 47 Marius Mauch (RETIRED) gentoo-dev 2008-03-20 18:15:21 UTC
This is supposed to be fixed in portage-2.2_pre5 or earlier.