Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 574752 - Rename portage-YYYYMMDD.tar* snapshots with gentoo-YYYYMMDD.tar*
Summary: Rename portage-YYYYMMDD.tar* snapshots with gentoo-YYYYMMDD.tar*
Status: IN_PROGRESS
Alias: None
Product: Gentoo Infrastructure
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Infrastructure
URL:
Whiteboard:
Keywords:
: 679814 (view as bug list)
Depends on: 693454
Blocks: 662982
  Show dependency tree
 
Reported: 2016-02-14 21:37 UTC by Michał Górny
Modified: 2020-10-20 11:02 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
mastermirror.diff (mastermirror.diff,3.03 KB, patch)
2019-09-04 06:49 UTC, Michał Górny
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2016-02-14 21:37:28 UTC
Since we will need to change our syncing stuffs to get proper GPG signatures on uncompressed tarballs, I think it would be nice to rename the tarballs as well, to gentoo-YYYYMMDD.tar*, following our effort on removing the conflicting use of term 'portage'.
Comment 1 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2019-04-16 10:30:00 UTC
*** Bug 679814 has been marked as a duplicate of this bug. ***
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2019-04-16 10:31:08 UTC
I guess what we have to do is to generate extra snapshots along the old ones.  While at it, we may also leave .bz2 for old snapshots only.
Comment 3 Ulrich Müller gentoo-dev 2019-04-16 11:26:45 UTC
Plus, the top-level directory should be renamed from portage to gentoo.
Comment 4 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2019-04-16 11:27:46 UTC
(In reply to Ulrich Müller from comment #3)
> Plus, the top-level directory should be renamed from portage to gentoo.

To gentoo-YYYYMMDD even, to match tarball name.
Comment 5 Zac Medico gentoo-dev 2019-04-18 18:14:05 UTC
In order to rename the top-level directory, tools like emerge-webrsync and emerge-delta-webrsync need to be updated to use tar commands with --strip-components 1.
Comment 6 Zac Medico gentoo-dev 2019-04-18 18:41:47 UTC
Also, if the top-level directory name is not constant, emerge-webrsync and emerge-delta-webrsync will have to find an alternative to this command that's used to extract the timestamp:

> tar --to-stdout -x portage/metadata/timestamp.x

It seems that --strip-components 1 is not helpful for this particular operation, but this works:

> tar --to-stdout --wildcards -x '*/metadata/timestamp.x'
Comment 7 Michael 'veremitz' Everitt 2019-09-04 01:32:19 UTC
No progress on this? What are the blockers? If nobody is using the snapshots, in favour of emerge-webrsync then surely these can be dropped? Otherwise, the tarballs still contain a 'portage' subpath which doesn't match new locations vis https://bugs.gentoo.org/show_bug.cgi?id=662982 .

Looks like the master scripts are in https://gitweb.gentoo.org/infra/mastermirror-scripts.git

From comments looks like three discrete steps - fix emerge-xxx scripts/functions, update root path, rename tar-balls, unless I'm mistaken?
Comment 8 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2019-09-04 06:49:55 UTC
Created attachment 589026 [details, diff]
mastermirror.diff

Here's a cheap patch I proposed at some point.
Comment 9 Alec Warner archtester Gentoo Infrastructure gentoo-dev Security 2019-09-04 15:10:10 UTC
(In reply to Michael 'veremitz' Everitt from comment #7)
> No progress on this? What are the blockers? If nobody is using the
> snapshots, in favour of emerge-webrsync then surely these can be dropped?
> Otherwise, the tarballs still contain a 'portage' subpath which doesn't
> match new locations vis https://bugs.gentoo.org/show_bug.cgi?id=662982 .
> 
> Looks like the master scripts are in
> https://gitweb.gentoo.org/infra/mastermirror-scripts.git
> 
> From comments looks like three discrete steps - fix emerge-xxx
> scripts/functions, update root path, rename tar-balls, unless I'm mistaken?

On infra's end we need to have both filenames for some time (possibly forever, but hopefully not.) Mgorny's patch is nice (its straightforward) but it increases the disk requirement for Gentoo Mirrors. If we can get away with this perhaps we should apply the patch. Otherwise I'm not super interested in writing a bunch of complex shell to sustain a link farm just so a file has the correct name. That isn't enough value for the complexity required, IMHO.

-A
Comment 10 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2019-12-07 07:59:31 UTC
infra/mastermirror-scripts.git
tag 20191207T072430Z contains cleanups just prior to this new work.
tag 20191207T075744Z implements this bug.

20191207T072430Z is being deployed tonight, 20191207T075744Z will be deployed tommorow (2019/12/07).

mgorny: thanks for the clear example patch, I picked an alternate way of doing it that's much less IO intensive (transform the old uncompressed tarball instead of creating a new one).
Comment 11 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2019-12-08 18:22:47 UTC
ulm:
tag 20191208T062327Z is in place for the run that will take place at 2019/12/09 00:45 UTC
Comment 12 Zac Medico gentoo-dev 2019-12-16 00:31:33 UTC
In portage-2.3.82, emerge-webrsync supports gentoo-YYYYMMDD snapshots:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=b39c1731abd2b16e31e98e8deba9699dd73bbf7c

commit b39c1731abd2b16e31e98e8deba9699dd73bbf7c
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-12-15 08:14:00 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-12-15 08:26:21 +0000

    emerge-webrsync: support gentoo-YYYYMMDD snapshots
    
    Support gentoo-YYYYMMDD snapshots for forward compatibility, and
    portage-YYYYMMDD snapshots for backward compatibility.
    
    Bug: https://bugs.gentoo.org/693454
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 bin/emerge-webrsync | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
Comment 13 Zac Medico gentoo-dev 2020-01-12 22:52:36 UTC
A seamless transition is not possible for emerge-delta-webrsync, since it's not possible for the same deltas to apply to both a gentoo-YYYYMMDD snapshot and a portage-YYYYMMDD snapshot. Perhaps the best way to trigger the transition will be with a USE flag that we add to package.use.force when we want to force the transition.
Comment 14 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2020-01-13 08:07:08 UTC
zmedico: the bdelta performance on the gentoo* tarballs is ~100% larger than before, because the filename prefix keeps changing.

Switching to another alg supported by differ limits it to at most 50% larger than the existing deltas.

Here's the size in bytes for the different algs:

 8667219 snapshot-gentoo-20200111-20200112.bdelta
 4638241 snapshot-gentoo-20200111-20200112.bdelta.bz2
 4568484 snapshot-gentoo-20200111-20200112.bdelta.xz
 4649757 snapshot-portage-20200111-20200112.bdelta
 3997388 snapshot-portage-20200111-20200112.bdelta.bz2
 3971992 snapshot-portage-20200111-20200112.bdelta.xz

 8959287 snapshot-gentoo-20200111-20200112.bdiff
 5287214 snapshot-gentoo-20200111-20200112.bdiff.bz2
 4779192 snapshot-gentoo-20200111-20200112.bdiff.xz
 4573762 snapshot-portage-20200111-20200112.bdiff
 4119809 snapshot-portage-20200111-20200112.bdiff.bz2
 4052188 snapshot-portage-20200111-20200112.bdiff.xz

 6984950 snapshot-gentoo-20200111-20200112.gdiff
 5262625 snapshot-gentoo-20200111-20200112.gdiff.bz2
 4791808 snapshot-gentoo-20200111-20200112.gdiff.xz
 4502358 snapshot-portage-20200111-20200112.gdiff
 4109229 snapshot-portage-20200111-20200112.gdiff.bz2
 4040656 snapshot-portage-20200111-20200112.gdiff.xz

 6394907 snapshot-gentoo-20200111-20200112.switching
 4693603 snapshot-gentoo-20200111-20200112.switching.bz2
 4618812 snapshot-gentoo-20200111-20200112.switching.xz
 4428992 snapshot-portage-20200111-20200112.switching
 4013645 snapshot-portage-20200111-20200112.switching.bz2
 3991848 snapshot-portage-20200111-20200112.switching.xz
Comment 15 Georgy Yakovlev gentoo-dev 2020-04-05 08:14:46 UTC
hi, any news here in this bug? what's the current status? asking for council meeting =)
Comment 16 Zac Medico gentoo-dev 2020-04-05 19:04:31 UTC
(In reply to Robin Johnson from comment #14)
> zmedico: the bdelta performance on the gentoo* tarballs is ~100% larger than
> before, because the filename prefix keeps changing.
> 
> Switching to another alg supported by differ limits it to at most 50% larger
> than the existing deltas.
> 
> Here's the size in bytes for the different algs:

I think any one of the algorithms will be fine, since the diffs compress reasonably well regardless of the algorithm.
Comment 17 Ulrich Müller gentoo-dev 2020-10-11 18:39:26 UTC
(In reply to Robin Johnson from comment #14)
> zmedico: the bdelta performance on the gentoo* tarballs is ~100% larger than
> before, because the filename prefix keeps changing.

Since there is little progress here, how about the obvious solution? Namely, use the constant name "gentoo" for the top-level directory.
Comment 18 Zac Medico gentoo-dev 2020-10-11 20:13:12 UTC
Maybe we should last-rite emerge-delta-webrsync, given that sync-type = git is a viable alternative for minimal-bandwidth sync these days.
Comment 19 Ulrich Müller gentoo-dev 2020-10-12 06:39:43 UTC
(In reply to Zac Medico from comment #18)
> Maybe we should last-rite emerge-delta-webrsync, given that sync-type = git
> is a viable alternative for minimal-bandwidth sync these days.

Maybe have a forums poll to find out if people still use it?
Comment 20 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2020-10-20 06:46:08 UTC
(In reply to Ulrich Müller from comment #17)
> (In reply to Robin Johnson from comment #14)
> > zmedico: the bdelta performance on the gentoo* tarballs is ~100% larger than
> > before, because the filename prefix keeps changing.
> 
> Since there is little progress here, how about the obvious solution? Namely,
> use the constant name "gentoo" for the top-level directory.

That means carrying a series of tarballs with the undated directory in them; if zmedico is fine with that (last time he wasn't, and wanted to keep the date in the prefix path inside gentoo-* tarballs for good reason); then this entire bug becomes moot.(In reply to Ulrich Müller from comment #19)

> (In reply to Zac Medico from comment #18)
> > Maybe we should last-rite emerge-delta-webrsync, given that sync-type = git
> > is a viable alternative for minimal-bandwidth sync these days.
> 
> Maybe have a forums poll to find out if people still use it?
ulm: can council please start a formal poll on it, announced widely on -dev & -dev-announce; just please use somewhere other than Forums ;-)

I think getting rid of emerge-delta-webrsync, to be replaced with some incremental git bundles would probably work, but some details need to be worked out.
Comment 21 Ulrich Müller gentoo-dev 2020-10-20 11:02:20 UTC
(In reply to Robin Johnson from comment #20)
> > > Maybe we should last-rite emerge-delta-webrsync, given that sync-type = git
> > > is a viable alternative for minimal-bandwidth sync these days.
> > 
> > Maybe have a forums poll to find out if people still use it?
> ulm: can council please start a formal poll on it, announced widely on -dev
> & -dev-announce; just please use somewhere other than Forums ;-)

IIUC, we won't need a poll if we change to an undated top-level directory. So I'd suggest that we settle that other question first.