Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 693454 - sys-apps/portage: emerge-webrsync support for arbitrary top-level directory name in gentoo repository snapshots
Summary: sys-apps/portage: emerge-webrsync support for arbitrary top-level directory n...
Status: IN_PROGRESS
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Tools (show other bugs)
Hardware: All All
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 574752 701268
  Show dependency tree
 
Reported: 2019-09-04 01:53 UTC by Zac Medico
Modified: 2019-12-15 23:03 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2019-09-04 01:53:54 UTC
In order to migrate the top-level directory name from "portage" to "gentoo" or "gentoo-YYYYMMDD" (bug 574752), we need to do something like this among other things:


> diff --git a/bin/emerge-webrsync b/bin/emerge-webrsync
> index 25daaf8eb..9c6e89dd3 100755
> --- a/bin/emerge-webrsync
> +++ b/bin/emerge-webrsync
> @@ -239,7 +239,7 @@ check_file_signature() {
>  get_snapshot_timestamp() {
>         local file="$1"
>  
> -       do_tar "${file}" --to-stdout -xf - portage/metadata/timestamp.x | cut -f 1 -d " "
> +       do_tar "${file}" --to-stdout --wildcards -x '*/metadata/timestamp.x' | cut -f 1 -d " "
>  }
>  
>  sync_local() {
Comment 1 Zac Medico gentoo-dev 2019-12-05 22:31:37 UTC
Also the sync_local function needs to be updated to not assume that the tarball contains a directory named portage. It can use tar --strip-components=1 to strip the top-level directory during extraction.
Comment 2 Zac Medico gentoo-dev 2019-12-07 22:06:19 UTC
Per bug 574752, comment 10, we're expecting a http://distfiles.gentoo.org/snapshots/gentoo-20191207.tar.xz snapshot that we can use for testing purposes.
Comment 3 Zac Medico gentoo-dev 2019-12-09 01:42:58 UTC
The first one is available now:

> $ curl -sS https://gentoo.osuosl.org/snapshots/gentoo-20191208.tar.xz | xz -d | tar --list -v | head
> drwxr-xr-x portage/portage   0 2019-12-08 16:39 gentoo-20191208/
> -rw-r--r-- portage/portage 1349 2019-12-08 16:39 gentoo-20191208/Manifest
> -rw-r--r-- portage/portage 29445 2019-12-08 16:39 gentoo-20191208/Manifest.files.gz
> drwxr-xr-x portage/portage     0 2019-12-08 16:39 gentoo-20191208/acct-group/
> -rw-r--r-- portage/portage 16028 2019-12-08 10:39 gentoo-20191208/acct-group/Manifest.gz
> drwxr-xr-x portage/portage     0 2019-12-08 16:39 gentoo-20191208/acct-group/adm/
> -rw-r--r-- portage/portage   592 2019-07-29 09:39 gentoo-20191208/acct-group/adm/Manifest
> -rw-r--r-- portage/portage   144 2019-07-29 09:39 gentoo-20191208/acct-group/adm/adm-0.ebuild
> -rw-r--r-- portage/portage   219 2019-07-29 09:39 gentoo-20191208/acct-group/adm/metadata.xml
> drwxr-xr-x portage/portage     0 2019-12-08 16:39 gentoo-20191208/acct-group/asterisk/
Comment 4 Larry the Git Cow gentoo-dev 2019-12-09 09:19:07 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=92f4b283c86a74c99fcfa98f8baccaf1b0603dc8

commit 92f4b283c86a74c99fcfa98f8baccaf1b0603dc8
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-12-09 09:12:11 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-12-09 09:14:05 +0000

    emerge-webrsync: handle gentoo-YYYYMMDD directory name
    
    Bug: https://bugs.gentoo.org/693454
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 bin/emerge-webrsync | 12 ++++--------
 1 file changed, 4 insertions(+), 8 deletions(-)
Comment 5 Larry the Git Cow gentoo-dev 2019-12-09 09:30:09 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=fa0f8720e168c95bcd6a8bd57d5abf09c488fd32

commit fa0f8720e168c95bcd6a8bd57d5abf09c488fd32
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-12-09 09:26:55 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-12-09 09:28:50 +0000

    emerge-delta-webrsync: handle gentoo-YYYYMMDD directory name
    
    Bug: https://bugs.gentoo.org/693454
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 misc/emerge-delta-webrsync | 13 ++++++-------
 1 file changed, 6 insertions(+), 7 deletions(-)
Comment 6 Zac Medico gentoo-dev 2019-12-09 09:37:48 UTC
The do_snapshot function needs to be updated to use the new file name, like:

> diff --git a/bin/emerge-webrsync b/bin/emerge-webrsync
> index 2190ad20b..b7deec879 100755
> --- a/bin/emerge-webrsync
> +++ b/bin/emerge-webrsync
> @@ -305,3 +305,3 @@ do_snapshot() {
>  
> -	local base_file="portage-${date}.tar"
> +	local base_file="${repo_name}-${date}.tar"
>  
> @@ -329,3 +329,3 @@ do_snapshot() {
>  		for compression in ${compressions} ; do
> -			local file="portage-${date}.tar.${compression}"
> +			local file="${repo_name}-${date}.tar.${compression}"
>  			local digest="${file}.md5sum"

Also, if the gentoo-YYYYMMDD tarballs only have xz compression, then the tarsync pathway has to be disabled since tarsync currently does not support xz compression.
Comment 7 Zac Medico gentoo-dev 2019-12-15 07:42:13 UTC
(In reply to Zac Medico from comment #6)
> tarsync pathway has to be disabled since tarsync currently does not support
> xz compression.

Actually tarsync supports xz compression now (see bug 702970).
Comment 9 Larry the Git Cow gentoo-dev 2019-12-15 22:01:58 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=b39c1731abd2b16e31e98e8deba9699dd73bbf7c

commit b39c1731abd2b16e31e98e8deba9699dd73bbf7c
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-12-15 08:14:00 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-12-15 08:26:21 +0000

    emerge-webrsync: support gentoo-YYYYMMDD snapshots
    
    Support gentoo-YYYYMMDD snapshots for forward compatibility, and
    portage-YYYYMMDD snapshots for backward compatibility.
    
    Bug: https://bugs.gentoo.org/693454
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 bin/emerge-webrsync | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)
Comment 10 Larry the Git Cow gentoo-dev 2019-12-15 23:00:49 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=9557f104aa85b65c7d394c52c5c8d8727a111651

commit 9557f104aa85b65c7d394c52c5c8d8727a111651
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-12-15 22:53:50 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-12-15 22:56:34 +0000

    sys-apps/portage: Bump to version 2.3.82
    
     #310009 emerge: Show package USE in conflict messages
     #680456 display relevant FEATURES when unshare fails
     #693454 emerge-webrsync: support gentoo-YYYYMMDD snapshots
     #702146 emerge: drop FEATURES=distcc-pump support
     #702970 emerge-webrsync: enable xz snapshots for tarsync
    
    Bug: https://bugs.gentoo.org/701268
    Bug: https://bugs.gentoo.org/310009
    Bug: https://bugs.gentoo.org/680456
    Bug: https://bugs.gentoo.org/693454
    Bug: https://bugs.gentoo.org/702146
    Bug: https://bugs.gentoo.org/702970
    Package-Manager: Portage-2.3.82, Repoman-2.3.20
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest              |   1 +
 sys-apps/portage/portage-2.3.82.ebuild | 272 +++++++++++++++++++++++++++++++++
 2 files changed, 273 insertions(+)