Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 737470 - sys-apps/portage: enable egencache --update-pkg-desc-index to display new packages and version bumps like esync
Summary: sys-apps/portage: enable egencache --update-pkg-desc-index to display new pac...
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All All
: Normal enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 735622 739718
  Show dependency tree
 
Reported: 2020-08-16 19:50 UTC by Zac Medico
Modified: 2020-11-22 07:19 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Zac Medico gentoo-dev 2020-08-16 19:50:50 UTC
Since app-portage/esearch is deprecated, it would be nice if egencache --update-pkg-desc-index could display new packages and version bumps like esync did. The relevant esync code displays search results for any new packages or packages with version bumps:

    for (pkg, version) in items:
        if (pkg not in old_keys) or (old[pkg] != new[pkg]):
            success = searchdb(config, ["^" + pkg + "$"], tree_new)
            haspkgs = True
Comment 1 Zac Medico gentoo-dev 2020-08-16 20:10:40 UTC
The esync output is uses the esearch --compact formatting produced by this function:

def do_compact(pkg):
    prefix0 = " "
    prefix1 = " "

    if pkg[3] == pkg[4]:
        color = darkgreen
        prefix1 = "I"
    elif not pkg[4]:
        color = darkgreen
        prefix1 = "N"
    else:
        color = turquoise
        prefix1 = "U"

    if pkg[2]:
        prefix0 = "M"

    return " [%s%s] %s (%s):  %s" % \
            (red(prefix0), color(prefix1), bold(pkg[1]), color(pkg[3]), pkg[7])

For a tuple like this:

('portage', 'sys-apps/portage', False, '3.0.2-r2', '3.0.2-r1', '1,025 kB', 'https://wiki.gentoo.org/wiki/Project:Portage', 'Portage is the package management and distribution system for Gentoo', 'GPL-2')

The output looks like this:

 [ U] sys-apps/portage (3.0.2-r2):  Portage is the package management and distribution system for Gentoo
Comment 3 Larry the Git Cow gentoo-dev 2020-09-08 00:43:20 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=00bb1df12827cfce898e6fce8514d59a4cc844b1

commit 00bb1df12827cfce898e6fce8514d59a4cc844b1
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-09-01 06:44:57 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-09-08 00:35:38 +0000

    egencache --update-pkg-desc-index: emulate esync --verbose output (bug 737470)
    
    When the --verbose flag is given, make --update-pkg-desc-index emulate
    esync --verbose output. Example:
    
     * Searching for changes
    
     [ N] acct-group/ultimaker (0):  Group for ultimaker
     [ N] acct-user/ultimaker (0):  User for ultimaker
     [ U] www-client/opera (70.0.3728.144):  A fast and secure web browser
     [MU] www-client/opera-developer (72.0.3798.0):  A fast and secure web browser
     [ U] x11-libs/gtksourceview (4.6.1-r1):  A text widget implementing syntax highlighting and other features
    
    Bug: https://bugs.gentoo.org/737470
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 bin/egencache               | 76 +++++++++++++++++++++++++++++++++++++++++----
 cnf/repo.postsync.d/example |  2 +-
 man/egencache.1             |  3 ++
 3 files changed, 74 insertions(+), 7 deletions(-)
Comment 4 Larry the Git Cow gentoo-dev 2020-09-08 03:51:38 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=fd840baeef963f5fe746f9167976ce0eed83a3b4

commit fd840baeef963f5fe746f9167976ce0eed83a3b4
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-09-08 03:44:51 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-09-08 03:51:23 +0000

    sys-apps/portage: Bump to version 3.0.6
    
     #668302 binrepos.conf: support fetchcommand customization
     #668334 Add binrepos.conf to replace PORTAGE_BINHOST
     #704416 env-update: create systemd user-session environment
     #737470 egencache --update-pkg-desc: emulate esync --verbose
     #737480 emerge --search: auto-detect regular expressions
     #739908 @change-deps: fix erroneous repeated rebuilds
     #740588 make.conf: Treat __* variables as local
     #740898 PORTAGE_BINHOST urlopen proxy support
     #740904 rsync and gemato proxy support
    
    Bug: https://bugs.gentoo.org/739718
    Bug: https://bugs.gentoo.org/668302
    Bug: https://bugs.gentoo.org/668334
    Bug: https://bugs.gentoo.org/704416
    Bug: https://bugs.gentoo.org/737470
    Bug: https://bugs.gentoo.org/737480
    Bug: https://bugs.gentoo.org/739908
    Bug: https://bugs.gentoo.org/740588
    Bug: https://bugs.gentoo.org/740898
    Bug: https://bugs.gentoo.org/740904
    Package-Manager: Portage-3.0.6, Repoman-3.0.1
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest             |   1 +
 sys-apps/portage/portage-3.0.6.ebuild | 265 ++++++++++++++++++++++++++++++++++
 2 files changed, 266 insertions(+)
Comment 5 Zac Medico gentoo-dev 2020-09-12 21:46:22 UTC
(In reply to PhobosK from bug 735622 comment #10)
> (In reply to Zac Medico from bug 735622 comment #9)
> > That means that it didn't have a previous /metadata/pkg_desc index to
> > compare with.
> 
> Well IMHO either you are wrong :) or I do not get what you are trying to
> explain :)
> 
> So here is what i mean:
> 
> When you do a simple: emerge --sync -v
> 
> the first thing portage does is to delete the /metadata/pkg_desc you talk
> about:
> 
> receiving incremental file list
> Manifest
> Manifest.files.gz
> deleting metadata/pkg_desc_index
> 
> Then it syncs it with main tree and thus if you have the "example" postsync
> hook enabled - you get the FULL portage package list EVERY TIME after the
> sync ends 
> 
> And if after such a sync, you try to run: egencache --update-pkg-desc-index
> --repo=gentoo --verbose
> 
> you get: * No updates found
> which is totally normal...
> 
> 
> So all this is far from the symple usage and output of the: esync --verbose
> 
> And it requires far more config from user point of view :(

Ah, I've tested with sync-type = git (using the raw gentoo repository which does not contain /metadata/pkg_desc_index), but sync-type = rsync introduces some complications:

1) You would have to rsync --exclude /metadata/pkg_desc_index.
2) Using rsync --exclude breaks verification (bug 657324).

In order to solve this problem, we'll need an egencache --upate-pkg-desc option that will force the pkg_desc_index to be saved in /var/cache/edb/dep instead (it already does this when the repository happens to be read-only for some reason).
Comment 6 Zac Medico gentoo-dev 2020-09-13 01:03:07 UTC
Add egencache --external-cache-only option (to avoid interference with repository verification):

https://archives.gentoo.org/gentoo-portage-dev/message/07f35baced46e865b0c1d9356d2e12de
https://github.com/gentoo/portage/pull/621
Comment 7 Larry the Git Cow gentoo-dev 2020-09-14 07:26:56 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/portage.git/commit/?id=c783c57205107bfa75441b90a35f414163baad42

commit c783c57205107bfa75441b90a35f414163baad42
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-09-12 23:09:56 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-09-14 06:23:58 +0000

    egencache: add --external-cache-only option (bug 737470)
    
    The --external-cache-only option is useful for client-side
    use cases where writing cache files inside the repository itself
    may interfere with repository verification. This option is
    currently supported for --update and --update-pkg-desc-index
    actions, for which consumers of the corresponding cache or
    index files are already capable of consuming files from the
    external cache directory (/var/cache/edb/dep).
    
    Bug: https://bugs.gentoo.org/737470
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 bin/egencache               | 44 ++++++++++++++++++++++++++------------------
 cnf/repo.postsync.d/example | 19 +++++++++----------
 man/egencache.1             |  9 ++++++++-
 3 files changed, 43 insertions(+), 29 deletions(-)
Comment 8 Larry the Git Cow gentoo-dev 2020-09-14 07:35:27 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=1d7645a12b16a08ad45b1c82971d30e99e8ab589

commit 1d7645a12b16a08ad45b1c82971d30e99e8ab589
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2020-09-14 07:01:26 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2020-09-14 07:35:19 +0000

    sys-apps/portage: Bump to version 3.0.7
    
     #737470 egencache: add --external-cache-only option
     #741474 get_mirror_url: urlquote only for ftp, http, and https
     #742446 support riscv ilp32 and ilp32d ABIs
    
    Bug: https://bugs.gentoo.org/739718
    Bug: https://bugs.gentoo.org/737470
    Bug: https://bugs.gentoo.org/741474
    Bug: https://bugs.gentoo.org/742446
    Package-Manager: Portage-3.0.7, Repoman-3.0.1
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 sys-apps/portage/Manifest             |   1 +
 sys-apps/portage/portage-3.0.7.ebuild | 267 ++++++++++++++++++++++++++++++++++
 2 files changed, 268 insertions(+)