Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!

Bug 735622

Summary: app-portage/esearch last rites
Product: Gentoo Linux Reporter: Zac Medico <zmedico>
Component: Current packagesAssignee: Portage Tools Team <tools-portage>
Status: CONFIRMED ---    
Severity: normal CC: mbucas, phobosk
Priority: Normal    
Version: unspecified   
Hardware: All   
OS: All   
Whiteboard:
Package list:
Runtime testing required: ---
Bug Depends on: 735626, 737470, 737480    
Bug Blocks:    

Description Zac Medico gentoo-dev 2020-08-02 21:42:59 UTC
I think it's time to remove the app-portage/esearch ebuild, given that emerge --search performance should be competitive with esearch when `egencache --update-pkg-desc-index <repo>` is used to generate a cache.

Portage installs an example repos.postsync.d hook which can easily be modified to execute the desired egencache --update-pkg-desc-index <repo>` command after sync:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=ddf87266433247e1ec9aa90b8ffab99a240be7e4

Users that use sync-type git with https://gitweb.gentoo.org/repo/sync/gentoo.git or https://github.com/gentoo-mirror/gentoo do not need a postsync hook, since a freshly generated metadata/pkg_desc_index file is included with this repository.
Comment 1 Mickaël Bucas 2020-08-03 07:09:16 UTC
I like the feature of esync that shows new packages and new versions after a sync. (It doesn't work since a few days because of some haskell ebuild, but that's another story)

Is it possible to generate something similar with a repos.postsync.d hook ?
Comment 2 Daniel Rozsnyo 2020-08-16 18:41:39 UTC
It seems the emerge --search is not really compatible replacement of the esearch, I use to find packages that I know by name, so using the slash and dollar 

example:
  esearch /man-db$
  esearch man-db$

both output 1 package found, but it does not work here:
  emerge --search /man-db$  
  emerge --search man-db$

the / in the search pattern is not a slash? only the second query works, but its not what I want to do. ( portage-2.3.103 )

Another difference is a more problematic quoting if the search term starts with - (minus), like in:
  esearch -- -db$

what is the correct way to invoke emerge --search, since the --search option is NOT consuming the next parameter as the search key, but tryes to parse it, so these do not work:

  emerge --search -db$
results in: emerge: error: argument -d: ignored explicit argument '$'
  emerge --search -- -db$
Comment 3 Zac Medico gentoo-dev 2020-08-16 19:40:05 UTC
(In reply to Daniel Rozsnyo from comment #2)
> It seems the emerge --search is not really compatible replacement of the
> esearch, I use to find packages that I know by name, so using the slash and
> dollar 
> 
> example:
>   esearch /man-db$
>   esearch man-db$
> 
> both output 1 package found, but it does not work here:
>   emerge --search /man-db$  
>   emerge --search man-db$
> 
> the / in the search pattern is not a slash? only the second query works, but
> its not what I want to do. ( portage-2.3.103 )
> 
> Another difference is a more problematic quoting if the search term starts
> with - (minus), like in:
>   esearch -- -db$
> 
> what is the correct way to invoke emerge --search, since the --search option
> is NOT consuming the next parameter as the search key, but tryes to parse
> it, so these do not work:
> 
>   emerge --search -db$
> results in: emerge: error: argument -d: ignored explicit argument '$'
>   emerge --search -- -db$

For regular expressions, you need to prefix the search string with the % character. I've tested these strings, and they all appear to work correctly:

%^man-db$
%/man-db$
%^sys-apps/man-db
%-db$
Comment 4 Zac Medico gentoo-dev 2020-08-16 19:54:06 UTC
(In reply to Mickaël Bucas from comment #1)
> I like the feature of esync that shows new packages and new versions after a
> sync. (It doesn't work since a few days because of some haskell ebuild, but
> that's another story)
> 
> Is it possible to generate something similar with a repos.postsync.d hook ?

Opened bug 735622. If we implement that, then a hook similar to this one from bug 737470 will do the trick:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=f5ffdbf201e784e1cf559ffe33d0da1c6935b687
Comment 5 Zac Medico gentoo-dev 2020-08-16 19:55:17 UTC
I mean bug 737470 for the esync-like output.
Comment 6 Zac Medico gentoo-dev 2020-08-16 21:31:43 UTC
(In reply to Zac Medico from comment #3)
> For regular expressions, you need to prefix the search string with the %
> character. I've tested these strings, and they all appear to work correctly:
> 
> %^man-db$
> %/man-db$
> %^sys-apps/man-db
> %-db$

I think it could be nice to enable the regular expression support automatically when appropriate, so I've opened bug 737480 for this.
Comment 7 Zac Medico gentoo-dev 2020-09-08 04:28:19 UTC
In portage-3.0.6 both bug 737470 and bug 737480 are implemented.

Here's some example output from the esync --verbose emulation:

>  * Searching for changes
> 
>  [ N] acct-group/mongodb_exporter (0):  System group: mongodb_exporter
>  [ N] acct-user/mongodb_exporter (0):  user for mongodb_exporter
>  [MN] app-emulation/firecracker-bin (0.22.0):  Secure and fast microVMs for serverless computing (static build)
>  [MU] app-metrics/mongodb_exporter (0.11.1):  Prometheus exporter for MongoDB
>  [MU] dev-python/python-ctags (1.5.0):  Exuberant Ctags indexing python bindings
>  [ U] sys-apps/file (5.39-r3):  identify a file's format by scanning binary data for patterns
>  [ U] sys-devel/bison (3.7.1-r1):  A general-purpose (yacc-compatible) parser generator
>  [MU] www-client/google-chrome-beta (86.0.4240.22):  The web browser from Google
>  [MU] www-client/google-chrome-unstable (87.0.4252.0):  The web browser from Google
>  [MU] x11-misc/py3status (3.29):  py3status is an extensible i3status wrapper written in python

There's an example in /etc/portage/repo.postsync.d/example.
Comment 8 PhobosK 2020-09-10 18:00:26 UTC
(In reply to Zac Medico from comment #7)
> In portage-3.0.6 both bug 737470 and bug 737480 are implemented.
> 
> Here's some example output from the esync --verbose emulation:
> 
> There's an example in /etc/portage/repo.postsync.d/example.


This is not even close to an emulation of the esync --verbose results..... :(

Currently it lists all available packages in the portage tree...

esync used to list only the changes between last successful sync and the current sync
Comment 9 Zac Medico gentoo-dev 2020-09-10 18:15:39 UTC
(In reply to PhobosK from comment #8)
> (In reply to Zac Medico from comment #7)
> > In portage-3.0.6 both bug 737470 and bug 737480 are implemented.
> > 
> > Here's some example output from the esync --verbose emulation:
> > 
> > There's an example in /etc/portage/repo.postsync.d/example.
> 
> 
> This is not even close to an emulation of the esync --verbose results..... :(
> 
> Currently it lists all available packages in the portage tree...
> 
> esync used to list only the changes between last successful sync and the
> current sync

That means that it didn't have a previous /metadata/pkg_desc index to compare with.
Comment 10 PhobosK 2020-09-12 20:52:59 UTC
(In reply to Zac Medico from comment #9)
> That means that it didn't have a previous /metadata/pkg_desc index to
> compare with.

Well IMHO either you are wrong :) or I do not get what you are trying to explain :)

So here is what i mean:

When you do a simple: emerge --sync -v

the first thing portage does is to delete the /metadata/pkg_desc you talk about:

receiving incremental file list
Manifest
Manifest.files.gz
deleting metadata/pkg_desc_index

Then it syncs it with main tree and thus if you have the "example" postsync hook enabled - you get the FULL portage package list EVERY TIME after the sync ends 

And if after such a sync, you try to run: egencache --update-pkg-desc-index --repo=gentoo --verbose

you get: * No updates found
which is totally normal...


So all this is far from the symple usage and output of the: esync --verbose

And it requires far more config from user point of view :(
Comment 11 Zac Medico gentoo-dev 2020-09-12 21:47:49 UTC
(In reply to PhobosK from comment #10)
Replied in bug 737470, comment 5.