Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 402647 - sys-apps/portage emerge utility add layman sync functionality
Summary: sys-apps/portage emerge utility add layman sync functionality
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: AMD64 Linux
: Normal enhancement (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks: 240187
  Show dependency tree
 
Reported: 2012-02-08 08:11 UTC by Sergey S. Starikoff
Modified: 2015-03-28 21:04 UTC (History)
4 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sergey S. Starikoff 2012-02-08 08:11:48 UTC
During update of my home PC I've again missed syncronization of installed overlays (normally required in system update process).
Issue (questions provided by missing this phase) also regard to some questions on gentoo.ru

The suggestion is to add 'layman' USE flag for sys-apps/portage (requires app-portage/layman package and adding (or changing behaviour of existiong) some option to sync all used external-managed overlays by one command (of course I can write a script or use an alias, but as for me it don't looks a good idea), like works search in eix or web (http://gpo.zugaina.org/).

For example (one of possible solutions):
Building sys-apps/portage with suggested USE 'layman' changes '--sync' option behaviour from syncing only main tree ('gentoo' overlay) to syncing by default all installed overlays and adds handling of optional argument (i.e. emerge --sync overlay_name will sync overlay_name (if it is installed, and exit with error code if overlay is not installed).

Reproducible: Always

Steps to Reproduce:
Now to sync total portage tree (main tree i.e. 'gentoo' overlay + other used overlays) I need to use two commands:

Actual Results:  
1. # emerge --sync
2. # layman -S

Expected Results:  
If sys-apps/portage is built with suggested 'layman' USE flag, all branches could by synced by one standard command:
# emerge --sync
Comment 1 Lars Wendler (Polynomial-C) (RETIRED) gentoo-dev 2012-02-08 08:30:16 UTC
Why don't you just create some kind of alias like

  alias esync='layman -S ; emerge --sync'

for this purpose?

I'm fine with this solution for mayn months.
Comment 2 Sergey S. Starikoff 2012-02-08 10:47:09 UTC
(In reply to comment #1)
> Why don't you just create some kind of alias like
> 
>   alias esync='layman -S ; emerge --sync'
> 
> for this purpose?
> 
> I'm fine with this solution for mayn months.

To my mind alias like:
alias esync='emerge --sync && layman -S'
looks better.
And it is rather pritty workaround.

But as a question (feature request) to future portage development, to my mind this bug looks reasonable.
Comment 3 Jeroen Roovers (RETIRED) gentoo-dev 2012-02-08 14:23:29 UTC
Oh, so you want to merge portage and layman...
Comment 4 Jeroen Roovers (RETIRED) gentoo-dev 2012-02-08 14:24:19 UTC
You could just add the appropriate call to emerge (emerge --sync, say) to layman and be done with it.
Comment 5 Brian Dolbec (RETIRED) gentoo-dev 2012-02-08 16:19:29 UTC
Actually, With the layman-2.0 API I have added layman's sync to portage, it was very easy.  I also want to add layman's sync to pkgcore.

That said it would also be relatively easy to have layman sync the main gentoo tree.  It would only need a repo definition and alternate config due to the different storage location.  But, there are a couple drawbacks to that.  One is that layman does not do the round robin mirror stuff that portage does.  Second is that it also does not do the user privileges changes.

But back to adding using layman inside portage.  I proposed using layman as a library and perform the layman sync inside portage.  

First, main problem so far is that layman is not quite python-3 ready. 

Second,  layman's code would have to run in python-2 and python-3 unchanged. But my testing has shown that is not easily possible.  When I have it runnable in python-3, it will not run correctly in python-2 and vice-versa.  It will need to have the 2to3 migration helper be run on the code to install and run correctly for python-3.  Another part of that is that portage currently installs outside of python's site-packages into it's own namespace.  Layman installs to site-packages

Third, even though I have a nice high level api to do it, Zac wanted to have portage do the individual syncs manually rather than have layman handle it using layman's config and installed overlay list.


So, perhaps that will change with layman-2.1.  2.1 should be fully python-3 capable.  I have a little more work to do to have a clean 2to3 run without additional patches.  Plus I may add the gentoo round robin mirror capability to it if there is still resistance to using layman's api from emerge.


For a current fully working alternative, emerge >=app-portage/esearch-1.0  It has been re-written and fully updated.  It also has layman-sync functionality built in.  So it will update layman's overlays and the main tree (use the -l, --layman-sync option).  Then you will also have a fast simple search capability.  If you prefer eix, it too can sync installed overlays as well as the main tree.
Comment 6 Sergey S. Starikoff 2012-02-08 17:15:48 UTC
(In reply to comment #3)
> Oh, so you want to merge portage and layman...

Not exactly.
Only common functions.
Now both main tree and overlays packages are installed by portage's utility 'emerge'.
I suggest to manage with this utility not only installation (of all packages independing on overlay), but also the syncing of all trees.

Leaving for layman utility only the overlay's management itself: i.e. listing, adding and removing overlays.
Comment 7 Sergey S. Starikoff 2012-02-08 17:28:34 UTC
(In reply to comment #5)
> For a current fully working alternative, emerge >=app-portage/esearch-1.0  It
> has been re-written and fully updated.  It also has layman-sync functionality
> built in.  So it will update layman's overlays and the main tree (use the -l,
> --layman-sync option).  Then you will also have a fast simple search
> capability.  If you prefer eix, it too can sync installed overlays as well as
> the main tree.

Thank you!
If so (there is a choyce of utilities for common sync), emerge --sync option may be leaved as an utility for low-level main tree only sync.

And this bug becomes more document-related:
To my mind the first mentioning of sync-tree option in Handbook should be followed with note, listing utilities, which could sync main tree together with used overlays.
Comment 8 Sergey S. Starikoff 2012-02-09 06:29:42 UTC
(In reply to comment #5)
> layman does not do the round robin mirror stuff that portage does.
At least currently it is reasonable, because now only main tree (i.e. gentoo overlay) has mirroring (tree and distfiles) infrastructure.
So, for layman it (doing the round robin mirror stuff) is reasonable only after categorising overlays (providing for some of them mirroring infrastructure, the question is in that each overlay commonly should possess it's own mirroring infrastructure).

Also (to the question of mirroring) I want to remind bug #367479 (categorizing distfiles atoms for ebuilds in overlays, don't searching on main tree distfiles mirrors the distfiles, that aren't provided by them).


> For a current fully working alternative, emerge >=app-portage/esearch-1.0  It
> has been re-written and fully updated.  It also has layman-sync functionality
> built in.  So it will update layman's overlays and the main tree (use the -l,
> --layman-sync option).  Then you will also have a fast simple search
> capability.  If you prefer eix, it too can sync installed overlays as well as
> the main tree.

Again thank you.
I like the idea with esync alias (using basic utilities, eix or esearch).
Comment 9 Dennis Schridde 2014-01-06 23:38:18 UTC
In sys-apps/portage-2.2 there is /etc/portage/repos.conf/, which probably does what you want.
Comment 10 Brian Dolbec (RETIRED) gentoo-dev 2014-01-07 00:34:53 UTC
yes, I still have to make the repos.conf changes needed in layman.

I would probably add  a layman value to the suitable sync types accepted there.  That way it would handle url changes, etc. automatically through layman.  That is provided I get the auto-update functionality done. The will come the remaining py3 code changes needed.
Comment 11 Brian Dolbec (RETIRED) gentoo-dev 2014-05-18 23:31:34 UTC
With the new plugin sync system I'm nearly finished for portage, It will be easy to add a layman sync module.  It is also planned as part of a layman gsoc-2014 project.

This should be available by this fall.
Comment 12 Devan Franchini (RETIRED) gentoo-dev 2014-07-11 22:03:24 UTC
(In reply to Brian Dolbec from comment #11)
> With the new plugin sync system I'm nearly finished for portage, It will be
> easy to add a layman sync module.  It is also planned as part of a layman
> gsoc-2014 project.
> 
> This should be available by this fall.

The plugin is complete and will hopefully be merged with the master branch by the end of these upcoming weeks. However, that doesn't ensure that portage will be able to use the plugin.

As far as I'm aware, the plugin sync system for portage hasn't been made stable yet. That's out of my control. The layman sync plugin isn't though ;)
Comment 13 Dennis Schridde 2014-07-12 05:33:47 UTC
Great! Is there or will there be an overlay with this enhanced version of portage?
Comment 14 Devan Franchini (RETIRED) gentoo-dev 2014-07-13 22:57:02 UTC
(In reply to Dennis Schridde from comment #13)
> Great! Is there or will there be an overlay with this enhanced version of
> portage?

There is already a branch in the main portage repo:

http://git.overlays.gentoo.org/gitweb/?p=proj/portage.git;a=shortlog;h=refs/heads/plugin-sync

I however, don't advocate you using this until all of it gets pushed as portage's main version on the tree. If you want to, I can't stop you though. :)
Comment 15 Devan Franchini (RETIRED) gentoo-dev 2014-09-29 13:52:00 UTC
The functionality is in layman-2.2.0 but I'm going to leave this open as portage doesn't have the sync plug-in module system in the tree yet.