Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 395035 - [perl-experimental overlay] eix-update too slow due to lack of metadata
Summary: [perl-experimental overlay] eix-update too slow due to lack of metadata
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Perl team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-12-17 07:23 UTC by Alex Efros
Modified: 2013-12-23 12:25 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alex Efros 2011-12-17 07:23:27 UTC
Overlay perl-experimental have nearly same amount of ebuilds as main portage in dev-perl category, but for some reason eix-update took too many time to index perl-experimental overlay (minutes!).

I suppose this is either bug in eix, or lack of some internal meta information in perl-experimental overlay (like metadata/cache/ in portage).

Probably this isn't best place to report this issue, but I failed to find better one, sorry.
Comment 1 Martin Väth 2011-12-17 15:08:10 UTC
(In reply to comment #0)
> or lack of some internal meta information
> in perl-experimental overlay (like metadata/cache/ in portage).

It is the latter: The overlay has neither a metadata/cache nor
a metadata/md5-cache subdirectory. I guess if you want to change this
you should contact the overlay maintainers directly.
Or you produce the metadata by yourself using egencache.
Comment 2 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2011-12-20 09:07:18 UTC
I've considered making a derivative branch of the overlay with metadata automatedly generated via a cron job for people who need it, but really wanted to keep metadata out of the main repository due to the amount of excessive commit churn it generates.
Comment 3 Alex Efros 2011-12-20 09:31:55 UTC
(In reply to comment #2)
> I've considered making a derivative branch of the overlay with metadata
> automatedly generated via a cron job for people who need it, but really wanted
> to keep metadata out of the main repository due to the amount of excessive
> commit churn it generates.

Hmm. Why main portage doesn't suffer from this issue?
Comment 4 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2011-12-20 10:50:50 UTC
(In reply to comment #3)
> (In reply to comment #2)
> > I've considered making a derivative branch of the overlay with metadata
> > automatedly generated via a cron job for people who need it, but really wanted
> > to keep metadata out of the main repository due to the amount of excessive
> > commit churn it generates.
> 
> Hmm. Why main portage doesn't suffer from this issue?

Because when you do "log" in CVS, I believe it defaults to being relative to $PWD, not relative to the entire source-code. 

We're using Git. 

( And its been a while since I myself maintained a metadata cache generated overlay, but last time I did do it, doing it as part of the check-in process just hindered doing any development work ) 

For this reason, I think its better if the metadata cache /was/ generated to keep out of the way of developers unless they /need/ to see it.
Comment 5 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2012-04-05 17:35:44 UTC
To anyone who is interested, I've set up an experimental derivative of perl-experimental called "perl-experimental-snapshots" , which makes a nightly copy of perl-experimental @ master, and does a metadata generation pass. 

( And amongst other things, weeds out a few entries that are only useful for the perl-experimental overlay )

https://github.com/gentoo-perl/perl-experimental-snapshots

Its not listed in Layman yet, so if you want to track it yourself with layman:

custom.xml
---
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE repositories SYSTEM "/dtd/repositories.dtd">
<repositories xmlns="" version="1.0">
  <repo quality="experimental" status="unofficial">
    <name>perl-experimental-snapshots</name>
    <description>Nightly perl-experimental snapshots</description>
    <homepage></homepage>
    <source type="git">https://github.com/gentoo-perl/perl-experimental-snapshots.git</source>
    <owner>
      <email>kentfredric@gmail.com</email>
    </owner>
  </repo>
</repositories>
---

/etc/layman/layman.cfg
---
overlays  : http://www.gentoo.org/proj/en/overlays/repositories.xml
            file:///mnt/grafts/repositories/custom.xml      # except use your path
---


then `layman -f && layman -a perl-experimental-snapshots` and you should be good to go.

Dramatically speeds up the eix-update pass.
Comment 6 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2012-04-05 17:53:28 UTC
(In reply to comment #5)
> To anyone who is interested, I've set up an experimental derivative of
> perl-experimental called "perl-experimental-snapshots" , which makes a
> nightly copy of perl-experimental @ master, and does a metadata generation
> pass. 

Why don't you just provide a simple rsync mirror for your overlay? Then you can add it to layman directly and easily with metadata.
Comment 7 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2012-10-26 15:52:34 UTC
(In reply to comment #6)
> 
> Why don't you just provide a simple rsync mirror for your overlay? Then you
> can add it to layman directly and easily with metadata.


Slow response is slow... 


"Why don't you just provide a simple rsync mirror for your overlay?"

Mostly, because I don't have the resources at hand to provide an Rsync mirror, and its quite straight forward for me to make a git mirror.

ie:

1. I just locally clone perl-experimental
2. Just run some code to clean up the tree
3. Metadata pass runs
4. Code is committed in and then pushed to github. 

And this workflow isn't at all made easier by using rsync instead, and at least with this approach, there is already a platform I can push it to that people can use. 

"Then you can add it to layman directly and easily with metadata."

The same could be said for the git repo.

All thats required for layman to have this in it by default is for somebody to inject the relevant XML blob into the official layman 'repositories.xml'


<repo quality="experimental" status="unofficial">
    <name>perl-experimental-snapshots</name>
    <description>Nightly perl-experimental snapshots</description>
    <homepage></homepage>
    <source type="git">https://github.com/gentoo-perl/perl-experimental-snapshots.git</source>
    <owner>
      <email>kentfredric@gmail.com</email>
    </owner>
</repo>

The rest of what I pasted is just a hack people can use until such time as the official repositories.xml contains this blob. =)

I guess if I get off my ass and do the Dev quiz ( one day... ) then I might be able to produce an rsync service on the infra , and I know that from local usage, its much easier to rsync the repo than it is to use git to keep it in sync.

( well, at least it seems that way, but I've only tested this on a local-lan connection, as I rsync this repo myself from the box I do it on to my workstation, internet latencies might make this a different ball-game  )
Comment 8 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2012-10-26 15:56:24 UTC
In fact, with newer versions of layman, you don't need to edit layman.cfg at all, as newer laymans have an "overlays" directory, and all .xml files in it are automatically parsed: 

https://github.com/gentoo-perl/perl-experimental-snapshots/wiki/Usage
Comment 9 Alex Efros 2012-10-26 16:09:58 UTC
(In reply to comment #7)
> All thats required for layman to have this in it by default is for somebody
> to inject the relevant XML blob into the official layman 'repositories.xml'
> 
> <repo quality="experimental" status="unofficial">
>     <name>perl-experimental-snapshots</name>
>     <description>Nightly perl-experimental snapshots</description>
>     <homepage></homepage>
>     <source
> type="git">https://github.com/gentoo-perl/perl-experimental-snapshots.git</
> source>
>     <owner>
>       <email>kentfredric@gmail.com</email>
>     </owner>
> </repo>

Why don't you just email this to "Layman centralized list <overlays@gentoo.org>"? AFAIK that's all you need to have this overlay added to official list.

If someone will complain about "duplicates", then I think it's better to have perl-experimental-snapshots overlay in official list than perl-experimental - because last one is too slow to be usable for end-users.
Comment 10 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2012-10-31 16:58:32 UTC
(In reply to comment #9)

> Why don't you just email this to "Layman centralized list
> <overlays@gentoo.org>"? AFAIK that's all you need to have this overlay added
> to official list.
> 
> If someone will complain about "duplicates", then I think it's better to
> have perl-experimental-snapshots overlay in official list than
> perl-experimental - because last one is too slow to be usable for end-users.


I'm glad to inform you, that perl-experimental-snapshots is available via the core layman repositories.xml list =)
Comment 11 Alex Efros 2012-10-31 17:09:26 UTC
(In reply to comment #10)
> I'm glad to inform you, that perl-experimental-snapshots is available via
> the core layman repositories.xml list =)

Thanks!

One small issue with it - layman -L show it as:

  * perl-experimental-snap... [Git       ] (git://github.com/…

so without knowing it's name end with "…shots" it's impossible to add it.
And only way to find it is non-trivial:

  $ grep '<name>perl-experimental-snap' /var/lib/layman/overlays.xml 
    <name>perl-experimental-snapshots</name>

So, sorry for annoying you, but can you please request overlay renaming to "perl-experimental-snap"?
Comment 12 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2012-10-31 17:15:38 UTC
(In reply to comment #11)
> 
> So, sorry for annoying you, but can you please request overlay renaming to
> "perl-experimental-snap"?

I can do that, but also, I think I should file a bug about that, either 

a: Overlays should not be able to be added with longer names/aliases than the truncation limit 

or

b: Layman should not truncate overlay names
Comment 13 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2012-11-01 04:14:05 UTC
If you're interested:

bug #440628 - layman -L shows long overlay names truncated 

I'm not sure if its appropriate or not to mark this bug as blocked on that one, either way, I don't have perms to do that =)
Comment 14 Kent Fredric (IRC: kent\n) (RETIRED) gentoo-dev 2013-12-23 12:25:59 UTC
Changing assignment to Perl as it seems really a bug about the overlay being inefficient.

And I believe this bug can be considered resolved due to the existence of -snapshots

If that solution is not acceptable, please reopen the bug.

I might consider using a submodule to make metadata visible in the main repo, but I wouldn't bet on it, because that'll be messy and fragile.

And Layman -L is also long fixed in its display problems.