Bugzilla Bug 199788

As discussed on Flameeyes' blog and my blog. It's been hashed out in
#gentoo-dev and a lot of us would like to start taking advantage of it.

http://farragut.flameeyes.is-a-geek.org/articles/2007/11/19/lets-actually-get-some-metadata

http://blog.cardoe.com/archives/2007/11/19/use-flag-metadata/

Created an attachment (id=136492) [edit]
metadata DTD diff

DTD patched.

IMO metadata.xml should be validated at commit-time like our docs are, i.e. a
call to checkxml.pl should be added to commitinfo.

FYI,
neysx@polly ~/gentoo/cvs/gentoo-x86 $ find . -name metadata.xml -exec xmllint
--noout --valid {} \;
./dev-dotnet/edtftpnet/metadata.xml:2: validity error : Validation failed: no
DTD found !
<pkgmetadata>
            ^
./dev-dotnet/monocalendar/metadata.xml:2: validity error : Validation failed:
no DTD found !
<pkgmetadata>
            ^
./dev-util/debugedit/metadata.xml:2: validity error : Validation failed: no DTD
found !
<pkgmetadata>
            ^
./dev-util/mono-debugger/metadata.xml:2: validity error : Validation failed: no
DTD found !
<pkgmetadata>
            ^

I fixed those instances you found Xavier. Once my cvs up of the whole tree
finishes I'll run the same find and fix any other situations.

(In reply to comment #4)
> I fixed those instances you found Xavier. Once my cvs up of the whole tree
> finishes I'll run the same find and fix any other situations.

Thanks. They all validate now.

Created an attachment (id=137127) [edit]
Developer Handbook Metadata.xml Documentation Updates

Developer Handbook Metadata.xml Documentation Updates. I have also cleaned up
some issues the docs team had with the original file and edited some Engrish.

As discussed on g-dev@, could we please hold off and get a GLEP for this?  Have
it discussed so any issues can be ironed out now, rather than later.  Unless
something has changed while I was gone, GLEPs are the process to go through for
introducing changes like this.

Council, could someone please step in here?

I, as a council member, don't see a reason for holding anything off. If someone
wants to write a GLEP, I won't obviously complain, but I don't see the absolute
need for it. As I said before, I leave open to other council members to state
their position.

Created an attachment (id=137138) [edit]
With minor formatting fixes

Devrel, yours to commit, or just say 'yes' :)

(In reply to comment #9)
> I, as a council member, don't see a reason for holding anything off. If someone
> wants to write a GLEP, I won't obviously complain, but I don't see the absolute
> need for it. As I said before, I leave open to other council members to state
> their position.
> 

The whole purpose of a GLEP is to announce, document, and discuss global
changes like this.  There were other metadata changes proposed in GLEPs, and I
don't see why this one should be any different.

Its not a difficult process...you just need to do it so the whole development
community can give suggestions.  I'm not saying you need to incorporate
everyone's suggestions, but you should atleast listen.  This way everyone knows
what is going on.

A GLEP is necessary. Whilst the idea is good, there're some technicalities that
need sorting out, particularly regarding <pkg> and references to other use flag
names, and USE_EXPAND.

(In reply to comment #12)
> A GLEP is necessary. Whilst the idea is good, there're some technicalities that
> need sorting out, particularly regarding <pkg> and references to other use flag
> names, and USE_EXPAND.
> 

Even if GLEP wouldn't be necessary this change should never have gone in
without any discussion on the gentoo-dev mailing list. I agree with Ciaramn
that it would be best to get a GLEP for these changes. If we don't do GLEPs for
big changes then we should remove the whole GLEP process which I don't want to
do. GLEPs are for example very useful during the recruiting process.

(In reply to comment #13)
> (In reply to comment #12)
> > A GLEP is necessary. Whilst the idea is good, there're some technicalities that
> > need sorting out, particularly regarding <pkg> and references to other use flag
> > names, and USE_EXPAND.
> > 
> 
> Even if GLEP wouldn't be necessary this change should never have gone in
> without any discussion on the gentoo-dev mailing list. 

It is being discussed on gentoo-dev, without any interest. The only interest is
in a GLEP. So far everyone has said they agree with the format. Feel free to
bring it up on gentoo-dev if you don't agree with the format.

(In reply to comment #14)
> It is being discussed on gentoo-dev, without any interest. The only interest is
> in a GLEP. So far everyone has said they agree with the format. Feel free to
> bring it up on gentoo-dev if you don't agree with the format.

Er, no, I've said that there're deficiencies in the format that need to be
addressed.

(In reply to comment #15)
> (In reply to comment #14)
> > It is being discussed on gentoo-dev, without any interest. The only interest is
> > in a GLEP. So far everyone has said they agree with the format. Feel free to
> > bring it up on gentoo-dev if you don't agree with the format.
> 
> Er, no, I've said that there're deficiencies in the format that need to be
> addressed.
> 

Please air them out then.

I'm moving this over to council as there seems to be a lot of disagreement.

Another thing to point out, http://www.gentoo.org/proj/en/metastructure/herds/
has an older copy of metadata.xml documentation and should potentially be
removed with a reference pointing to the more up to date Developer Handbook.

(In reply to comment #17)
> I'm moving this over to council as there seems to be a lot of disagreement.
> 

The actual request was that infra validate metadata.xml on commit. This is
regardless any DTD changes. It should be done in the first place. As pointed
out by Xavier and I, there were several bad metadata.xml's in the tree that we
had to find and correct. If you want to remove the load off of infra, repoman
should check your metadata.xml.

Surely that won't need yet another GLEP will it?

(In reply to comment #16)
> Please air them out then.

* The <pkg> element needs formalising better. Does it contain an exact cat/pkg?
Or can it be a full spec with operators? If the latter, is it better done as
<pkg spec="blah" /> to avoid XML issues? Does it have to refer to an existent
package?

* There needs to be a way of referencing other use flags, including on a per
package basis.

* There should probably be a way of referencing categories too.

* How is USE_EXPAND to be handled? How about USE_EXPAND_HIDDEN?

* How is version-specific use flag documentation to be described? Using the
never-formally-approved version hack? Or something nicer? Is IUSE changing
between versions implicitly handled by just ignoring irrelevant descriptions,
or is an explicit range required?

* What's to be done with use.desc and use.local.desc?

None of these are massively at odds with the proposal. But equally, they all
need addressing and most of them require at least small changes.

Now, bear in mind that the above are only the issues that I personally have
noticed when implementing a client that handles the changes (Paludis uses
use.local.desc for --show-use-descriptions). I'd imagine that other issues will
crop up when other people do client implementations.

(In reply to comment #20)
> (In reply to comment #16)
> > Please air them out then.
> 
> * The <pkg> element needs formalising better. Does it contain an exact cat/pkg?
> Or can it be a full spec with operators? If the latter, is it better done as
> <pkg spec="blah" /> to avoid XML issues? Does it have to refer to an existent
> package?

It's merely cat/pkg as was suggested by the packages.gentoo.org guys so that
they could provide a simple link. I believe the GuideXML guys currently use <c>
but they want a <pkg> of their own for that purpose.

> 
> * There needs to be a way of referencing other use flags, including on a per
> package basis.

Please give an example or be more clear.

> 
> * There should probably be a way of referencing categories too.

Explain.

> 
> * How is USE_EXPAND to be handled? How about USE_EXPAND_HIDDEN?

The <use name="???"> is documented as being exactly the names found in the USE
variable. So it works identical to the USE variable.

> 
> * How is version-specific use flag documentation to be described? Using the
> never-formally-approved version hack? Or something nicer? Is IUSE changing
> between versions implicitly handled by just ignoring irrelevant descriptions,
> or is an explicit range required?

The <flag> tag allows the restrict attribute, which was already part of the
pre-existing metadata.dtd and was documented previously.

> 
> * What's to be done with use.desc and use.local.desc?

Nothing as of yet. I would like to see how these features evolve before
proposing a removal of use.local.desc. I, would however need to discuss all
these scenarios with the PMS guys, the Portage guys, the Paludis guys, and the
pkgcore guys because that would be a lot more involved then a little user
documentation.

> 
> None of these are massively at odds with the proposal. But equally, they all
> need addressing and most of them require at least small changes.

Nope. They're all valid questions and I agree they should be answered
appropriately in the documentation for the features.

> 
> Now, bear in mind that the above are only the issues that I personally have
> noticed when implementing a client that handles the changes (Paludis uses
> use.local.desc for --show-use-descriptions). I'd imagine that other issues will
> crop up when other people do client implementations.

The idea was exactly for this use. Think of net-print/cups.. USE=png, what does
that mean? Assuming you have it disabled, does it mean you can't print pngs?
Does it mean the web interface won't use pngs? Does it mean you can't do png
transforms? Not even the maintainers of the net-print/cups package know the
answer. The USE flag was added there long before the current maintainers took
over. This is a standardized place for developers to record what and how each
USE flag affects that package. It's useful for users to see this information as
well. 

Possibly net-print/cups was a bad example since it has a USE=png which prevents
it from linking to libpng, however, it's own hard depends hard dep on libpng so
the USE flag is realistically pointless. But that's another issue/discussion.

as there seems to be a need for further discussion, please do so on the dev
mailinglist

(In reply to comment #22)
> as there seems to be a need for further discussion, please do so on the dev
> mailinglist
> 

Like I've told everyone, feel free to ask questions on the gentoo-dev ML and I
will happily answer them. Additionally, any council questions regarding this,
feel free to ask them on the council ML and I'll direct my replies there as
well.

(In reply to comment #21)
> > * The <pkg> element needs formalising better. Does it contain an exact cat/pkg?
> > Or can it be a full spec with operators? If the latter, is it better done as
> > <pkg spec="blah" /> to avoid XML issues? Does it have to refer to an existent
> > package?
> 
> It's merely cat/pkg as was suggested by the packages.gentoo.org guys so that
> they could provide a simple link. I believe the GuideXML guys currently use <c>
> but they want a <pkg> of their own for that purpose.

See, I consider that to be a rather arbitrary limitation. Allowing a full spec
there still permits links, but it also lets smarter clients provide more
information where appropriate. For example, if a description is "Enables
frozbinate functionality available in <pkg>&gt;=cat/someclient-2</pkg>", a
package manager that, by convention, uses different colours for package names
depending upon whether they're installed, installable or masked (Paludis does
this) could select the appropriate colour.

> > * There needs to be a way of referencing other use flags, including on a per
> > package basis.
> 
> Please give an example or be more clear.

If you have "Enables the frozbinate functionality, which is used by
<pkg>foo/bar</pkg> when USE=baz", it would be nice if clients could display the
baz in different colours depending upon whether it's enabled / disabled /
forced / masked. To do this, the USE flag needs marking and, if it's for a
particular package, it also needs to be associated with that package.

> > * There should probably be a way of referencing categories too.
> 
> Explain.

"Enables support for spell checking, as described in ':help spell.txt'.
Dictionaries are available in the <cat>app-dicts</cat> category."

> > * How is USE_EXPAND to be handled? How about USE_EXPAND_HIDDEN?
> 
> The <use name="???"> is documented as being exactly the names found in the USE
> variable. So it works identical to the USE variable.

Ok. This should be documented explicitly.

> > * How is version-specific use flag documentation to be described? Using the
> > never-formally-approved version hack? Or something nicer? Is IUSE changing
> > between versions implicitly handled by just ignoring irrelevant descriptions,
> > or is an explicit range required?
> 
> The <flag> tag allows the restrict attribute, which was already part of the
> pre-existing metadata.dtd and was documented previously.

OK. And the second part of the question?

There're enough issues here that this really does need a GLEP. It doesn't have
to take long, or sit around for ages (that only happens for bad or
unimplementable ideas, which this isn't; sane proposals can be through very
quickly), but it does need more formalism and discussion.

You are more then welcome to write a GLEP for this. I however am done with
this. Everyone wins. Gentoo loses. Progress forward is stopped as everyone has
requested.

That's rather childish of you. You're saying things either get done exactly the
way you initially proposed, even though you acknowledge that there are
improvements that can be made, or you throw your toys out of the pram and do
nothing at all?

removing myself as I'm not interested in the bug.  Please don't re-add me.

(In reply to comment #19)
> 
> The actual request was that infra validate metadata.xml on commit. This is
> regardless any DTD changes. It should be done in the first place. As pointed
> out by Xavier and I, there were several bad metadata.xml's in the tree that we
> had to find and correct. If you want to remove the load off of infra, repoman
> should check your metadata.xml.
> 

repoman does check metadata.xml as long as xmllint is installed

Re-opening

re-assigning to infrastructure, as the original request was to validate the xml
with a server side commit hook.

Resolving as LATER (not WONTFIX) as this bug depends upon a GLEP being written
or a council decision (or both).

I know you guys love the bugspam.

I'd be glad if you did NOT remove me from CC without asking first. You seem to
have problems to understand how CC works, don't you?

For what concerns this bug, if the point is validation, I doubt that _that_
needs a GLEP. You need validation with or without the USE documentation in the
file.

If you want to handle the bug correctly, report a new one for the validation,
and assign THAT to infra, while leaving this assigned to devrel for the update
for the documentation, and resolve it as LATER.

Somebody please take this bug away from infra please.. 
We don't care about any of the details till we actually need to change
something.. 

fine ;)

Created an attachment (id=160085) [edit]
Formating fixes from previous patch

This patch contains the formating and grammatical fixes from the previous
patch. This patch is only provided to clean up the handbook and does not
contain any new bits related to GLEP 56 and metadata changes.

Created an attachment (id=160087) [edit]
GLEP 56 changes

This patch contains the necessary changes to implement GLEP 56 in full in
accordance to it's final version. The patch is against current CVS.

Created an attachment (id=160144) [edit]
repoman's utilities.py patch to implement GLEP 56

Implements a function to get USE flag info from metadata.xml. It does not
handle the restrict attribute currently.

Pardon the crappy code, this is the first real python outside of hello world
I've even done.

Created an attachment (id=160145) [edit]
repoman changes to implement GLEP 56

Changes necessary to use the above patch's function in repoman

CCing dev-portage for the Portage changes, docs-team for the DTD changes, and
devrel for the Developer Handbook changes

Created an attachment (id=160377) [edit]
GLEP 56 changes 

Previous patch was a handbook patch badly named.. doh.

DTD changes committed.

(In reply to comment #35)
> Created an attachment (id=160085) [edit]
> Formating fixes from previous patch
> 
> This patch contains the formating and grammatical fixes from the previous
> patch. This patch is only provided to clean up the handbook and does not
> contain any new bits related to GLEP 56 and metadata changes.
> 

formatting patch applied. Please add us back when there is something else to
update.

(In reply to comment #41)
> DTD changes committed.

removing docs-team@, please add us back when you need anything else.

Created an attachment (id=160458) [edit]
GLEP 56 changes to handbook

Here are the changes to the Handbook for GLEP 56

I've attached the missing patch to the Gentoo Developer's Handbook for devrel
to commit.

Created an attachment (id=160472) [edit]
GLEP 56 changes to handbook

Correct patches to the Gentoo Developer Handbook

Created an attachment (id=160476) [edit]
GLEP 56 changes to handbook

Third time is a charm. I've been uploading the wrong damn file the whole
time... over and over and over.

(In reply to comment #47)
> Created an attachment (id=160476) [edit]
> GLEP 56 changes to handbook
> 
> Third time is a charm. I've been uploading the wrong damn file the whole
> time... over and over and over.
> 

committed

(In reply to comment #38)
> Created an attachment (id=160145) [edit]
> repoman changes to implement GLEP 56
> 
> Changes necessary to use the above patch's function in repoman
> 

Thanks, those patches are in portage trunk r11126.

The entire tree has been converted. This bug is a wrap.