Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 199788 (glep56) - GLEP56: metadata DTD updates for USE flag descriptions & validate metadata.xml at commit
Summary: GLEP56: metadata DTD updates for USE flag descriptions & validate metadata.xm...
Status: RESOLVED FIXED
Alias: glep56
Product: [OLD] Docs on www.gentoo.org
Classification: Unclassified
Component: Other documents (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Alec Warner
URL:
Whiteboard:
Keywords:
Depends on: 200583 233208 233212
Blocks:
  Show dependency tree
 
Reported: 2007-11-20 15:24 UTC by Doug Goldstein (RETIRED)
Modified: 2008-08-23 06:44 UTC (History)
7 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
metadata DTD diff (metadata.diff,1.88 KB, patch)
2007-11-20 15:24 UTC, Doug Goldstein (RETIRED)
Details | Diff
Developer Handbook Metadata.xml Documentation Updates (hb-guide-metadata.diff,8.14 KB, patch)
2007-11-27 16:49 UTC, Doug Goldstein (RETIRED)
Details | Diff
With minor formatting fixes (hb-guide-metadata.xml.patch,12.40 KB, text/plain)
2007-11-27 17:34 UTC, Xavier Neys (RETIRED)
Details
Formating fixes from previous patch (hb-guide-metadata-formating.patch,3.70 KB, patch)
2008-07-10 21:26 UTC, Doug Goldstein (RETIRED)
Details | Diff
GLEP 56 changes (metadata.dtd.patch,3.70 KB, patch)
2008-07-10 21:42 UTC, Doug Goldstein (RETIRED)
Details | Diff
repoman's utilities.py patch to implement GLEP 56 (repoman-utilities-glep-56.py.patch,1.30 KB, patch)
2008-07-11 21:42 UTC, Doug Goldstein (RETIRED)
Details | Diff
repoman changes to implement GLEP 56 (repoman-glep-56.patch,889 bytes, patch)
2008-07-11 21:42 UTC, Doug Goldstein (RETIRED)
Details | Diff
GLEP 56 changes to metadata.dtd (metadata.dtd.patch,1.08 KB, patch)
2008-07-14 19:43 UTC, Doug Goldstein (RETIRED)
Details | Diff
GLEP 56 changes to handbook (hb-guide-metadata-glep56.patch,4.16 KB, patch)
2008-07-15 15:55 UTC, Doug Goldstein (RETIRED)
Details | Diff
GLEP 56 changes to handbook (hb-guide-metadata-glep56.patch,4.16 KB, patch)
2008-07-15 19:26 UTC, Doug Goldstein (RETIRED)
Details | Diff
GLEP 56 changes to handbook (hb-guide-metadata-glep56.patch,6.35 KB, patch)
2008-07-15 19:30 UTC, Doug Goldstein (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Doug Goldstein (RETIRED) gentoo-dev 2007-11-20 15:24:14 UTC
As discussed on Flameeyes' blog and my blog. It's been hashed out in #gentoo-dev and a lot of us would like to start taking advantage of it.

http://farragut.flameeyes.is-a-geek.org/articles/2007/11/19/lets-actually-get-some-metadata

http://blog.cardoe.com/archives/2007/11/19/use-flag-metadata/
Comment 1 Doug Goldstein (RETIRED) gentoo-dev 2007-11-20 15:24:39 UTC
Created attachment 136492 [details, diff]
metadata DTD diff
Comment 2 Xavier Neys (RETIRED) gentoo-dev 2007-11-20 17:42:29 UTC
DTD patched.

IMO metadata.xml should be validated at commit-time like our docs are, i.e. a call to checkxml.pl should be added to commitinfo.
Comment 3 Xavier Neys (RETIRED) gentoo-dev 2007-11-20 19:08:01 UTC
FYI,
neysx@polly ~/gentoo/cvs/gentoo-x86 $ find . -name metadata.xml -exec xmllint --noout --valid {} \;
./dev-dotnet/edtftpnet/metadata.xml:2: validity error : Validation failed: no DTD found !
<pkgmetadata>
            ^
./dev-dotnet/monocalendar/metadata.xml:2: validity error : Validation failed: no DTD found !
<pkgmetadata>
            ^
./dev-util/debugedit/metadata.xml:2: validity error : Validation failed: no DTD found !
<pkgmetadata>
            ^
./dev-util/mono-debugger/metadata.xml:2: validity error : Validation failed: no DTD found !
<pkgmetadata>
            ^
Comment 4 Doug Goldstein (RETIRED) gentoo-dev 2007-11-22 16:51:38 UTC
I fixed those instances you found Xavier. Once my cvs up of the whole tree finishes I'll run the same find and fix any other situations.
Comment 5 Xavier Neys (RETIRED) gentoo-dev 2007-11-22 17:01:34 UTC
(In reply to comment #4)
> I fixed those instances you found Xavier. Once my cvs up of the whole tree
> finishes I'll run the same find and fix any other situations.

Thanks. They all validate now.
Comment 6 Doug Goldstein (RETIRED) gentoo-dev 2007-11-27 16:49:05 UTC
Created attachment 137127 [details, diff]
Developer Handbook Metadata.xml Documentation Updates

Developer Handbook Metadata.xml Documentation Updates. I have also cleaned up some issues the docs team had with the original file and edited some Engrish.
Comment 7 Mark Loeser (RETIRED) gentoo-dev 2007-11-27 16:55:55 UTC
As discussed on g-dev@, could we please hold off and get a GLEP for this?  Have it discussed so any issues can be ironed out now, rather than later.  Unless something has changed while I was gone, GLEPs are the process to go through for introducing changes like this.
Comment 8 Mark Loeser (RETIRED) gentoo-dev 2007-11-27 17:05:31 UTC
Council, could someone please step in here?
Comment 9 Diego Elio Pettenò (RETIRED) gentoo-dev 2007-11-27 17:31:24 UTC
I, as a council member, don't see a reason for holding anything off. If someone wants to write a GLEP, I won't obviously complain, but I don't see the absolute need for it. As I said before, I leave open to other council members to state their position.
Comment 10 Xavier Neys (RETIRED) gentoo-dev 2007-11-27 17:34:33 UTC
Created attachment 137138 [details]
With minor formatting fixes

Devrel, yours to commit, or just say 'yes' :)
Comment 11 Mark Loeser (RETIRED) gentoo-dev 2007-11-27 18:10:35 UTC
(In reply to comment #9)
> I, as a council member, don't see a reason for holding anything off. If someone
> wants to write a GLEP, I won't obviously complain, but I don't see the absolute
> need for it. As I said before, I leave open to other council members to state
> their position.
> 

The whole purpose of a GLEP is to announce, document, and discuss global changes like this.  There were other metadata changes proposed in GLEPs, and I don't see why this one should be any different.

Its not a difficult process...you just need to do it so the whole development community can give suggestions.  I'm not saying you need to incorporate everyone's suggestions, but you should atleast listen.  This way everyone knows what is going on.
Comment 12 Ciaran McCreesh 2007-11-27 18:43:17 UTC
A GLEP is necessary. Whilst the idea is good, there're some technicalities that need sorting out, particularly regarding <pkg> and references to other use flag names, and USE_EXPAND.
Comment 13 Petteri Räty (RETIRED) gentoo-dev 2007-11-27 18:52:51 UTC
(In reply to comment #12)
> A GLEP is necessary. Whilst the idea is good, there're some technicalities that
> need sorting out, particularly regarding <pkg> and references to other use flag
> names, and USE_EXPAND.
> 

Even if GLEP wouldn't be necessary this change should never have gone in without any discussion on the gentoo-dev mailing list. I agree with Ciaramn that it would be best to get a GLEP for these changes. If we don't do GLEPs for big changes then we should remove the whole GLEP process which I don't want to do. GLEPs are for example very useful during the recruiting process.
Comment 14 Doug Goldstein (RETIRED) gentoo-dev 2007-11-27 19:26:18 UTC
(In reply to comment #13)
> (In reply to comment #12)
> > A GLEP is necessary. Whilst the idea is good, there're some technicalities that
> > need sorting out, particularly regarding <pkg> and references to other use flag
> > names, and USE_EXPAND.
> > 
> 
> Even if GLEP wouldn't be necessary this change should never have gone in
> without any discussion on the gentoo-dev mailing list. 

It is being discussed on gentoo-dev, without any interest. The only interest is in a GLEP. So far everyone has said they agree with the format. Feel free to bring it up on gentoo-dev if you don't agree with the format.
Comment 15 Ciaran McCreesh 2007-11-27 19:30:54 UTC
(In reply to comment #14)
> It is being discussed on gentoo-dev, without any interest. The only interest is
> in a GLEP. So far everyone has said they agree with the format. Feel free to
> bring it up on gentoo-dev if you don't agree with the format.

Er, no, I've said that there're deficiencies in the format that need to be addressed.
Comment 16 Doug Goldstein (RETIRED) gentoo-dev 2007-11-27 19:32:44 UTC
(In reply to comment #15)
> (In reply to comment #14)
> > It is being discussed on gentoo-dev, without any interest. The only interest is
> > in a GLEP. So far everyone has said they agree with the format. Feel free to
> > bring it up on gentoo-dev if you don't agree with the format.
> 
> Er, no, I've said that there're deficiencies in the format that need to be
> addressed.
> 

Please air them out then.
Comment 17 Mike Doty (RETIRED) gentoo-dev 2007-11-27 19:39:53 UTC
I'm moving this over to council as there seems to be a lot of disagreement.
Comment 18 Doug Goldstein (RETIRED) gentoo-dev 2007-11-27 19:42:34 UTC
Another thing to point out, http://www.gentoo.org/proj/en/metastructure/herds/ has an older copy of metadata.xml documentation and should potentially be removed with a reference pointing to the more up to date Developer Handbook.
Comment 19 Doug Goldstein (RETIRED) gentoo-dev 2007-11-27 19:45:13 UTC
(In reply to comment #17)
> I'm moving this over to council as there seems to be a lot of disagreement.
> 

The actual request was that infra validate metadata.xml on commit. This is regardless any DTD changes. It should be done in the first place. As pointed out by Xavier and I, there were several bad metadata.xml's in the tree that we had to find and correct. If you want to remove the load off of infra, repoman should check your metadata.xml.

Surely that won't need yet another GLEP will it?
Comment 20 Ciaran McCreesh 2007-11-27 19:47:30 UTC
(In reply to comment #16)
> Please air them out then.

* The <pkg> element needs formalising better. Does it contain an exact cat/pkg? Or can it be a full spec with operators? If the latter, is it better done as <pkg spec="blah" /> to avoid XML issues? Does it have to refer to an existent package?

* There needs to be a way of referencing other use flags, including on a per package basis.

* There should probably be a way of referencing categories too.

* How is USE_EXPAND to be handled? How about USE_EXPAND_HIDDEN?

* How is version-specific use flag documentation to be described? Using the never-formally-approved version hack? Or something nicer? Is IUSE changing between versions implicitly handled by just ignoring irrelevant descriptions, or is an explicit range required?

* What's to be done with use.desc and use.local.desc?

None of these are massively at odds with the proposal. But equally, they all need addressing and most of them require at least small changes.

Now, bear in mind that the above are only the issues that I personally have noticed when implementing a client that handles the changes (Paludis uses use.local.desc for --show-use-descriptions). I'd imagine that other issues will crop up when other people do client implementations.
Comment 21 Doug Goldstein (RETIRED) gentoo-dev 2007-11-27 20:47:00 UTC
(In reply to comment #20)
> (In reply to comment #16)
> > Please air them out then.
> 
> * The <pkg> element needs formalising better. Does it contain an exact cat/pkg?
> Or can it be a full spec with operators? If the latter, is it better done as
> <pkg spec="blah" /> to avoid XML issues? Does it have to refer to an existent
> package?

It's merely cat/pkg as was suggested by the packages.gentoo.org guys so that they could provide a simple link. I believe the GuideXML guys currently use <c> but they want a <pkg> of their own for that purpose.

> 
> * There needs to be a way of referencing other use flags, including on a per
> package basis.

Please give an example or be more clear.

> 
> * There should probably be a way of referencing categories too.

Explain.

> 
> * How is USE_EXPAND to be handled? How about USE_EXPAND_HIDDEN?

The <use name="???"> is documented as being exactly the names found in the USE variable. So it works identical to the USE variable.

> 
> * How is version-specific use flag documentation to be described? Using the
> never-formally-approved version hack? Or something nicer? Is IUSE changing
> between versions implicitly handled by just ignoring irrelevant descriptions,
> or is an explicit range required?

The <flag> tag allows the restrict attribute, which was already part of the pre-existing metadata.dtd and was documented previously.

> 
> * What's to be done with use.desc and use.local.desc?

Nothing as of yet. I would like to see how these features evolve before proposing a removal of use.local.desc. I, would however need to discuss all these scenarios with the PMS guys, the Portage guys, the Paludis guys, and the pkgcore guys because that would be a lot more involved then a little user documentation.

> 
> None of these are massively at odds with the proposal. But equally, they all
> need addressing and most of them require at least small changes.

Nope. They're all valid questions and I agree they should be answered appropriately in the documentation for the features.

> 
> Now, bear in mind that the above are only the issues that I personally have
> noticed when implementing a client that handles the changes (Paludis uses
> use.local.desc for --show-use-descriptions). I'd imagine that other issues will
> crop up when other people do client implementations.

The idea was exactly for this use. Think of net-print/cups.. USE=png, what does that mean? Assuming you have it disabled, does it mean you can't print pngs? Does it mean the web interface won't use pngs? Does it mean you can't do png transforms? Not even the maintainers of the net-print/cups package know the answer. The USE flag was added there long before the current maintainers took over. This is a standardized place for developers to record what and how each USE flag affects that package. It's useful for users to see this information as well. 

Possibly net-print/cups was a bad example since it has a USE=png which prevents it from linking to libpng, however, it's own hard depends hard dep on libpng so the USE flag is realistically pointless. But that's another issue/discussion.

Comment 22 Markus Ullmann (RETIRED) gentoo-dev 2007-11-27 21:23:29 UTC
as there seems to be a need for further discussion, please do so on the dev mailinglist
Comment 23 Doug Goldstein (RETIRED) gentoo-dev 2007-11-27 21:25:44 UTC
(In reply to comment #22)
> as there seems to be a need for further discussion, please do so on the dev
> mailinglist
> 

Like I've told everyone, feel free to ask questions on the gentoo-dev ML and I will happily answer them. Additionally, any council questions regarding this, feel free to ask them on the council ML and I'll direct my replies there as well.
Comment 24 Ciaran McCreesh 2007-11-27 21:31:17 UTC
(In reply to comment #21)
> > * The <pkg> element needs formalising better. Does it contain an exact cat/pkg?
> > Or can it be a full spec with operators? If the latter, is it better done as
> > <pkg spec="blah" /> to avoid XML issues? Does it have to refer to an existent
> > package?
> 
> It's merely cat/pkg as was suggested by the packages.gentoo.org guys so that
> they could provide a simple link. I believe the GuideXML guys currently use <c>
> but they want a <pkg> of their own for that purpose.

See, I consider that to be a rather arbitrary limitation. Allowing a full spec there still permits links, but it also lets smarter clients provide more information where appropriate. For example, if a description is "Enables frozbinate functionality available in <pkg>&gt;=cat/someclient-2</pkg>", a package manager that, by convention, uses different colours for package names depending upon whether they're installed, installable or masked (Paludis does this) could select the appropriate colour.

> > * There needs to be a way of referencing other use flags, including on a per
> > package basis.
> 
> Please give an example or be more clear.

If you have "Enables the frozbinate functionality, which is used by <pkg>foo/bar</pkg> when USE=baz", it would be nice if clients could display the baz in different colours depending upon whether it's enabled / disabled / forced / masked. To do this, the USE flag needs marking and, if it's for a particular package, it also needs to be associated with that package.

> > * There should probably be a way of referencing categories too.
> 
> Explain.

"Enables support for spell checking, as described in ':help spell.txt'. Dictionaries are available in the <cat>app-dicts</cat> category."

> > * How is USE_EXPAND to be handled? How about USE_EXPAND_HIDDEN?
> 
> The <use name="???"> is documented as being exactly the names found in the USE
> variable. So it works identical to the USE variable.

Ok. This should be documented explicitly.

> > * How is version-specific use flag documentation to be described? Using the
> > never-formally-approved version hack? Or something nicer? Is IUSE changing
> > between versions implicitly handled by just ignoring irrelevant descriptions,
> > or is an explicit range required?
> 
> The <flag> tag allows the restrict attribute, which was already part of the
> pre-existing metadata.dtd and was documented previously.

OK. And the second part of the question?

There're enough issues here that this really does need a GLEP. It doesn't have to take long, or sit around for ages (that only happens for bad or unimplementable ideas, which this isn't; sane proposals can be through very quickly), but it does need more formalism and discussion.
Comment 25 Doug Goldstein (RETIRED) gentoo-dev 2007-11-27 21:46:12 UTC
You are more then welcome to write a GLEP for this. I however am done with this. Everyone wins. Gentoo loses. Progress forward is stopped as everyone has requested.
Comment 26 Ciaran McCreesh 2007-11-27 21:51:32 UTC
That's rather childish of you. You're saying things either get done exactly the way you initially proposed, even though you acknowledge that there are improvements that can be made, or you throw your toys out of the pram and do nothing at all?
Comment 27 Mike Doty (RETIRED) gentoo-dev 2007-11-27 22:05:43 UTC
removing myself as I'm not interested in the bug.  Please don't re-add me.
Comment 28 Petteri Räty (RETIRED) gentoo-dev 2007-11-28 01:19:05 UTC
(In reply to comment #19)
> 
> The actual request was that infra validate metadata.xml on commit. This is
> regardless any DTD changes. It should be done in the first place. As pointed
> out by Xavier and I, there were several bad metadata.xml's in the tree that we
> had to find and correct. If you want to remove the load off of infra, repoman
> should check your metadata.xml.
> 

repoman does check metadata.xml as long as xmllint is installed
Comment 29 Alec Warner (RETIRED) archtester gentoo-dev Security 2007-11-28 03:24:27 UTC
Re-opening
Comment 30 Alec Warner (RETIRED) archtester gentoo-dev Security 2007-11-28 03:25:16 UTC
re-assigning to infrastructure, as the original request was to validate the xml with a server side commit hook.
Comment 31 Alec Warner (RETIRED) archtester gentoo-dev Security 2007-11-28 03:26:14 UTC
Resolving as LATER (not WONTFIX) as this bug depends upon a GLEP being written or a council decision (or both).

I know you guys love the bugspam.
Comment 32 Diego Elio Pettenò (RETIRED) gentoo-dev 2007-11-28 09:40:39 UTC
I'd be glad if you did NOT remove me from CC without asking first. You seem to have problems to understand how CC works, don't you?

For what concerns this bug, if the point is validation, I doubt that _that_ needs a GLEP. You need validation with or without the USE documentation in the file.

If you want to handle the bug correctly, report a new one for the validation, and assign THAT to infra, while leaving this assigned to devrel for the update for the documentation, and resolve it as LATER.
Comment 33 solar (RETIRED) gentoo-dev 2007-11-28 16:11:30 UTC
Somebody please take this bug away from infra please.. 
We don't care about any of the details till we actually need to change something.. 
Comment 34 Alec Warner (RETIRED) archtester gentoo-dev Security 2007-11-30 03:18:47 UTC
fine ;)
Comment 35 Doug Goldstein (RETIRED) gentoo-dev 2008-07-10 21:26:12 UTC
Created attachment 160085 [details, diff]
Formating fixes from previous patch

This patch contains the formating and grammatical fixes from the previous patch. This patch is only provided to clean up the handbook and does not contain any new bits related to GLEP 56 and metadata changes.
Comment 36 Doug Goldstein (RETIRED) gentoo-dev 2008-07-10 21:42:11 UTC
Created attachment 160087 [details, diff]
GLEP 56 changes

This patch contains the necessary changes to implement GLEP 56 in full in accordance to it's final version. The patch is against current CVS.
Comment 37 Doug Goldstein (RETIRED) gentoo-dev 2008-07-11 21:42:03 UTC
Created attachment 160144 [details, diff]
repoman's utilities.py patch to implement GLEP 56

Implements a function to get USE flag info from metadata.xml. It does not handle the restrict attribute currently.

Pardon the crappy code, this is the first real python outside of hello world I've even done.
Comment 38 Doug Goldstein (RETIRED) gentoo-dev 2008-07-11 21:42:46 UTC
Created attachment 160145 [details, diff]
repoman changes to implement GLEP 56

Changes necessary to use the above patch's function in repoman
Comment 39 Doug Goldstein (RETIRED) gentoo-dev 2008-07-11 21:53:41 UTC
CCing dev-portage for the Portage changes, docs-team for the DTD changes, and devrel for the Developer Handbook changes
Comment 40 Doug Goldstein (RETIRED) gentoo-dev 2008-07-14 19:43:54 UTC
Created attachment 160377 [details, diff]
GLEP 56 changes to metadata.dtd

Previous patch was a handbook patch badly named.. doh.
Comment 41 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2008-07-14 19:49:04 UTC
DTD changes committed.
Comment 42 Petteri Räty (RETIRED) gentoo-dev 2008-07-15 15:34:18 UTC
(In reply to comment #35)
> Created an attachment (id=160085) [edit]
> Formating fixes from previous patch
> 
> This patch contains the formating and grammatical fixes from the previous
> patch. This patch is only provided to clean up the handbook and does not
> contain any new bits related to GLEP 56 and metadata changes.
> 

formatting patch applied. Please add us back when there is something else to update.
Comment 43 Jan Kundrát (RETIRED) gentoo-dev 2008-07-15 15:54:23 UTC
(In reply to comment #41)
> DTD changes committed.

removing docs-team@, please add us back when you need anything else.

Comment 44 Doug Goldstein (RETIRED) gentoo-dev 2008-07-15 15:55:17 UTC
Created attachment 160458 [details, diff]
GLEP 56 changes to handbook

Here are the changes to the Handbook for GLEP 56
Comment 45 Doug Goldstein (RETIRED) gentoo-dev 2008-07-15 15:56:23 UTC
I've attached the missing patch to the Gentoo Developer's Handbook for devrel to commit.
Comment 46 Doug Goldstein (RETIRED) gentoo-dev 2008-07-15 19:26:19 UTC
Created attachment 160472 [details, diff]
GLEP 56 changes to handbook

Correct patches to the Gentoo Developer Handbook
Comment 47 Doug Goldstein (RETIRED) gentoo-dev 2008-07-15 19:30:19 UTC
Created attachment 160476 [details, diff]
GLEP 56 changes to handbook

Third time is a charm. I've been uploading the wrong damn file the whole time... over and over and over.
Comment 48 Petteri Räty (RETIRED) gentoo-dev 2008-07-15 19:32:33 UTC
(In reply to comment #47)
> Created an attachment (id=160476) [edit]
> GLEP 56 changes to handbook
> 
> Third time is a charm. I've been uploading the wrong damn file the whole
> time... over and over and over.
> 

committed
Comment 49 Zac Medico gentoo-dev 2008-07-18 12:42:31 UTC
(In reply to comment #38)
> Created an attachment (id=160145) [edit]
> repoman changes to implement GLEP 56
> 
> Changes necessary to use the above patch's function in repoman
> 

Thanks, those patches are in portage trunk r11126.
Comment 50 Doug Goldstein (RETIRED) gentoo-dev 2008-08-23 06:44:18 UTC
The entire tree has been converted. This bug is a wrap.