Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 532054 - [arch-tools] Utils currently broken as pybugz dropped python2 support
Summary: [arch-tools] Utils currently broken as pybugz dropped python2 support
Status: RESOLVED FIXED
Alias: None
Product: Gentoo Linux
Classification: Unclassified
Component: Current packages (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Paweł Hajdan, Jr. (RETIRED)
URL:
Whiteboard:
Keywords:
: 489618 532370 (view as bug list)
Depends on: 532368
Blocks:
  Show dependency tree
 
Reported: 2014-12-09 11:35 UTC by Pacho Ramos
Modified: 2015-01-05 14:44 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
0001-Port-most-tools-to-python-3.patch (0001-Port-most-tools-to-python-3.patch,13.65 KB, patch)
2014-12-13 20:20 UTC, William Hubbs
Details | Diff
0001-Port-most-tools-to-python-3.patch (0001-Port-most-tools-to-python-3.patch,18.72 KB, patch)
2014-12-17 20:51 UTC, William Hubbs
Details | Diff
0001-Port-most-tools-to-python-3.patch (0001-Port-most-tools-to-python-3.patch,18.91 KB, patch)
2014-12-18 20:31 UTC, William Hubbs
Details | Diff
0002-Fix-subprocess-handling-in-batch-stabilize.patch (0002-Fix-subprocess-handling-in-batch-stabilize.patch,1.47 KB, patch)
2014-12-23 16:31 UTC, William Hubbs
Details | Diff
0003-batch-stabilize-use-include-arches-option.patch (0003-batch-stabilize-use-include-arches-option.patch,954 bytes, patch)
2014-12-30 17:08 UTC, William Hubbs
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Pacho Ramos gentoo-dev 2014-12-09 11:35:55 UTC
Not sure what would be better, if try to revert back python2 support to pybugz or try to port the utils to python3 :(
Comment 1 Pacho Ramos gentoo-dev 2014-12-09 11:36:37 UTC
$ stabilization-candidates 
Traceback (most recent call last):
  File "/usr/bin/stabilization-candidates", line 14, in <module>
    from bugz.bugzilla import BugzillaProxy
ImportError: No module named bugz.bugzilla

Well, that is the error because python2 is not supported by pybugz anymore
Comment 2 Pacho Ramos gentoo-dev 2014-12-09 11:37:11 UTC
  30 Aug 2014; William Hubbs <williamh@gentoo.org> pybugz-9999.ebuild:
  only support python-3.3 or newer due to upstream bugs in python 2.x

And that looks to be the reason for dropping python2 support
Comment 3 William Hubbs gentoo-dev 2014-12-13 18:35:05 UTC
This should not be an issue for pybugz-0.11.1, but it will be for 0.12, I'm checking now.
Comment 4 William Hubbs gentoo-dev 2014-12-13 18:43:57 UTC
(In reply to William Hubbs from comment #3)
> This should not be an issue for pybugz-0.11.1, but it will be for 0.12, I'm
> checking now.

This statement is incorrect, pybugz-0.11.x is in fact py3 only. I'll start working on porting arch-tools.
Comment 5 William Hubbs gentoo-dev 2014-12-13 20:20:35 UTC
Created attachment 391598 [details, diff]
0001-Port-most-tools-to-python-3.patch

This patch ports all tools except bugzilla-viewer to python 3.

@ago, @pacho:
Please test and report back.

Thanks,

William
Comment 6 Pacho Ramos gentoo-dev 2014-12-13 20:36:08 UTC
I am not sure if this could be handled too -> bug 532370
Comment 7 William Hubbs gentoo-dev 2014-12-13 22:50:23 UTC
(In reply to Pacho Ramos from comment #6)
> I am not sure if this could be handled too -> bug 532370

Sure, but that's not relevant to this bug. All this patch does is port the current code to python 3.
Comment 8 Pacho Ramos gentoo-dev 2014-12-14 13:08:19 UTC
OK :)

- stabilization-candidates -> Looks to work ok
- file-stabilization-bugs -> Looks to not :(
$ file-stabilization-bugs --repo=/home/pacho/gentoo-x86
You will be prompted for your Gentoo Bugzilla username and password (https://bugs.gentoo.org/xmlrpc.cgi).
Bugzilla Username: pacho@gentoo.org
Password: 
Traceback (most recent call last):
  File "/usr/bin/file-stabilization-bugs", line 77, in <module>
    url = 'http://packages.gentoo.org/package/%s?arches=linux' % urllib.parse.quote(cp)
NameError: name 'urllib' is not defined

Please note that maybe the url stuff could/should be dropped -> bug 532368

- batch-stabilize: looks like cvs gets defunct when running it (but not sure if it's related with this):
$ batch-stabilize -i lista --arch=amd64 --repo=/home/pacho/gentoo-x86
You will be prompted for your Gentoo Bugzilla username and password (https://bugs.gentoo.org/xmlrpc.cgi).
Bugzilla Username: pacho@gentoo.org
Password: 
Working on bug 532542...
Working in /home/pacho/gentoo-x86/dev-python/kiwi...
Running ['cvs', 'up', 'dev-python/kiwi'] in /home/pacho/gentoo-x86...
(and nothing more)

$ $ ps axu | grep cvs
pacho     5615  0.0  0.0      0     0 pts/3    Z+   13:53   0:00 [cvs] <defunct>
pacho     5620  0.2  0.0  63256  2764 ?        Ssl  13:53   0:00 ssh: /home/pacho/.ssh/master-pacho@cvs.gentoo.org:22 [mux]
pacho     5632  0.0  0.0   6492  2048 pts/2    S+   13:53   0:00 grep --colour=auto cvs

Running cvs up there manually works ok

- maintainer-timeout -> fails with:
$ maintainer-timeout 
You will be prompted for your Gentoo Bugzilla username and password (https://bugs.gentoo.org/xmlrpc.cgi).
Bugzilla Username: pacho@gentoo.org
Password: 
Traceback (most recent call last):
  File "/usr/bin/maintainer-timeout", line 77, in <module>
    bugzilla.Bug.update(params)
  File "/usr/lib64/python3.4/xmlrpc/client.py", line 1089, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python3.4/xmlrpc/client.py", line 1419, in __request
    verbose=self.__verbose
  File "/usr/lib64/python3.4/site-packages/bugz/bugzilla.py", line 49, in request
    return self.parse_response(resp)
  File "/usr/lib64/python3.4/xmlrpc/client.py", line 1318, in parse_response
    return u.close()
  File "/usr/lib64/python3.4/xmlrpc/client.py", line 656, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 410: "You must log in before using this part of Gentoo's Bugzilla.">

-reverse-dependencies: how is this supposed to work? I have run it with:

$ reverse-dependencies -i /home/pacho/Descargas/lista
$ cat /home/pacho/Descargas/lista
# bug #532560
dev-libs/libtasn1-4.2

$ cat package.rdeps 
$

It's empty... but that lib has reverse dependencies as you can see running grep in the tree
Comment 9 William Hubbs gentoo-dev 2014-12-14 18:21:55 UTC
(In reply to Pacho Ramos from comment #8)
> - file-stabilization-bugs -> Looks to not :(
> $ file-stabilization-bugs --repo=/home/pacho/gentoo-x86
> You will be prompted for your Gentoo Bugzilla username and password
> (https://bugs.gentoo.org/xmlrpc.cgi).
> Bugzilla Username: pacho@gentoo.org
> Password: 
> Traceback (most recent call last):
>   File "/usr/bin/file-stabilization-bugs", line 77, in <module>
>     url = 'http://packages.gentoo.org/package/%s?arches=linux' %
> urllib.parse.quote(cp)
> NameError: name 'urllib' is not defined
> 
> Please note that maybe the url stuff could/should be dropped -> bug 532368

There is a patch for it on the above-mentioned bug. Please reply there if it works.

> - batch-stabilize: looks like cvs gets defunct when running it (but not sure
> if it's related with this):
> $ batch-stabilize -i lista --arch=amd64 --repo=/home/pacho/gentoo-x86
> You will be prompted for your Gentoo Bugzilla username and password
> (https://bugs.gentoo.org/xmlrpc.cgi).
> Bugzilla Username: pacho@gentoo.org
> Password: 
> Working on bug 532542...
> Working in /home/pacho/gentoo-x86/dev-python/kiwi...
> Running ['cvs', 'up', 'dev-python/kiwi'] in /home/pacho/gentoo-x86...
> (and nothing more)
> 
> $ $ ps axu | grep cvs
> pacho     5615  0.0  0.0      0     0 pts/3    Z+   13:53   0:00 [cvs]
> <defunct>
> pacho     5620  0.2  0.0  63256  2764 ?        Ssl  13:53   0:00 ssh:
> /home/pacho/.ssh/master-pacho@cvs.gentoo.org:22 [mux]
> pacho     5632  0.0  0.0   6492  2048 pts/2    S+   13:53   0:00 grep
> --colour=auto cvs
> 
> Running cvs up there manually works ok

I'll take a look at this further and let you know.

> - maintainer-timeout -> fails with:
> $ maintainer-timeout 
> You will be prompted for your Gentoo Bugzilla username and password
> (https://bugs.gentoo.org/xmlrpc.cgi).
> Bugzilla Username: pacho@gentoo.org
> Password: 
> Traceback (most recent call last):
>   File "/usr/bin/maintainer-timeout", line 77, in <module>
>     bugzilla.Bug.update(params)
>   File "/usr/lib64/python3.4/xmlrpc/client.py", line 1089, in __call__
>     return self.__send(self.__name, args)
>   File "/usr/lib64/python3.4/xmlrpc/client.py", line 1419, in __request
>     verbose=self.__verbose
>   File "/usr/lib64/python3.4/site-packages/bugz/bugzilla.py", line 49, in
> request
>     return self.parse_response(resp)
>   File "/usr/lib64/python3.4/xmlrpc/client.py", line 1318, in parse_response
>     return u.close()
>   File "/usr/lib64/python3.4/xmlrpc/client.py", line 656, in close
>     raise Fault(**self._stack[0])
> xmlrpc.client.Fault: <Fault 410: "You must log in before using this part of
> Gentoo's Bugzilla.">

This means we are trying to modify or post a bug before we are logged in; I'll look at what is going on in the code here as well.

> -reverse-dependencies: how is this supposed to work? I have run it with:
> 
> $ reverse-dependencies -i /home/pacho/Descargas/lista
> $ cat /home/pacho/Descargas/lista
> # bug #532560
> dev-libs/libtasn1-4.2
> 
> $ cat package.rdeps 
> $
> 
> It's empty... but that lib has reverse dependencies as you can see running
> grep in the tree

This one I'm not sure about. I can tell you that it doesn't talk to bugzilla at all, so, if you want to test it before my patch it would be helpful to me because that would let me know if the issue you are describing exists with the python2 version. If it does, please file a separate bug for it.
Comment 10 Pacho Ramos gentoo-dev 2014-12-15 14:25:48 UTC
(In reply to William Hubbs from comment #9)
> (In reply to Pacho Ramos from comment #8)
> > - file-stabilization-bugs -> Looks to not :(
> > $ file-stabilization-bugs --repo=/home/pacho/gentoo-x86
> > You will be prompted for your Gentoo Bugzilla username and password
> > (https://bugs.gentoo.org/xmlrpc.cgi).
> > Bugzilla Username: pacho@gentoo.org
> > Password: 
> > Traceback (most recent call last):
> >   File "/usr/bin/file-stabilization-bugs", line 77, in <module>
> >     url = 'http://packages.gentoo.org/package/%s?arches=linux' %
> > urllib.parse.quote(cp)
> > NameError: name 'urllib' is not defined
> > 
> > Please note that maybe the url stuff could/should be dropped -> bug 532368
> 
> There is a patch for it on the above-mentioned bug. Please reply there if it
> works.
> 

I get further but:
$ file-stabilization-bugs --repo=/home/pacho/gentoo-x86
You will be prompted for your Gentoo Bugzilla username and password (https://bugs.gentoo.org/xmlrpc.cgi).
Bugzilla Username: pacho@gentoo.org
Password: 
<Fault 410: "You must log in before using this part of Gentoo's Bugzilla.">
Failed to submit bug for app-accessibility/emacspeak-39.0-r2. :-(

I guess it's because of the inner problem causing the similar failures in some other scripts :/

[...] 
> > -reverse-dependencies: how is this supposed to work? I have run it with:
> > 
> > $ reverse-dependencies -i /home/pacho/Descargas/lista
> > $ cat /home/pacho/Descargas/lista
> > # bug #532560
> > dev-libs/libtasn1-4.2
> > 
> > $ cat package.rdeps 
> > $
> > 
> > It's empty... but that lib has reverse dependencies as you can see running
> > grep in the tree
> 
> This one I'm not sure about. I can tell you that it doesn't talk to bugzilla
> at all, so, if you want to test it before my patch it would be helpful to me
> because that would let me know if the issue you are describing exists with
> the python2 version. If it does, please file a separate bug for it.

Yeah, it's the same -> bug 532646 I am not sure if I should CC you there... anyway, you can find existing bugs simply looking for "arch-tools" ;)

Thanks!
Comment 11 William Hubbs gentoo-dev 2014-12-17 20:51:18 UTC
Created attachment 391888 [details, diff]
0001-Port-most-tools-to-python-3.patch

This is the latest version of the patch to port most of these tools to
python-3.x.

This also includes all fixes from bug #532370.

Please test and report back.

Thanks,

William
Comment 12 William Hubbs gentoo-dev 2014-12-17 21:19:15 UTC
*** Bug 532370 has been marked as a duplicate of this bug. ***
Comment 13 William Hubbs gentoo-dev 2014-12-17 22:15:19 UTC
*** Bug 489618 has been marked as a duplicate of this bug. ***
Comment 14 Pacho Ramos gentoo-dev 2014-12-18 20:16:32 UTC
- batch-stabilize -> keeps causing cvs to get defunct
- reverse-dependencies -> keeps generating only an empty file... but we have a -- - separate bug for it anyway ;)
- file-stabilization-bugs -> works fine!
- maintainer-timeout -> works fine
- stabilization-candidates -> works fine

Nice, seems that most useful tools are now fixed :D
Comment 15 Pacho Ramos gentoo-dev 2014-12-18 20:17:09 UTC
(needs to be done with bug 532368)
Comment 16 William Hubbs gentoo-dev 2014-12-18 20:31:27 UTC
Created attachment 391988 [details, diff]
0001-Port-most-tools-to-python-3.patch

This patch merges in the fix for bug #532368 and includes everyting
referred to in comment #14.
Comment 17 William Hubbs gentoo-dev 2014-12-23 16:31:47 UTC
Created attachment 392288 [details, diff]
0002-Fix-subprocess-handling-in-batch-stabilize.patch

This patch is an attempt to fix the cvs issue described above; it should
be applied on top of the patch in the previous comment.

Thanks,

William
Comment 18 Pacho Ramos gentoo-dev 2014-12-26 11:08:37 UTC
For some reason the first letter is dropped from the category of the package ("ev-libs" instead of dev-libs):
$ batch-stabilize -i /home/pacho/Descargas/lista --arch=amd64 --repo=/home/pacho/gentoo-x86
You will be prompted for your Gentoo Bugzilla username and password (https://bugs.gentoo.org/xmlrpc.cgi).
Bugzilla Username: pacho@gentoo.org
Password: 
/home/pacho/gentoo-x86/ev-libs/libtasn1/libtasn1-4.2.ebuild: file does not exist
Sanity check failed. Please make sure your CVS repo is up to date (cvs up).

$ cat /home/pacho/Descargas/lista
# bug #532560
dev-libs/libtasn1-4.2

Thanks and sorry for the delay, I cannot reply so fast during this days :)
Comment 19 Agostino Sarubbo gentoo-dev 2014-12-26 11:16:56 UTC
(In reply to Pacho Ramos from comment #18)
> For some reason the first letter is dropped from the category of the package
> ("ev-libs" instead of dev-libs)

That's fine and expected. Try to use =dev-libs/libtasn1-4.2
Comment 20 Pacho Ramos gentoo-dev 2014-12-26 11:33:10 UTC
ah, nice, with the "=" now:
reverse-dependencies -> works ok :)
batch-stabilize -> the same :D

I thought the script were going to drop "=", not any "first" character but... no problem ;)

I didn't know the --ignore-arches option from repoman that batch-stabilize uses... looks like it makes it go really fast (I expect it will still fail if in cases dependencies of the package to be stabilized are incomplete)
Comment 21 Agostino Sarubbo gentoo-dev 2014-12-26 11:41:54 UTC
(In reply to Pacho Ramos from comment #20)
> ah, nice, with the "=" now:
> reverse-dependencies -> works ok :)
> batch-stabilize -> the same :D
> 
> I thought the script were going to drop "=", not any "first" character
> but... no problem ;)
> 
> I didn't know the --ignore-arches option from repoman that batch-stabilize
> uses... looks like it makes it go really fast (I expect it will still fail
> if in cases dependencies of the package to be stabilized are incomplete)

--ignore-arches is wrong.

see http://blogs.gentoo.org/ago/2013/05/05/new-repoman-option-include-arches-the-benefit-for-an-arch-team-member/
Comment 22 William Hubbs gentoo-dev 2014-12-30 17:08:17 UTC
Created attachment 392704 [details, diff]
0003-batch-stabilize-use-include-arches-option.patch

@ago:
This patch should be applied on top of the other two in this bug. It
changes batch-stabilize to use the --include-arches option of repoman.

Let me know if this works.
Comment 23 Agostino Sarubbo gentoo-dev 2014-12-30 17:47:29 UTC
(In reply to William Hubbs from comment #22)
> Created attachment 392704 [details, diff] [details, diff]
> 0003-batch-stabilize-use-include-arches-option.patch
> 
> @ago:
> This patch should be applied on top of the other two in this bug. It
> changes batch-stabilize to use the --include-arches option of repoman.
> 
> Let me know if this works.

Since Pawel did not accept my changes, I created my own version of the script...but your change is the same I did.
Comment 24 William Hubbs gentoo-dev 2014-12-30 18:12:51 UTC
@phajdan.jr:
Please apply the last three patches in this bug to arch-tools.

Also, since there appears to be a team working on these scripts, would
you please consider giving myself and ago write access to the git
repository, or making it writable by all Gentoo developers?

Thanks much,

William
Comment 25 Pacho Ramos gentoo-dev 2014-12-31 10:26:19 UTC
@ago, are there any more changes in arch-tools that would be interesting to share with others? :)
Comment 26 Agostino Sarubbo gentoo-dev 2014-12-31 10:51:16 UTC
(In reply to Pacho Ramos from comment #25)
> @ago, are there any more changes in arch-tools that would be interesting to
> share with others? :)

Atm I use batch-stabilize only for the interaction with cvs. Since the interaction with the bugzilla is half broken ( bug 488848 ) I removed this part and I use my own version in bash. So I have a wrapper called stabilize, which calls first the batch-stabilize and then the script which writes on the bugzilla.
In this manner I avoid to put everytime my passwd.

I hope we can improve batch-stabilize in the future to have a single tool that does everything.
Comment 27 William Hubbs gentoo-dev 2015-01-03 06:57:13 UTC
(In reply to Agostino Sarubbo from comment #26)
@ago:
I will respond on the bug you reference, but the interaction between Bugzilla and batch-stabilize is not broken once you apply all of the patches on this bug. If you want a separate feature, that is another issue.
Comment 28 Paweł Hajdan, Jr. (RETIRED) gentoo-dev 2015-01-05 14:44:14 UTC
(In reply to William Hubbs from comment #24)
> @phajdan.jr:
> Please apply the last three patches in this bug to arch-tools.

Done. Sorry about the delays. Thanks a lot for the patches!