Bugzilla Bug 162493

When I run any glsa-check query that reads 200701-12 it chokes on 
the unicode in someone's name. 


Reproducible: Always

Steps to Reproduce:
1. glsa-check -l 200701-12

Actual Results:  
% glsa-check -l 200701-12                                                      
     chrisg@AmonDin:/home/chrisg
[A] means this GLSA was already applied,
[U] means the system is not affected and
[N] indicates that the system might be affected.

Traceback (most recent call last):
  File "/usr/bin/glsa-check", line 205, in ?
    sys.exit(summarylist(glsalist))
  File "/usr/bin/glsa-check", line 171, in summarylist
    myglsa = Glsa(myid, glsaconfig)
  File "/usr/lib/gentoolkit/pym/glsa.py", line 414, in __init__
    self.read()
  File "/usr/lib/gentoolkit/pym/glsa.py", line 432, in read
    self.parse(urllib.urlopen(myurl))
  File "/usr/lib/gentoolkit/pym/glsa.py", line 470, in parse
    self.description = getText(myroot.getElementsByTagName("description")[0],
format="xml")
  File "/usr/lib/gentoolkit/pym/glsa.py", line 233, in getText
    return str(rValue)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position 8:
ordinal not in range(128)


Expected Results:  
glsa-check should handle unicode properly.

In my environment the following 'quick fix' avoids the crash but replaces the
two characters with question marks. I suspect this is not the 'correct' fix. 

% diff -c /usr/lib/gentoolkit/pym/glsa.py /home/chrisg/temp/                  
chrisg@AmonDin:/home/chrisg/temp
*** /usr/lib/gentoolkit/pym/glsa.py     Wed Jan 17 01:13:07 2007
--- /home/chrisg/temp/glsa.py   Wed Jan 17 02:25:48 2007
***************
*** 230,235 ****
--- 230,236 ----
        if format == "strip":
                rValue = rValue.strip(" \n\t")
                rValue = re.sub("[\s]{2,}", " ", rValue)
+       rValue=rValue.encode('ascii','replace')  # fix to handle unicode input
        return str(rValue)

  def getMultiTagsText(rootnode, tagname, format):

*** Bug 162532 has been marked as a duplicate of this bug. ***

just emerge --sync; we've replaced the offending characters in that GLSA
meanwhile...

Another one....  I think Chris's solution would be quite reasonable.

glsa-check -l 200703-26
[A] means this GLSA was already applied,
[U] means the system is not affected and
[N] indicates that the system might be affected.

Traceback (most recent call last):
  File "/usr/bin/glsa-check", line 212, in ?
    sys.exit(summarylist(glsalist))
  File "/usr/bin/glsa-check", line 172, in summarylist
    myglsa = Glsa(myid, glsaconfig)
  File "/usr/lib/gentoolkit/pym/glsa.py", line 414, in __init__
    self.read()
  File "/usr/lib/gentoolkit/pym/glsa.py", line 432, in read
    self.parse(urllib.urlopen(myurl))
  File "/usr/lib/gentoolkit/pym/glsa.py", line 470, in parse
    self.description = getText(myroot.getElementsByTagName("description")[0],
format="xml")
  File "/usr/lib/gentoolkit/pym/glsa.py", line 233, in getText
    return str(rValue)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position
11: ordinal not in range(128)

A better fix would be changing line 233 to:

return str(rValue.encode('utf-8'));

glsa*.xml is already encoded in utf-8, reencode to ascii (python's default
encoding) would bound to have problems.

Did anyone actually test those changes with a GLSA containing non-Ascii
characters (with all glsa-check operations)? I have to admit that I'm pretty
ignorant when it comes to Unicode issues, so I'm not exactly the most qualified
person to test this.
Two things one should be aware of here:
1) the current conversion mainly exists to ensure that we only pass Ascii
strings into portage as portage does a few type checks that would fail with
Unicode strings resulting in even nastier error messages.
2) in recent versions glsa-check got a new --mail option, if glsa.py would
return strings containing non-ascii characters one would have to make sure that
we set the correct MIME type for mails.

I have the exact same bug, same traceback and everything. Just synced a hour
ago.

just tried the fix provided, works fine. This should probably be committed if
there is some more testing.

Just synced, same problem with glsa-200706-02.xml.
Fix from comment #1 did not work, fix #4 works fine. Please apply this glsa

And another one this morning. This seems like such a simple problem to fix
permanently... 

I'm suffering too on several boxes.

  File "/usr/lib/gentoolkit/pym/glsa.py", line 233, in getText
    return str(rValue)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xf6' in position
34: ordinal not in range(128)

Is there something I've not done - like emerge python with the right unicode
support? If so, I'll do it.

Same problem with glsa-200709-18.xml.

Traceback (most recent call last):
  File "/usr/bin/glsa-check", line 168, in <module>
    myglsa = Glsa(x, glsaconfig)
  File "/usr/lib/gentoolkit/pym/glsa.py", line 441, in __init__
    self.read()
  File "/usr/lib/gentoolkit/pym/glsa.py", line 459, in read
    self.parse(urllib.urlopen(myurl))
  File "/usr/lib/gentoolkit/pym/glsa.py", line 497, in parse
    self.description = getText(myroot.getElementsByTagName("description")[0],
format="xml")
  File "/usr/lib/gentoolkit/pym/glsa.py", line 242, in getText
    return str(rValue)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe9' in position
544: ordinal not in range(128)

*** Bug 194404 has been marked as a duplicate of this bug. ***

Is this ever going to get fixed or are we just going to continue to come back
here every 2-3 months and complain about it?

Agreed. As someone who only updates packages based on the cron output of
glsa-check -l affected, this is quite important to me.
If it stops working, I don't get any notice of security alerts.
They already stopped issuing kernel GLSAs (for some reason I can't fathom) -
but this needs to be solid.
Not all of us have time to read all the bugzilla security entries.

Who's going to change the severity then? :)

Added a few things to my proposed fix at bug 194404 .

All my security scripts fail every time we have some strange glsa entry and i
fail to get report by mail. Leaving my servers vulnerable while thinking i did
my work.

PLEASE some1 fix this and change severity.

Same problem here with gentoolkit-0.2.3-r1

Fixed changing line 233 in "gentoolkit/pym/glsa.py"
from:
return str(rValue)
to:
return rValue.encode('utf-8')

This works fine for me.
Many thanks


(In reply to comment #17)
> Same problem here with gentoolkit-0.2.3-r1
> 
> Fixed changing line 233 in "gentoolkit/pym/glsa.py"
> from:
> return str(rValue)
> to:
> return rValue.encode('utf-8')
> 

(In reply to comment #18)
> This works fine for me.
> Many thanks
> 
> 
It will untill you emerge gentoolkit again :-)

Released in gentoolkit-0.2.4_rc2