[ebuild U ] app-admin/ulogd-2.0.5_p20161017 [2.0.5_p20160205] USE="dbi doc* json mysql nfacct nfct nflog pcap postgres sqlite ulog" make: Entering directory '/var/tmp/portage/app-admin/ulogd-2.0.5_p20161017/work/ulogd-2.0.5_p20161017/doc' sgml2txt ulogd.sgml Processing file ulogd.sgml nsgmls:<OSFD>0:190:12:E: end tag for "DESCRIP" omitted, but its declaration does not permit this nsgmls:<OSFD>0:182:0: start tag was here nsgmls:<OSFD>0:190:12:E: end tag for "ARTICLE" omitted, but its declaration does not permit this nsgmls:<OSFD>0:3:0: start tag was here LinuxdocTools::process_file: nsgmls failed with exit status 1. Aborting ... Makefile:2: recipe for target 'all' failed make: *** [all] Error 25
Created attachment 460874 [details] build.log
Created attachment 460876 [details] emerge --info
Does cat /var/tmp/portage/app-admin/ulogd-2.0.5_p20161017/work/ulogd-2.0.5_p20161017/doc/ulogd.sgml | sed 's/_/\\_/g' | /usr/bin/sgmlpre output=txt | ( nsgmls -D /usr/share/sgml -D /usr/share/linuxdoc-tools -ifmttxt ) > /dev/null command work?
No response. Can't reproduce.
I'm sorry. I meant to do this. I had the bug report on my computer for me to take care of. I completely forgot. nsgmls:<OSFD>0:190:5:E: end tag for "DESCRIP" omitted, but its declaration does not permit this nsgmls:<OSFD>0:182:0: start tag was here nsgmls:<OSFD>0:190:5:E: end tag for "ARTICLE" omitted, but its declaration does not permit this nsgmls:<OSFD>0:3:0: start tag was here
(In reply to deference from comment #5) > I'm sorry. I meant to do this. I had the bug report on my computer for me to > take care of. I completely forgot. > nsgmls:<OSFD>0:190:5:E: end tag for "DESCRIP" omitted, but its declaration > does not permit this > nsgmls:<OSFD>0:182:0: start tag was here > nsgmls:<OSFD>0:190:5:E: end tag for "ARTICLE" omitted, but its declaration > does not permit this > nsgmls:<OSFD>0:3:0: start tag was here This is the error when running command from comment #3, right?
Yes.
(In reply to deference from comment #7) > Yes. Ok. Thanks. Now let's find where this pipeline fails. Please execute the following commands and paste links into a comment here: cat /var/tmp/portage/app-admin/ulogd-2.0.5_p20161017/work/ulogd-2.0.5_p20161017/doc/ulogd.sgml | wgetpaste cat /var/tmp/portage/app-admin/ulogd-2.0.5_p20161017/work/ulogd-2.0.5_p20161017/doc/ulogd.sgml | sed 's/_/\\_/g' | wgetpaste cat /var/tmp/portage/app-admin/ulogd-2.0.5_p20161017/work/ulogd-2.0.5_p20161017/doc/ulogd.sgml | sed 's/_/\\_/g' | /usr/bin/sgmlpre output=txt | wgetpaste
Cmd 1: http://bpaste.net/raw/0d50e2d86f93 Cmd 2: http://bpaste.net/raw/cd0fd08f161a Cmd 3: http://bpaste.net/raw/9bc9839cf458 Thanks
(In reply to deference from comment #9) > Cmd 1: http://bpaste.net/raw/0d50e2d86f93 > Cmd 2: http://bpaste.net/raw/cd0fd08f161a > Cmd 3: http://bpaste.net/raw/9bc9839cf458 > Thanks For the last couple of days I've been trying to look at your links, but bpaste.net seem to be malfunctioning: ~ $ wget --tries=3 http://bpaste.net/raw/9bc9839cf458 URL transformed to HTTPS due to an HSTS policy --2017-05-01 20:54:19-- https://bpaste.net/raw/9bc9839cf458 Resolving bpaste.net (bpaste.net)... 149.202.95.76 Connecting to bpaste.net (bpaste.net)|149.202.95.76|:443... failed: Connection timed out. Retrying. --2017-05-01 20:56:32-- (try: 2) https://bpaste.net/raw/9bc9839cf458 Connecting to bpaste.net (bpaste.net)|149.202.95.76|:443... failed: Connection timed out. Retrying. --2017-05-01 20:58:44-- (try: 3) https://bpaste.net/raw/9bc9839cf458 Connecting to bpaste.net (bpaste.net)|149.202.95.76|:443... failed: Connection timed out. Giving up. ~ $ wget --tries=3 https://gentoo.org --2017-05-01 21:05:01-- https://gentoo.org/ Resolving gentoo.org (gentoo.org)... 89.16.167.134 Connecting to gentoo.org (gentoo.org)|89.16.167.134|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 23996 (23K) [text/html] Saving to: ‘index.html’ index.html 100%[=========================>] 23,43K --.-KB/s in 0,004s 2017-05-01 21:05:01 (5,13 MB/s) - ‘index.html’ saved [23996/23996] Could you please upload those to a more reliable pastebin?
I verified these links are good after pasting: http://dpaste.com/3JD6NJH http://dpaste.com/3KWD054 http://dpaste.com/3JD6NJH Thanks again.
(In reply to deference from comment #11) > I verified these links are good after pasting: > http://dpaste.com/3JD6NJH > http://dpaste.com/3KWD054 > http://dpaste.com/3JD6NJH > Thanks again. Something is likely wrong. Two links are the same.
(In reply to Coacher from comment #12) > Something is likely wrong. Two links are the same. To clarify: I expect 3 different links, but there are only 2 different links.
Sorry, clipboard error. 1. http://dpaste.com/2111XZ2 2. http://dpaste.com/2HFRQ74 3. http://dpaste.com/2VRNWXG
Your files are correct and nsgmls successfully processes the last one. There's no problem with file generation. It seems your issue occurs because nsgmls at the end of the pipeline from comment #3 receives only the first 8192 bytes of the output from previous commands instead of the whole file. Do you have any ideas why this could be happening? Do you have some resource limits set up?
*** Bug 599948 has been marked as a duplicate of this bug. ***
I don't want you to think that I'm dispersing on you again. I don't remember ever setting any limits on anything except sudo, but I needed to rebuild my system because of a gcc upgrade, so I created a new partition, chrooted and am rebuilding. If this persists then it is because of something I did not do, but I've not yet gotten to the point where I install ulogd.
(In reply to deference from comment #17) > I don't want you to think that I'm dispersing on you again. > I don't remember ever setting any limits on anything except sudo, but I > needed to rebuild my system because of a gcc upgrade, so I created a new > partition, chrooted and am rebuilding. If this persists then it is because > of something I did not do, but I've not yet gotten to the point where I > install ulogd. I don't think so, don't worry :) Please see upstream bugreport in ${URL}. It would be great if you could provide info requested in these comments: https://gitlab.com/agmartin/linuxdoc-tools/issues/10#note_30112403 https://gitlab.com/agmartin/linuxdoc-tools/issues/10#note_30302841 Instead of BrlAPI.sgml you can use ulogd.sgml. If you don't want to register on gitlab feel free to send the results to me. I'll attach them there.
Created attachment 480596 [details] out.3 OK, I rebuild my system. I'm using the same locale, timezone, hosts, /etc/portage/FOO, and login.defs files. I am still getting the same error. As requested attached is out.3. I'll add the others in subsequent comments.
Created attachment 480598 [details] strace.cat
Created attachment 480602 [details] out strace cat
(In reply to deference from comment #19) > Created attachment 480596 [details] > out.3 > > OK, I rebuild my system. I'm using the same locale, timezone, hosts, > /etc/portage/FOO, and login.defs files. > I am still getting the same error. > As requested attached is out.3. > I'll add the others in subsequent comments. Thank you. Looks like onsgmls on your system truncates input. This means that the problem is likely in app-text/opensp package.
Should I file a new bug for that package, and if so, how do I make this bug depend on that one?
(In reply to deference from comment #23) > Should I file a new bug for that package, and if so, how do I make this bug > depend on that one? You should open a new bug only if the same truncation as seen in comment #19 happens with other sgml files. I'll take care of the bug dependencies then.
*** Bug 624488 has been marked as a duplicate of this bug. ***
Yarda, thank you for the report. Please attach out.3 from `cat /var/tmp/portage/app-text/linuxdoc-tools-0.9.72/work/linuxdoc-tools-0.9.72/doc/guide.sgml | onsgmls > out.3`
Created attachment 482990 [details] cat guide.sgml | onsgmls > out.3
(In reply to Coacher from comment #26) > Yarda, thank you for the report. > > Please attach out.3 from > `cat > /var/tmp/portage/app-text/linuxdoc-tools-0.9.72/work/linuxdoc-tools-0.9.72/ > doc/guide.sgml | onsgmls > out.3` Thanks for prompt response, output attached. I also tried to rebuild app-text/opensp-1.5.2-r3 and also update to app-test/opensp-1.5.2-r4, but it didn't help.
I am not sure whether my problem is the same, because after some debugging it seems the tool doesn't truncate the pipe input. For me the problem seems to be in app-text/docbook-sgml-dtd. I fixed it by uninstallation of the old DTDs: # emerge -C app-text/docbook-sgml-dtd:3.0 app-text/docbook-sgml-dtd:3.1 app-text/docbook-sgml-dtd:4.0 app-text/docbook-sgml-dtd:4.1 And installation of the latest version: # emerge app-text/docbook-sgml-dtd which installed app-text/docbook-sgml-dtd-4.4 which wasn't previously there (emerge -uDN world didn't updated it). Now it correctly emerges app-text/linuxdoc-tools with the USE=doc.
(In reply to Yarda from comment #29) > And installation of the latest version: > > # emerge app-text/docbook-sgml-dtd > > which installed app-text/docbook-sgml-dtd-4.4 which wasn't previously there > (emerge -uDN world didn't updated it). Just manual upgrade to app-text/docbook-sgml-dtd-4.4 is not enough, from the strace it seems the tool is still using the old DTDs, I had to remove the old DTDs for onsgmls to start working.
(In reply to Yarda from comment #30) > (In reply to Yarda from comment #29) > > And installation of the latest version: > > > > # emerge app-text/docbook-sgml-dtd > > > > which installed app-text/docbook-sgml-dtd-4.4 which wasn't previously there > > (emerge -uDN world didn't updated it). > > Just manual upgrade to app-text/docbook-sgml-dtd-4.4 is not enough, from the > strace it seems the tool is still using the old DTDs, I had to remove the > old DTDs for onsgmls to start working. I am not sure whether it's real solution or just sideeffect that it compiled because all the removed DTDs seems to be needed on my system: # emerge -uDNpv --tree world [nomerge ] app-text/docbook-sgml-utils-0.6.14-r1::gentoo USE="jadetex" [ebuild NS ] app-text/docbook-sgml-dtd-4.0-r3:4.0::gentoo [4.4:4.4::gentoo] 0 KiB [ebuild NS ] app-text/docbook-sgml-dtd-4.1-r3:4.1::gentoo [4.4:4.4::gentoo] 0 KiB [ebuild NS ] app-text/docbook-sgml-dtd-3.0-r3:3.0::gentoo [4.4:4.4::gentoo] 0 KiB [ebuild NS ] app-text/docbook-sgml-dtd-3.1-r3:3.1::gentoo [4.4:4.4::gentoo] 0 KiB When I emerged them app-text/linuxdoc-tools with the USE=doc again stop compiling.
(In reply to Yarda from comment #31) > (In reply to Yarda from comment #30) > > (In reply to Yarda from comment #29) On my system merging docbook-sgml-dtd:4.4 or any other version alone results in a broken linuxdoc-tools build. But having 4.3 and 4.4 merged together works fine. deference@null.net, can you confirm that with all docbook-sgml-dtd versions removed linuxdoc-tools builds fine?
(In reply to Coacher from comment #32) > (In reply to Yarda from comment #31) > > (In reply to Yarda from comment #30) > > > (In reply to Yarda from comment #29) > On my system merging docbook-sgml-dtd:4.4 or any other version alone results > in a broken linuxdoc-tools build. But having 4.3 and 4.4 merged together > works fine. > > deference@null.net, can you confirm that with all docbook-sgml-dtd versions > removed linuxdoc-tools builds fine? Or ulogd[doc], it shouldn't matter.
*** Bug 625348 has been marked as a duplicate of this bug. ***
I unmerged all versions of docbook-sgml-dtd and ulogd[doc] does not build. The same applies to linuxdoc-tools[doc]. I have had to force the removal of several of the docbook-sgml-dtd, that it to say, they would not depclean and be gone, I used --unmerge. I tested ulogd[doc] after depclean while having the following versions of docbook-sgml-dtd installed without success: 3.0-r3, 4.0-r3, 4.1-r3, 4.4 and 3.1-r3. I then emerged only 4.3-r2 and then tried both packages again without success. I don't know where to go from here.
Let's see now I'll try merging 4.4 and see if that works... No, both packages still will not merge.
I am most sorry, my desktop system failed and my laptop also did about 1 mouth ago. I can no longer contribute to Gentoo Linux until I get a new system and my life back in order. I am currently working off an old i386 system and borrowed laptop so I don't have the compute power to spare for running Gentoo. I'll be back, but in the meantime please take what action you will. Thanks for all the penguins and God bless!
Cannot reproduce with ulogd-2.0.7 and current amd64 system. If someone can, please comment, otherwise I'm going to close this bug once ulogd-2.0.7 goes stable.
I have the same problem on my system. Thanks to the command in comment 3, I was able to track the problem down to what I think is a bug in the nsgmls program. I wrote a little demo program to demonstrate the problem: $ ./genBugTxt.py demo.txt Detected block size: 4096 Wrote demo.txt (size: 4114 bytes). $ nsgmls -D dtd <demo.txt >/dev/null nsgmls:<OSFD>0:170:19:E: end tag for "ARTICLE" omitted, but its declaration does not permit this nsgmls:<OSFD>0:3:0: start tag was here $ nsgmls -D dtd demo.txt >/dev/null The last command succeeds without error. I will try attaching the script and DTD I used. Here is the version of nsgmls I used: $ equery b /usr/bin/nsgmls * Searching for /usr/bin/nsgmls ... app-text/openjade-1.3.2-r7 (/usr/bin/nsgmls -> onsgmls) app-text/opensp-1.5.2-r3 (/usr/bin/onsgmls) I do not think the exact compilation environment matters, as I was able to reproduce the problem with a manual compile in a debug environment as well. I dove into the source code for nsgmls, and quickly reached a point where I think one of the developers for the package would make better progress. The issue appears to be that nsgmls creates multiple PosixStorageObject instances on the main input (demo.txt in the first example, and stdin in the second). Each time, it seems to check a character, which causes it to read a block in to a buffer, presumably after attempting to reset the file position to 0. This appears to work when reading demo.txt, but seeking does not work on stdin, so it does not try. Each PosixStorageObject gets a different block from stdin in the demo case, until eventually reading stdin returns EOF. I think it is the first PosixStorageObject that eventually gets passed to the parsing code, because the parsing code only receives the first block. When that PosixStorageObject attempts to read more from stdin, it gets an EOF, and the parsing terminates there. nsgmls appears to get a block size from a system stat() call on the file (or stdin). On my system, it appears to return 4096 (I do not know what limits that -- is it set by the filesystem?). The output passed to nsgmls via stdin in the command in comment 3 is over 25000 bytes, but the parsing code in nsgmls only receives the first 4096 bytes (the first block), because the rest was read by other PosixStorageObject instances during initialization. This appears to be a bug, but I do not know enough about why those PosixStorageObject instances are created to know what to do about it. On my system, the equivalent of the command in comment 3 fails with the missing end tag warnings, but if I save the output of sgmlpre to a file and call nsgmls on the file instead of stdin, it succeeds, just like my example. In my example, the genBugTxt.py script creates a simple demo.txt file, uses a stat() call to determine the block size, and padds demo.txt until it is larger than that. Passing demo.txt to nsgmls via stdin fails, as it does not get the full file, and passing demo.txt as a filename on the command line succeeds. Please let me know if it helps in reproducing the problem (it only appears to work with python2 [2.7] and not python3).
Created attachment 537848 [details] Python 2 script for generating a test case for possible nsgmls bug Run: ./genBugTxt.py demo.txt This will hopefully create a demo.txt file of the appropriate size to demonstrate the bug.
Created attachment 537850 [details] DTD file for output of genBugTxt.py This is file in the dtd directory passed via the -D option to nsgmls when parsing the output of genBugTxt.py: nsgmls -D dtd <demo.txt >/dev/null vs. nsgmls -D dtd demo.txt >/dev/null
(In reply to Anon Emuss from comment #41) > Created attachment 537850 [details] > DTD file for output of genBugTxt.py > > This is file in the dtd directory passed via the -D option to nsgmls when > parsing the output of genBugTxt.py: > nsgmls -D dtd <demo.txt >/dev/null > vs. > nsgmls -D dtd demo.txt >/dev/null This was taken from linuxdoc-tools-0.9.72 (/usr/share/linuxdoc-tools/dtd/linuxdoc.dtd -> /usr/share/linuxdoc-tools/dtd/linuxdoc96.dtd) in the hopes that this demo would work without having linuxdoc-tools installed, but that does not appear to be the case. The general trick of inflating any input file that works to be larger than the block size will hopefully still trigger the problem.
Anon Emuss, thank you for the extensive report. I too suspected nsgmls, but didn't have any sound evidence. It's interesting that with the same versions of installed software some people trigger this and some people don't. Since you have a reliable way to reproduce this, could you please test one of my guesses: recompile app-text/opensp-1.5.2-r3 with all optimizations disabled (-O0 in CFLAGS/CXXFLAGS) and see if you can still trigger this bug.
(In reply to Coacher from comment #43) > Anon Emuss, thank you for the extensive report. You are welcome. I hope it turns out to be helpful. > I too suspected nsgmls, but didn't have any sound evidence. > It's interesting that with the same versions of installed software some > people trigger this and some people don't. Other than weird ideas like strange compiler malfunctions (I have gcc-7.3.0-r3 and glibc-2.26-r7, if that is useful), I was guessing maybe it was the block size. It is 4096 for me. Comment 15 makes it sound like it was 8192 for deference. The data used when installing ulogd[doc], for me, is under 26000 bytes, so anybody with a block size of at least 32k should not have a problem. It sounds like the problem does not occur for you. I am curious what your block size is. genBugTxt.py reports it, or a Perl one-liner can do the trick, too: $ perl -e 'my @s=stat("/var/tmp/portage");print $s[11].$/;' 4096 > Since you have a reliable way to reproduce this, could you please test one > of my guesses: recompile app-text/opensp-1.5.2-r3 with all optimizations > disabled (-O0 in CFLAGS/CXXFLAGS) and see if you can still trigger this > bug. I tried two ways. One way was to use emerge: $ CFLAGS=-O0 CXXFLAGS=-O0 emerge -a1 opensp This installed app-text/opensp-1.5.2-r3. With that, I ran (using demo.txt generated from genBugTxt.py): $ nsgmls -ifmttxt <demo.txt >/dev/null nsgmls:<OSFD>0:170:19:E: end tag for "ARTICLE" omitted, but its declaration does not permit this nsgmls:<OSFD>0:3:0: start tag was here $ nsgmls -ifmttxt demo.txt >/dev/null The last command ran without error. This appears to reproduce the problem. In case the CFLAGS/CXXFLAGS did not work as I expected, or emerge overrode something, I also tried a more manual approach. I unpacked the source with: $ ebuild /usr/portage/app-text/opensp/opensp-1.5.2-r3.ebuild unpack Then, I copied the source to another location, where I ran (I did not configure system catalog locations, so I supplied them on the command line): $ CFLAGS=-O0 CXXFLAGS=-O0 ./configure <cut> $ make <cut> $ nsgmls/onsgmls -c /etc/sgml/catalog -ifmttxt </tmp/demo.txt >/dev/null nsgmls/.libs/lt-onsgmls:<OSFD>0:170:19:E: end tag for "ARTICLE" omitted, but its declaration does not permit this nsgmls/.libs/lt-onsgmls:<OSFD>0:3:0: start tag was here $ nsgmls/onsgmls -c /etc/sgml/catalog -ifmttxt /tmp/demo.txt >/dev/null The last command ran without error. This also appears to reproduce the problem. The environment at both configure and compilation time did not have any compiler variables set (save for the ones on the command line), so I think that should have the most minimal flags.
Yes. Problem for some reason doesn't occur here. ``` coacher@Photon ~/Downloads $ python2 genBugTxt.py demo.txt Detected block size: 4096 Wrote demo.txt (size: 4114 bytes). coacher@Photon ~/Downloads $ nsgmls -ifmttxt <demo.txt >/dev/null coacher@Photon ~/Downloads $ nsgmls -ifmttxt demo.txt >/dev/null ``` All commands work fine here. Reasonable compiler flags: CFLAGS="-O2 -march=native -mfpmath=sse -fomit-frame-pointer -pipe -frecord-gcc-switches -Wimplicit-function-declaration" CXXFLAGS="-O2 -march=native -mfpmath=sse -fomit-frame-pointer -pipe -frecord-gcc-switches" app-text/opensp-1.5.2-r3 USE="nls -doc -static-libs {-test}" app-text/openjade-1.3.2-r7 USE="-static-libs"
Also I remember that installed dtds affected this some time in the past. I have: ``` coacher@Photon ~ $ cat /etc/sgml/catalog CATALOG "/etc/sgml/sgml-docbook.cat" CATALOG "/etc/sgml/sgml-ent.cat" CATALOG "/etc/sgml/xml-docbook-4.2.cat" CATALOG "/etc/sgml/xml-docbook-4.1.2.cat" CATALOG "/etc/sgml/xml-docbook-4.5.cat" CATALOG "/etc/sgml/xml-docbook-4.3.cat" CATALOG "/etc/sgml/xml-docbook-4.4.cat" CATALOG "/etc/sgml/openjade-1.3.2.cat" CATALOG "/etc/sgml/linuxdoc.cat" ``` What do you have, Anon Emuss?
(In reply to Coacher from comment #46) > Also I remember that installed dtds affected this some time in the past. >... Coacher: That is a good point. I should have played with that earlier. My /etc/sgml/catalog has more entries, so I tried cutting it down. I found a 2-line version that allows both of the commands I have to work. $ cat /etc/sgml/catalog CATALOG "/etc/sgml/sgml-ent.cat" CATALOG "/etc/sgml/linuxdoc.cat" $ nsgmls -ifmttxt <demo.txt >/dev/null $ nsgmls -ifmttxt demo.txt >/dev/null With that 2-line /etc/sgml/catalog, both of the two methods of parsing demo.txt appear to work. Next, I tried adding back various lines of the original /etc/sgml/catalog, and when I found a line that triggered the problem, I descended into it. This was the file that appeared to trigger the problem: /etc/sgml/dsssl-docbook-stylesheets.cat I notice it does NOT appear in your /etc/sgml/catalog. On my system, it is also referenced in a few of the files that DO appear in your /etc/sgml/catalog (e.g. /etc/sgml/sgml-docbook.cat), so it is unlikely that just trying to get our topmost catalog files to match will work. I am hoping that will not be necessary, because I think I found the important lines. On my system, /etc/sgml/dsssl-docbook-stylesheets.cat contains a single line which essentially includes this file: /usr/share/sgml/docbook/dsssl-stylesheets-1.79/catalog which is owned by app-text/docbook-dsssl-stylesheets-1.79-r2. Playing around with the lines of that file, I found a single line (split into two lines, technically) that seemed to cause the problem: DTDDECL "-//Norman Walsh//DTD DocBook HTML 1.0//EN" dtds/html/dbhtml.dcl As near as I can tell, dbhtml.dcl does not exist on my system (locate does not find it, and find on that directory tree does not find anything with that name). It appears that any DTDDCL line causes the problem, whether or not the file exists. It does not appear to be required that it be included directly through /etc/sgml/catalog, so I am hoping that this will reproduce the problem on your system: $ cat /tmp/dtddcl.cat DTDDECL "FooBar" /foo/bar $ nsgmls -ifmttxt <demo.txt >/dev/null $ nsgmls -c /tmp/dtddcl.cat -ifmttxt <demo.txt >/dev/null nsgmls:<OSFD>0:170:19:E: end tag for "ARTICLE" omitted, but its declaration does not permit this nsgmls:<OSFD>0:3:0: start tag was here I can either include that dummy DTDDECL line directly in /etc/sgml/catalog, or via another catalog using the -c option as above, and it triggers the problem. There is a little more to it than that, as I was able to put together a different txt file that is over 4k that DOES work. It uses the declarations in /usr/share/sgml/sgml-dtd-4.1/ instead of something from /usr/share/linuxdoc-tools/, so there is apparently something about the linuxdoc-tools DTDs that do not work with a DTDDECL line. Would you be willing to try running the same commands, with the same demo.txt, with the dummy DTDDECL line either added to your /etc/sgml/catalog or added via a -c switch on the command line? I am curious whether that will trigger the problem on your system. I am hoping it will trigger the problem without needing to modify your /etc/sgml/catalog. Thanks for your help with this.
Success! Now I can reproduce this: ``` coacher@Photon ~/Downloads $ cat dtddcl.cat DTDDECL "FooBar" /foo/bar coacher@Photon ~/Downloads $ python2 genBugTxt.py demo.txt Detected block size: 4096 Wrote demo.txt (size: 4114 bytes). coacher@Photon ~/Downloads $ nsgmls -ifmttxt demo.txt >/dev/null coacher@Photon ~/Downloads $ nsgmls -c dtddcl.cat -ifmttxt demo.txt >/dev/null coacher@Photon ~/Downloads $ nsgmls -ifmttxt <demo.txt >/dev/null coacher@Photon ~/Downloads $ nsgmls -c dtddcl.cat -ifmttxt <demo.txt >/dev/null nsgmls:<OSFD>0:170:19:E: end tag for "ARTICLE" omitted, but its declaration does not permit this nsgmls:<OSFD>0:3:0: start tag was here ``` Re dtds/html/dbhtml.dcl, please check if the file /usr/share/sgml/docbook/dsssl-stylesheets-1.79/html/dbhtml.dsl exists. Here docbook-dsssl-stylesheets-1.79-r2.ebuild installs it. Seems paths are a bit broken in usr/share/sgml/docbook/dsssl-stylesheets-1.79/catalog. The files are there, but in different locations. Should be easy to fix. Could you please manually do the following as root: cd /usr/share/sgml/docbook/dsssl-stylesheets-1.79/dtds/ ln -s ../html html And then re-run your tests with all your regular system-wide dtds. See if this solves the issue. Afterwards you can remove the created link via (notice no trailing slash): rm /usr/share/sgml/docbook/dsssl-stylesheets-1.79/dtds/html
(In reply to Coacher from comment #48) > Success! Now I can reproduce this: That sounds like progress. I am frequently amused by the amount of work that goes into causing a problem before even being able to fix it or work around it. >... > Re dtds/html/dbhtml.dcl, please check if the file > /usr/share/sgml/docbook/dsssl-stylesheets-1.79/html/dbhtml.dsl exists. Here > docbook-dsssl-stylesheets-1.79-r2.ebuild installs it. It exists, and appears to be installed by the same ebuild. > Seems paths are a bit broken in > usr/share/sgml/docbook/dsssl-stylesheets-1.79/catalog. The files are there, > but in different locations. Should be easy to fix. I am not so sure the files are there. The extensions are different, and the contents look different. See comments below. > Could you please manually do the following as root: > cd /usr/share/sgml/docbook/dsssl-stylesheets-1.79/dtds/ > ln -s ../html html >... All relative paths below are relative to /usr/share/sgml/docbook/dsssl-stylesheets-1.79/ (except demo.txt, which is the usual file). Adding the symlink dtds/html -> ../html did not affect the problem; demo.txt still failed to parse when passed through stdin to nsgmls. catalog points to dtds/html/dbhtml.dcl. Adding the symlink would allow it to find html/dbhtml.dcl, but that does not exist -- the file in that directory is html/dbhtml.dsl (slightly different extension). I tried modifying catalog to point at html/dbhtml.dsl, and that failed to fix the problem. I also tried modifying it to point at an existing .dcl file (/usr/share/sgml/docbook/sgml-dtd-3.0/docbook.dcl), and that also failed to fix the problem. Looking at the contents of /usr/share/sgml/xml.dcl and /usr/share/sgml/docbook/sgml-dtd-3.0/docbook.dcl, the contents look pretty different from html/dbhtml.dsl; I suspect the latter is a different type of file. As far as missing files go, /usr/share/sgml/docbook/dsssl-stylesheets-1.79/ has several. Here are some others (along with some guesses at which files in /usr/share/sgml/ might be similar): -------- dtds/dbdsssl/dbdsssl.dtd There are plenty of .dtd files in /usr/share/sgml/, but no matches for dbdsssl.*. dtds/html/ISOlat1.gml There are no .gml files in /usr/share/sgml/. There are several versions of ISOlat1.ent and ISOlat1.sgm. docsrc/{docbook,olinktarget,olinksemantics}.sgm The only docsrc/ directory is /usr/share/sgml/xml-stylesheets/docsrc/. It does not include any .sgm files, or anything with docbook or olink in its name. The only .sgm files are: /usr/share/sgml/openjade-1.3.2/pubtext/ISOlat1.sgm /usr/share/sgml/opensp-1.5.2/OpenSP/ISOlat1.sgm /usr/share/sgml/opensp-1.5.2/OpenSP/demo.sgm which do not seem like a match. -------- So far, it appears that nsgmls only parses the first block of stdin if using the linuxdoc-tools docbook style, and the catalogs include a DTDDCL line, whether or not the file referenced in that line exists. I suppose a next step would be to try taking apart the linuxdoc-tools files and see if any particular line seems to trigger the problem. That, or I could dive into the nsgmls source code again and see if I could find some lines that seem to trigger the issue, but perhaps the OpenSP developers would be more efficient at that. As I mentioned before, I got a file to work that uses a different document type, which suggests there is something specific about the linuxdoc-tools files that does not work well with the DTDDECL. I will probably play with the linuxdoc-tools files, but I am open to suggestions.
The files are there in the tarball (only docsrc is missing from the tarball). They just don't get installed because of a poor Gentoo Makefile. I am working on fixing it. In the meantime I'd appreciate if you could try to track down the cause of this problem in linuxdoc-tools. Thanks!
Anon Emuss, please give a try to the ebuild in the linked PR, see 'See Also'.
(In reply to Coacher from comment #50) > The files are there in the tarball (only docsrc is missing from the > tarball). They just don't get installed because of a poor Gentoo Makefile. I > am working on fixing it. > > In the meantime I'd appreciate if you could try to track down the cause of > this problem in linuxdoc-tools. Thanks! Sorry. I was away for a bit. I'll look into it, including your updated ebuild.
(In reply to Coacher from comment #51) > Anon Emuss, please give a try to the ebuild in the linked PR, see > 'See Also'. OK. On my system, with docbook-dsssl-stylesheets-1.79-r2 installed (before using your ebuild), the file referenced in the DTDDECL line of /usr/share/sgml/docbook/dsssl-stylesheets-1.79/catalog (dtds/html/dbhtml.dcl) does not exist (as mentioned earlier). Using a clean compile of onsgmls from opensp-1.5.2, here are the commands I have been using as test cases: -------- $ nsgmls/onsgmls -c /etc/sgml/catalog -ifmttxt /tmp/demo.txt >/dev/null $ nsgmls/onsgmls -c /etc/sgml/catalog -ifmttxt </tmp/demo.txt >/dev/null nsgmls/.libs/lt-onsgmls:<OSFD>0:170:19:E: end tag for "ARTICLE" omitted, but its declaration does not permit this nsgmls/.libs/lt-onsgmls:<OSFD>0:3:0: start tag was here -------- After I update from docbook-dsssl-stylesheets-1.79-r2 to docbook-dsssl-stylesheets-1.79-r3 using the pull requests mentioned above, the file referended in the DTDDECL line of /usr/share/sgml/docbook/dsssl-stylesheets-1.79/catalog is still dtds/html/dbhtml.dcl, but now it DOES exist (thanks!). However, this does not affect the error. It still happens: -------- $ nsgmls/onsgmls -c /etc/sgml/catalog -ifmttxt /tmp/demo.txt >/dev/null $ nsgmls/onsgmls -c /etc/sgml/catalog -ifmttxt </tmp/demo.txt >/dev/null nsgmls/.libs/lt-onsgmls:<OSFD>0:170:19:E: end tag for "ARTICLE" omitted, but its declaration does not permit this nsgmls/.libs/lt-onsgmls:<OSFD>0:3:0: start tag was here -------- I suspect any DTDDECL line, whether or not the file exists, will cause a problem like this one. I will try to summarize my findings on that in a separate comment.
I dug deeper into the opensp-1.5.2 source code to try to figure out what about linuxdoc-tools was causing the problem, and found that I was heading down the wrong track. This appears to be an opensp-1.5.2 bug, not a linuxdoc-tools bug. Most of what I describe below I found directly, but there are a few inferences which seem to fit quite well with the evidence so far. I found a brief description of what DTDDECL does in /usr/share/doc/opensp-1.5.2-r3/doc/catalog.htm: -------- DTDDECL <pubid> <sysid> This specifies that if the document does not contain an SGML declaration and uses a doctype declaration with public identifier <pubid>, the SGML declaration in <sysid> should be implied. -------- I used a debug build of opensp-1.5.2 to track down the differences between reading from standard input and a file given on the command-line, and it appears that onsgmls handles DTDDECL lines by repeating the initialization for each DTDDECL line it finds, presumably with the appropriate SGML declaration inserted. It does not appear to need to read each input file in its entirety each time it initializes, but it does attempt to read the beginning of each input file (including standard input, if using that), presumably to figure out declarations, character sets, line endings, and/or other similar things. Only after all of this does it return to the original open handle and use that to parse the entire file. When reading from a file on the command line, it can rewind and start reading from the beginning each time, and so the entire file gets parsed. That is not possible with a pipe such as the standard input, however, and so each new handle opened by a repeated initialization reads a block from standard input that is then lost to the original. The program uses a storage object class to wrap each handle, and that keeps a buffer, so the initial block read by the original handle is still available when it comes time for the final parse, but any blocks read by other handles wrapped in storage objects are lost. In comment 47, I remarked that I had a different input file that used /usr/share/sgml/sgml-dtd-4.1/ and nothing from linuxdoc-tools, and that it worked. I took this to mean that the problem was somehow associated with linuxdoc-tools. I looked into that again and found I was wrong. That other input file was being truncated just like the demo.txt being used here. The docbook declaration used in /usr/share/sgml/sgml-dtd-4.1/ does not require the ending tags to be specified, and so when they were truncated off, onsgmls implicitly assumed them and did not report an error. Checking the output more carefully, I see from its output that it was truncating the input file. This happens any time there is a DTDDECL line, if the file is read from standard input. If it would be useful, I can post all of the files required for that case as an example of the bug that does not use any linuxdoc-tools files. I checked the documentation for the opensp-1.5.2 package, and the closest I could find of any mention of this being a problem was in a list of possible ideas for improving the program, in /usr/share/doc/opensp-1.5.2-r3/doc/ideas.htm: -------- Entity Manager ... Try harder to rewind in StdioStorageObject. -------- I tried looking for more recent versions of the opensp package to test, and it does not look like there is much development happening. A lot of the links on the homepage do not seem to work any more. I saw nothing newer than opensp-1.5.2. Interestingly, I did find an old bug (for opensp-1.5) that appears to point out the same problem: https://sourceforge.net/p/openjade/bugs/64/ I tried using the files posted there, but they referenced a lot of other files which were not included and I do not have on my system, so it was unclear whether or not I could reproduce the problem with those files. The last comment in the bug report asks the submitter to try it again with opensp-1.5.2, but there is no response. That was in early 2006, but maybe pinging that with one or two sets of files that demonstrate the problem with opensp-1.5.2 would generate a response. As a result of this, I am pretty sure this is a bug in opensp-1.5.2 which apparently was introduced somewhere between sp-1.3.4 and opensp-1.5, according to the bug report, and not a problem with linuxdoc-tools. I have two ideas as to where to go from here, and am hoping for suggestions as to which ones are worth pursuing (if any). I am also open to any other ideas on how to proceed. My first idea is to fix opensp. My first choice would be to get the attention of one of the developers, either through direct contact, opening a new bug report, or posting to the existing one. If that fails or seems likely to fail, and nobody else reading this feels particularly qualified, I could have a go at it myself. I do not really understand what the program does, but I have gotten an understanding of the data structures used to deal with reading inputs, and there is a decent chance I can get it to reuse the same storage object each time it rereads the file instead of trying to rewind each time. I am not confident that I could test it particularly well seeing as how I do not understand the main program or what it is supposed to do. However, I think I could do this in a way that it only touches some of the low-level storage object wrappers. That would make my changes pretty much transparent to the entire program, and, if I could do that properly, it would be unlikely to introduce bugs. Any such patch could be included with an updated ebuild for opensp-1.5.2, and/or passed to the developers to be included in a later release. My second idea is to provide a workaround in linuxdoc-tools. The bug only manifests when reading from a pipe such as standard input, which is how some of the Perl scripts in linuxdoc-tools are running nsgmls (called onsgmls in the opensp package). Those scripts already use some temporary files. I see no reason why there could not be one more intermediate temporary file to hold the input of nsgmls, and have nsgmls read from that intermediate file directly instead of being piped in through standard input. That would avoid the bug, and should fix any package affected by it (including ulogd[doc]). As above, if nobody else more qualified chimes in, I am willing to have a go at it. I am much more confident in my ability to handle this than patching opensp, and should be able to get a workaround patch for linuxdoc-tools working pretty shortly (but would be sad that if the bug in opensp were never fixed). Again, that patch could be included with an updated ebuild for linuxdoc-tools-0.9.72, and/or passed to the developers to be included in a later release. Thanks for the help so far, Coacher. Any more advice for me?
Anon Emuss, thanks again for the detailed info. It looks like the opensp bug you've linked describes the problem we have here. AFAIU, opensp upstream is dead and in Gentoo it's on life support. CC'ing opensp maintainers to see if they're willing to tackle opensp. If not, I'd say we workaround it in linuxdoc-tools instead. @heroxbd, please take a look at comment 47 and below.
@heroxbd, steps in comment 48 reportedly don't trigger this bug on Debian. Debian has a huge patch for opensp. I was unable to find a possible fix in it, but you are probably more familiar with opensp codebase so maybe you will be able to spot it.
Created attachment 539722 [details, diff] 1000_LinuxDocTools.pm_nsgmls-no-pipe.diff Anon Emuss, could you please test the attached patch in the meantime?
(In reply to Coacher from comment #57) > Created attachment 539722 [details, diff] [details, diff] > 1000_LinuxDocTools.pm_nsgmls-no-pipe.diff > > Anon Emuss, could you please test the attached patch in the meantime? Apologies for the delay. I have been unavailable for the past few days. I tried the patch, and it works. Without the patch, ulogd[doc] fails with the message about missing SGML tags. If I then re-install linuxdoc-tools with your patch, ulogd[doc] installs fine. Thanks for putting that together.
(In reply to Coacher from comment #56) > @heroxbd, steps in comment 48 reportedly don't trigger this bug on Debian. > Debian has a huge patch for opensp. I was unable to find a possible fix in > it, but you are probably more familiar with opensp codebase so maybe you > will be able to spot it. I tracked down the problem in the opensp source and started trying to patch it before I had to take a pause to work on other things. If Debian has fixed the issue, they probably have tested it much better than I could, but I think my work would help me figure out whether or not they have a possible fix in there. If you could point me towards this huge patch for opensp, I could take a look.
(In reply to Anon Emuss from comment #59) > I tracked down the problem in the opensp source and started trying to patch > it before I had to take a pause to work on other things. If Debian has > fixed the issue, they probably have tested it much better than I could, but > I think my work would help me figure out whether or not they have a possible > fix in there. If you could point me towards this huge patch for opensp, I > could take a look. http://http.debian.net/debian/pool/main/o/opensp/opensp_1.5.2-13.diff.gz
(In reply to Coacher from comment #60) > (In reply to Anon Emuss from comment #59) > > I tracked down the problem in the opensp source and started trying to patch > > it before I had to take a pause to work on other things. If Debian has > > fixed the issue, they probably have tested it much better than I could, but > > I think my work would help me figure out whether or not they have a possible > > fix in there. If you could point me towards this huge patch for opensp, I > > could take a look. > > http://http.debian.net/debian/pool/main/o/opensp/opensp_1.5.2-13.diff.gz Also Debian apparently builds opensp with `--disable-dtddecl'. This might be the key.
(In reply to Coacher from comment #61) >... > Also Debian apparently builds opensp with `--disable-dtddecl'. > This might be the key. You beat me to it. I agree. I do not see a fix in the patch. I checked the whole patch, and there are very few code modifications, and they seem to mostly deal with patching a small memory leak. I even was able to reproduce the problem with the patch installed. Instead of a patch, they have the --disable-dtddecl workaround. In debian/rules (created by the patch), configure is called with the --disable-dtddecl option, which disables the DTDDECL handling, circumventing the bug. In debian/changelog (also created by the patch), this is described as a fix for Debian bugs #138924 and #208042, described here: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=138924 https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=208042 It seems that in addition to causing problems reading from standard inputs, the DTDDECL handling can also cause quite a slowdown, which is why Debian developed it. This supports my theory from comment 54 that onsgml performs its entire initialization once for every DTDDECL it finds on the first initialization. I can confirm that configuring with --disable-dtdecl works around the problem even without the Debian patch. I could continue trying to patch opensl to fix the standard input issue, but that would not fix the slowdown issue. Trying to handle that would be much more of an investment than I think I would be up for. I am not sure what the DTDDECL really does, except that it looks like some sort of default description if none is provided in the input document. If Debian thought it was worth disabling, maybe that would work for Gentoo as well?
linuxdoc-tools upstream pushed a workaround for this into master. I'll create a new revision with this patch in a couple of days. As for opensp, I think the situation deserves a separate bug, which I am planning to open sometime later if I don't forget.
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=f3a064e1a7d2f6f526ebdff215a981387a7d652d commit f3a064e1a7d2f6f526ebdff215a981387a7d652d Author: Ilya Tumaykin <itumaykin@gmail.com> AuthorDate: 2018-07-04 19:36:20 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2018-07-26 08:35:09 +0000 app-text/docbook-dsssl-stylesheets: install missing files, cleanup Due to Gentoo-originated poorly written Makefile not all required files were installed, which caused tools looking for them to fail, e.g. Gentoo bug 606768. Also one file that belong to another package was installed: dtds/decls/docbook.dcl is a part of docbook-sgml-dtd. Replace custom Makefile with a series of doins commands. Also remove dirty, ugly symlink hack to workaround Gentoo bug 15026. Current stable libgcrypt doesn't use docbook-dssl-stylesheets for docs. Bug: https://bugs.gentoo.org/606768 Bug: https://bugs.gentoo.org/15026 Package-Manager: Portage-2.3.41, Repoman-2.3.9 .../docbook-dsssl-stylesheets-1.79-r3.ebuild | 65 ++++++++++++++++++++++ 1 file changed, 65 insertions(+)
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=6725a37b5c4eceb45fa398ab087b98f1107faec5 commit 6725a37b5c4eceb45fa398ab087b98f1107faec5 Author: Ilya Tumaykin <itumaykin@gmail.com> AuthorDate: 2018-07-25 19:26:11 +0000 Commit: Michał Górny <mgorny@gentoo.org> CommitDate: 2018-10-05 21:28:19 +0000 app-text/linuxdoc-tools: verbump to 0.9.73 Among other things uses nsgmls without a pipe, which fixes bug 606768. Bug: https://bugs.gentoo.org/606768 Signed-off-by: Ilya Tumaykin <itumaykin@gmail.com> Package-Manager: Portage-2.3.50, Repoman-2.3.11 Signed-off-by: Michał Górny <mgorny@gentoo.org> Closes: https://github.com/gentoo/gentoo/pull/9350 app-text/linuxdoc-tools/Manifest | 1 + .../linuxdoc-tools/linuxdoc-tools-0.9.73.ebuild | 74 ++++++++++++++++++++++ 2 files changed, 75 insertions(+)
*** Bug 625360 has been marked as a duplicate of this bug. ***
