When booted as vmware guest with CONFIG_VMI=Y the VMI ROM is not initialized if kernels is later then gentoo-sources-2.6.26-gentoo-r3 or vanilla-sources-2.6.26.8 Reproducible: Always Steps to Reproduce: 1. Boot gentoo-sources-2.6.26-gentoo-r3 or vanilla-sources-2.6.26.8 2. # dmesg | grep -i VMI 3. Boot with later kernel i.e. vanilla-sources-2.6.27 4. # dmesg | grep -i VMI Actual Results: vanilla-sources-2.6.27 dmesg | grep -i VMI vmi: registering clock source khz=3000000 Expected Results: gentoo-sources-2.6.26-gentoo-r3 dmesg | grep -i VMI VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0 vmi: registering clock event vmi-timer. mult=12582912 shift=22 Booting paravirtualized kernel on vmi vmi: registering clock source khz=3000000 I would like to try bisection to locate the ofending patch but don't know how to approach that across kernel releases.
Since this problem is in the mainstream kernel, go ahead and use the git-bisect technique with the mainstream kernel repo linux-2.6.git http://www.kernel.org/doc/local/git-quick.html Let us know if you have any questions, or if not, what you discover.
(In reply to comment #0) > When booted as vmware guest with CONFIG_VMI=Y the VMI ROM is not initialized if > kernels is later then gentoo-sources-2.6.26-gentoo-r3 > or vanilla-sources-2.6.26.8 > Argh! Is this linked to http://bugzilla.kernel.org/show_bug.cgi?id=12167 or rather http://bugs.gentoo.org/show_bug.cgi?id=249751 I see, it's a different problem, but it's quite likely that the same area of "fragile code" (quoting someone here) might be affected.
Please can you confirm that vanilla-sources-2.6.27 is the *exact* earliest kernel that you reproduced this bug on? It wasn't 2.6.27.1 or 2.6.27.4 or something?
btw here is another guide on bisecting: http://www.reactivated.net/weblog/archives/2006/01/using-git-bisect-to-find-buggy-kernel-patches/ Use v2.6.26 as good and v2.6.27 as bad, after confirming that 2.6.27 is the exact kernel that you first saw the problem in. (alternatively, if 2.6.27 is OK and 2.6.27.x is not, then we have to bisect a different tree instead)
(In reply to comment #3) > Please can you confirm that vanilla-sources-2.6.27 is the *exact* earliest > kernel that you reproduced this bug on? It wasn't 2.6.27.1 or 2.6.27.4 or > something? My understanding of release order is ... 2.6.26.7 2.6.26.8 2.6.27 2.6.27-1 2.6.27-2 ... . Is this correct?
"order" can be interpreted in different ways and isn't really relevant.. please just clarify the exact kernel versions you have tested (and their results w.r.t. this bug)
Norman, as you have been bisecting the 2.6.27.x tree to find the bad commit that caused your INT 14 crashes http://bugzilla.kernel.org/show_bug.cgi?id=12167 http://bugs.gentoo.org/show_bug.cgi?id=249751 ... do you remember, if any of the not-crashing kernels did initialize the VMI ROM? Or did you just check whether it crashes or not.
(In reply to comment #6) > "order" can be interpreted in different ways and isn't really relevant.. please > just clarify the exact kernel versions you have tested (and their results > w.r.t. this bug) I'm sorry, I'll clarify: vanilla-sources-2.6.26.8 good dmesg | grep -i VMI VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0 vmi: registering clock event vmi-timer. mult=12582912 shift=22 Booting paravirtualized kernel on vmi vmi: registering clock source khz=3000000 vanilla-sources-2.6.27 bad dmesg | grep -i VMI vmi: registering clock source khz=3000000 Now, because this bridges release 26 and 27 I am unable determine which git source tree to clone and which tag to use as 'bisect bad' and bisect good'.
(In reply to comment #8) > (In reply to comment #6) > > "order" can be interpreted in different ways and isn't really relevant.. please > > just clarify the exact kernel versions you have tested (and their results > > w.r.t. this bug) > > I'm sorry, I'll clarify: > > vanilla-sources-2.6.26.8 good > dmesg | grep -i VMI > VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0 > vmi: registering clock event vmi-timer. mult=12582912 shift=22 > Booting paravirtualized kernel on vmi > vmi: registering clock source khz=3000000 > > vanilla-sources-2.6.27 bad > dmesg | grep -i VMI > vmi: registering clock source khz=3000000 > > Now, because this bridges release 26 and 27 I am unable determine which git > source tree to clone and which tag to use as 'bisect bad' and bisect good'. > In this case you'll have to use Linus' 2.6 tree. http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=summary start with 2.6.26 good and 2.6.27 bad
(In reply to comment #9) > In this case you'll have to use Linus' 2.6 tree. > > http://git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.gitgit.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git;a=summary > > start with 2.6.26 good and 2.6.27 bad Thanks. I've made a start on this: git clone git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.gitgit.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git git checkout v2.6.26 make clean make menuconfig # Here to save compile time I selected only those modules required to boot # into vmware and set them to Y rather than M make # Tested complied kernel in vmware and the VMI ROM was present git checkout v2.6.27 make clean make oldconfig # accepting the defualts make # Tested complied kernel in vmware and the VMI ROM was present. Uh! Provided that the above is logical (please comment) it appears that this issue is dependant on a particular module. If so, this might take a bit longer than expected. ( Yes I have retested with the original 2.6.27 kernel and it still fails.)
(In reply to comment #10) > Thanks. I've made a start on this: > git clone > git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.gitgit.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git > git checkout v2.6.26 > make clean > make menuconfig > # Here to save compile time I selected only those modules required to boot > # into vmware and set them to Y rather than M > make > # Tested complied kernel in vmware and the VMI ROM was present > > git checkout v2.6.27 > make clean > make oldconfig # accepting the defualts > make > # Tested complied kernel in vmware and the VMI ROM was present. Uh! > > Provided that the above is logical (please comment) it appears that this > issue is dependant on a particular module. If so, this might take a bit longer > than expected. ( Yes I have retested with the original 2.6.27 kernel and it > still fails.) > Do you have my any chance a copy of your originally NOT running kernel config on 2.6.27? Might be interesting to compare. Or is it possible that VMI works on 2.6.27 but not on 2.6.27-r4, i. e. 2.6.27.8
(In reply to comment #11) > Do you have my any chance a copy of your originally NOT running kernel config > on 2.6.27? Might be interesting to compare. Yes. I always copy and rename the .config file when I install the kernel into /boot. My next task is to examine the (many) differences. > Or is it possible that VMI works on 2.6.27 but not on 2.6.27-r4, i. e. 2.6.27.8 No. It is definately 2.6.27 that failed.
I have now compiled tag v2.6.27 in the git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.gitgit.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git tree, booted it and VMI ROM fails to initialise. So does anybody have any guidence or tool for bisecting the .config kernel modules file?
(In reply to comment #13) > I have now compiled tag v2.6.27 in the > git.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.gitgit.kernel.org/?p=linux/kernel/git/torvalds/linux-2.6.git > tree, booted it and VMI ROM fails to initialise. So does anybody have any > guidence or tool for bisecting the .config kernel modules file? > I don't know, if there's a ".config bisect". For a start and to let other's have a look, would you mind to attach both .config files, i. e. .config-2.6.27-bad and .config-2.6.27.good?
My suggestion would be too look for changes in PAE, high memory, and that kind of area. I don't know of any config-bisecting tools, but there is a tool in portage for kernel config comparisons: kccmp. I don't know how useful it is, so if you do try it, please tell us about your experiences!
I will shoot off a quick email to Zach @ vmware incase he has any immediate ideas that would save you some time.
Norman, please attach your working and broken .configs here ASAP, might be useful for Zach
Created attachment 174823 [details] config-2.6.26-good
Created attachment 174825 [details] config-2.6.27-bad
(In reply to comment #15) > My suggestion would be too look for changes in PAE, high memory, and that kind > of area. > > I don't know of any config-bisecting tools, but there is a tool in portage for > kernel config comparisons: kccmp. I don't know how useful it is, so if you do > try it, please tell us about your experiences! I've tried kccmp and it looks really useful. Unfortunatily, it displays the results using qt with no apparent copy & paste of text mode. This makes it difficult to copy the output here.
(In reply to comment #20) > I've tried kccmp and it looks really useful. Unfortunatily, it displays the > results using qt with no apparent copy & paste of text mode. This makes it > difficult to copy the output here. Hmm, I thought you had to configs for kernel 2.6.27, one being good and the other being bad. The configs you've attached are for differnent kernel versions ....
(In reply to comment #20) > I've tried kccmp and it looks really useful. Unfortunatily, it displays the > results using qt with no apparent copy & paste of text mode. This makes it > difficult to copy the output here. I've justed noted that 2.6.27 sources contain a small tool written in python. scripts/diffconfig
(In reply to comment #21) > (In reply to comment #20) > > I've tried kccmp and it looks really useful. Unfortunatily, it displays the > > results using qt with no apparent copy & paste of text mode. This makes it > > difficult to copy the output here. > Hmm, I thought you had to configs for kernel 2.6.27, one being good and the > other being bad. The configs you've attached are for differnent kernel versions The configs I sent are for those tags I am currently tring to bisect. This bug was original opened against vanilla 2.6.26.8 (good) and 2.6.27 (bad). I will attach these as config-2.6.26.8-1 and config-2.6.27-1
Created attachment 174869 [details] config-2.6.26.8-1 (original good)
Created attachment 174870 [details] config-2.6.27-1 (original bad)
(In reply to comment #23) Yeah, but in comment #10 you said that you suddenly had the VMI ROM detected under 2.6.27 (using a "minimal" config). So there also exists something like config-2.6.27-good which would be interesting to compare with config-2.6.27-bad.
(In reply to comment #26) > (In reply to comment #23) > Yeah, but in comment #10 you said that you suddenly had the VMI ROM detected > under 2.6.27 (using a "minimal" config). > > So there also exists something like config-2.6.27-good which would be > interesting to compare with config-2.6.27-bad. Agreed. I have extracted config-2.6.27-good from /proc/config.gz of the working 2.6.27 kernel. There are unfortunatily rather a large number of differneces.
Created attachment 174901 [details] config-2.6.27-good
I have tried again to bisect v2.6.26 to v2.6.27 but I seem to repeatedly hit other issues on the bisects, like kernel panic "Bug Int 6 CR2". I assume these are caused by mismatched patches between 2.6.26 and 2.6.27 and nothing to do with my issue. How do I work around this, any suggestions?
(In reply to comment #29) > I have tried again to bisect v2.6.26 to v2.6.27 but I seem to repeatedly hit > other issues on the bisects, like kernel panic "Bug Int 6 CR2". I assume these > are caused by mismatched patches between 2.6.26 and 2.6.27 and nothing to do > with my issue. How do I work around this, any suggestions? > You can manually move the current intersection some commits down or up by using a command like $ git reset --hard HEAD~3 # try 3 revs before what was suggested as described in "man git-bisect" under "Avoiding to test a commit"
(In reply to comment #27) > Agreed. I have extracted config-2.6.27-good from /proc/config.gz of the working > 2.6.27 kernel. There are unfortunatily rather a large number of differneces. > ooh yes! A lot of differences ... A lot more than between 2.6.26-good and 2.6.27-bad. While inspecting the .26-good / .27-bad config diff, I noticed that they mostly differ in the fields of DEBUG/TRACE options, e. g. MMIOTRACE, ... Have you already tried a 2.6.27 kernel without TRACING enabled?
(In reply to comment #29) > I have tried again to bisect v2.6.26 to v2.6.27 I think you are barking up the wrong tree. You do a 2.6.26-to-27 bisection when you are trying to track down a 2.6.27 regression. That is not the case here -- we know that 2.6.27 works in some configurations. If you continue, I think you will end up with bad results, or no results.
(In reply to comment #32) > (In reply to comment #29) > > I have tried again to bisect v2.6.26 to v2.6.27 > > I think you are barking up the wrong tree. You do a 2.6.26-to-27 bisection when > you are trying to track down a 2.6.27 regression. That is not the case here -- > we know that 2.6.27 works in some configurations. If you continue, I think you > will end up with bad results, or no results. I agree. I found too many uncompilable or unbootable splits. I'm now attempting to manually 'bisect' the config file.
Well that was fun ;-( Using scripts/diffconfig I finally got to the offending setting (after trying all the module differneces first): Bad: LOG_BUF_SHIFT 15 Good: LOG_BUF_SHIFT 17 Also tested good with LOG_BUF_SHIFT 16
*** Bug 250099 has been marked as a duplicate of this bug. ***
Please attach dmesg output from both log buffer size settings? I suspect in the 15 case, the messages you are looking for have simply been pushed off the edge of buffer?
(In reply to comment #36) > Please attach dmesg output from both log buffer size settings? > I suspect in the 15 case, the messages you are looking for have simply been > pushed off the edge of buffer? See the initial post for the good and bad output. Bad: dmesg | grep -i VMI vmi: registering clock source khz=3000000 Good: dmesg | grep -i VMI VMI: Found VMware, Inc. Hypervisor OPROM, API version 3.0, ROM version 1.0 vmi: registering clock event vmi-timer. mult=12582912 shift=22 Booting paravirtualized kernel on vmi vmi: registering clock source khz=3000000 I agree, the log was being truncated.
I was hoping to see the entire log (not just a few lines from unspecified positions) to see that for myself -- but I'll take your word for it. This isn't a bug then.
Created attachment 175192 [details] dmesg.bad
Created attachment 175194 [details] dmesg.good
(In reply to comment #38) > I was hoping to see the entire log (not just a few lines from unspecified > positions) to see that for myself -- but I'll take your word for it. This isn't > a bug then. dmesg files attached. No this isn't a bug, rather a lack of my understanding on how dmesg works and the implications of the LOG_BUF_SHIFT setting.
Great. At least we all learned that it's important to look at the full "dmesg" output before jumping to conclusions on bugs like this. Thanks!
