Created attachment 872129 [details, diff] Example patch of the Directory.Build.targets file to set it up for verbose syntax printing Simply put in order to get the dotnet-sdk bundle to fully build verbosely and too log to emerge log, alot of "hacking" of the xmls used as configs for msbuild/build.sh must be done, as so far all I have is a patch of the general config for true false <LogVerbosityOptOut>false</LogVerbosityOptOut> for this opiton found in the repo projects sub folder for each .proj file, this determines which syntax will be used for implenting the correct syntax for verbose options set in the Directory.Build.targets file (see patch above to correctly setup this file). However while this is a general method, some of the projects found in this bundle, ie the runtime use a Innerbuild command which does not listen to our verbosity due to looking for a completely different file for its config in terms of build options. Overall after changing the Directory target file (see patch) most will work verbosely however some cannont use the true option syntax (-v) and require the other syntax which is only applied when LogVerbosityOptOut is set to false in the individual proj files (ie roslyn needs this or it fails its build). Therefore, we will not be able to fully make each build verbose as some of the builds dont only use the Directory targets file as config (see innerbuild), of which if we tried to hack the file they use would cause issues as they also look for a file which is globally used in the build (we cant set it due to causing conflicts after the runtime build). Therefore in total this bug is to leave a space for verbose patches/hacks to get this ebuild printing more output as it works through all of the projects included in the bundle.
(In reply to Kyle Rabago from comment #0) > Created attachment 872129 [details, diff] [details, diff] > Example patch of the Directory.Build.targets file to set it up for verbose > syntax printing > > Simply put in order to get the dotnet-sdk bundle to fully build verbosely > and too log to emerge log, alot of "hacking" of the xmls used as configs for > msbuild/build.sh must be done, as so far all I have is a patch of the > general config for true false > <LogVerbosityOptOut>false</LogVerbosityOptOut> for this opiton found in the > repo projects sub folder for each .proj file, this determines which syntax > will be used for implenting the correct syntax for verbose options set in > the Directory.Build.targets file (see patch above to correctly setup this > file). However while this is a general method, some of the projects found in > this bundle, ie the runtime use a Innerbuild command which does not listen > to our verbosity due to looking for a completely different file for its > config in terms of build options. Overall after changing the Directory > target file (see patch) most will work verbosely however some cannont use > the true option syntax (-v) and require the other syntax which is only > applied when LogVerbosityOptOut is set to false in the individual proj files > (ie roslyn needs this or it fails its build). Therefore, we will not be able > to fully make each build verbose as some of the builds dont only use the > Directory targets file as config (see innerbuild), of which if we tried to > hack the file they use would cause issues as they also look for a file which > is globally used in the build (we cant set it due to causing conflicts after > the runtime build). Therefore in total this bug is to leave a space for > verbose patches/hacks to get this ebuild printing more output as it works > through all of the projects included in the bundle. Also included in this is the use of args to create a build log that can be set as our emerge log so we can have something for the user to use as a debug for failing builds/etc
(please use some whitespace if you can for big comments) I'm not one of the dotnet people right now, but how is this different from bug 915113 for the purpose of triaging/titling/...?
(In reply to Sam James from comment #2) > (please use some whitespace if you can for big comments) > > I'm not one of the dotnet people right now, but how is this different from > bug 915113 for the purpose of triaging/titling/...? I just wanted to clean up that bug post since it had alot and now that I know that in order to change this build to make it more verbose/loggable we have to make patches and such I wanted a clean place to start, my bad on the whitespaces I am really new to working with bugzilla
(In reply to Sam James from comment #2) > (please use some whitespace if you can for big comments) > > I'm not one of the dotnet people right now, but how is this different from > bug 915113 for the purpose of triaging/titling/...? Sorry if this seems a duplicate just wanted a clean place to start with working with patches and "hacks" of the build (once again I am really new to using bugzilla)
Thanks a lot for the patch Kyle!
(In reply to Maciej Barć from comment #5) > Thanks a lot for the patch Kyle! No worries I am working on testing the build, will upload patches required for the .proj files when build succeeds for v8.0, Thanks as well Maciej!
(In reply to Kyle Rabago from comment #3) > (In reply to Sam James from comment #2) > > (please use some whitespace if you can for big comments) > > > > I'm not one of the dotnet people right now, but how is this different from > > bug 915113 for the purpose of triaging/titling/...? > > I just wanted to clean up that bug post since it had alot and now that I > know that in order to change this build to make it more verbose/loggable we > have to make patches and such I wanted a clean place to start, my bad on the > whitespaces I am really new to working with bugzilla no worries at all - not a telling off :)
(In reply to Sam James from comment #7) > (In reply to Kyle Rabago from comment #3) > > (In reply to Sam James from comment #2) > > > (please use some whitespace if you can for big comments) > > > > > > I'm not one of the dotnet people right now, but how is this different from > > > bug 915113 for the purpose of triaging/titling/...? > > > > I just wanted to clean up that bug post since it had alot and now that I > > know that in order to change this build to make it more verbose/loggable we > > have to make patches and such I wanted a clean place to start, my bad on the > > whitespaces I am really new to working with bugzilla > > no worries at all - not a telling off :) Thanks you all for working with/on Gentoo by the way, really love this community and learning from it/contributing to it!
Other than these changes the only other part of this build that I guess you could say could be made more verbose is the whole Innerbuild thing, as it is really being a pain, as editing the config file for innerbuild may work for say the runtime but it will break on other projects so yeah, would be really nice if we found a way to just tell each build to use the normal level of verbosity like a variable or something but msbuild/and such are very confusing in terms of finding where variables are made in such a big project/bundle of projects
https://git.alpinelinux.org/aports/commit/?id=95d5a315a7a1b8676f7c0546e8249421ed412f9d intresting link by the way of the alpine linux team in terms of building this same sorta package and how they edited their proj/target files
A major question I must know is do we want the verbosity to be set to normal the entire time or have it be set to detailed for all, as detailed can be a big jump in terms of output than normal
Anything more then normal will produce too much logs.
(In reply to Kyle Rabago from comment #11) > A major question I must know is do we want the verbosity to be set to normal > the entire time or have it be set to detailed for all, as detailed can be a > big jump in terms of output than normal or we can even set it by project if we want, I think I have figured out a way to do this even, specfically on fsharp as its normal output is very minimal and does not show much while its detailed output is much more verbose and shows phases, etc
(In reply to Maciej Barć from comment #12) > Anything more then normal will produce too much logs. got it!
Just as a question maricej is there any advantage to creating mutiple ebuilds for each project over having them be one big source just wondering
Wow I actually got it to finally finish a build time to keep on testing though until I can get it to build without any errors
Wow I am also so blind, we still need to edit the directory targets file but we instead of setting verbosity there we can just pass it as an argument to each build.sh script, how did I not think of this earlier!
testing now if this is the case we may be almost perfect with version 8!!!
(In reply to Kyle Rabago from comment #18) > testing now if this is the case we may be almost perfect with version 8!!! As it turns out messing with the source-build-externals and refernce packages to get the verbosity of the nuget pkgs being built is a no go as I ran into errors however I will keep trying to fix this!
Offically have v8 building with all edits needed for a fully automated build, will work on patches (need to make one for each build.sh used) and such later (already have logging working)
In terms of the patches needed, a specific script will be needed to cd into each src projects directory (the one that contains the build.sh) and then patch it according to said name of the project (ie runtime.build.sh.patch will need to be patched in the runtime dir since each patch applies to a file of the same name build.sh. this script will most likely need to be ran after unpacking the tarball in the ebuild before anything else is ran as once its run, we can then use the src_build/top level build.sh to create our emerge log. Speaking of which at the start of every time you run this ebuild we need to make sure we delete our old logs as the size of them is very big (we could also delete it at the end of each build since its not needed unless it fails) to save space. Lemme know if you have a way to apply this script idea and can send me a script to then place in the patches. once again thanks to everyone working on this its been a great learning experience!
(In reply to Kyle Rabago from comment #21) > In terms of the patches needed, a specific script will be needed to cd into > each src > > projects directory (the one that contains the build.sh) and then patch it > according to > > said name of the project (ie runtime.build.sh.patch will need to be patched > in the runtime > > dir since each patch applies to a file of the same name build.sh. this > script will most > > likely need to be ran after unpacking the tarball in the ebuild before > anything else is > > ran as once its run, we can then use the src_build/top level build.sh to > create our emerge > > log. Speaking of which at the start of every time you run this ebuild we > need to make sure > > we delete our old logs as the size of them is very big (we could also delete > it at the end > > of each build since its not needed unless it fails) to save space. Lemme > know if you have > > a way to apply this script idea and can send me a script to then place in > the patches. > > once again thanks to everyone working on this its been a great learning > experience! sorry if whitespacing is bad not sure how to format this correctly (tried to put some space between each comment)
In terms of the patches needed, a specific script will be needed to cd into each src projects directory (the one that contains the build.sh) and then patch it according to said name of the project (ie runtime.build.sh.patch will need to be patched in the runtime dir since each patch applies to a file of the same name build.sh. this script will most likely need to be ran after unpacking the tarball in the ebuild before anything else is ran as once its run, we can then use the src_build/top level build.sh to create our emerge log. Speaking of which at the start of every time you run this ebuild we need to make sure we delete our old logs as the size of them is very big (we could also delete it at the end of each build since its not needed unless it fails) to save space. Lemme know if you have a way to apply this script idea and can send me a script to then place in the patches. once again thanks to everyone working on this its been a great learning experience! this should be a bit better sorry
Are you sure a script is needed? Can we not use some Directory.Build.props file?
(In reply to Maciej Barć from comment #24) > Are you sure a script is needed? > Can we not use some Directory.Build.props file? no because of the way we add the verbosity flag is on each build.sh for each one of the projects (runtime,sdk,msbuild,etc) therefore we need to patch each one seperately as its the only way to make each build work as we need, I am working on the patches right now will upload them as a tarball (each patch is named for what it needs to be applied to ie fsharpbuildsh.patch, etc)
Created attachment 872191 [details] tarball of verbosity patches (needed for logging to work) This is a tarball for all the patches needed and also includes the Directory.Build.targets so old attachment is obsolete, lemme know what you think
(In reply to Kyle Rabago from comment #26) > Created attachment 872191 [details] > tarball of verbosity patches (needed for logging to work) > > This is a tarball for all the patches needed and also includes the > Directory.Build.targets so old attachment is obsolete, lemme know what you > think Once you apply all the patches it should work for version 8
(In reply to Kyle Rabago from comment #27) > (In reply to Kyle Rabago from comment #26) > > Created attachment 872191 [details] > > tarball of verbosity patches (needed for logging to work) > > > > This is a tarball for all the patches needed and also includes the > > Directory.Build.targets so old attachment is obsolete, lemme know what you > > think > > Once you apply all the patches it should work for version 8 Also make sure you add the correct -flp to the top level build.sh to get the log as well!
(In reply to Kyle Rabago from comment #28) > > Also make sure you add the correct -flp to the top level build.sh to get the > log as well! I do not think I understand. If you could provide a (as complete as is possible right now) ebuild how you see it that would help a lot.
I'm pretty sure "normal" verbosity is too verbose, that still shows all the compiler switches and file access checks. "CoreCompile:" is probably the "worst" offender. Not surprised you wnated to delete log files then they would have been extremely big. And to test just how verbose "normal" is: > mkdir Tst > cd Tst > dotnet new console > dotnet build --verbosity normal
(In reply to Maciej Barć from comment #30) > I'm pretty sure "normal" verbosity is too verbose, that still shows all the > compiler switches and file access checks. > > "CoreCompile:" is probably the "worst" offender. Not surprised you wnated to > delete log files then they would have been extremely big. > > And to test just how verbose "normal" is: > > > mkdir Tst > > cd Tst > > dotnet new console > > dotnet build --verbosity normal what you can do is have the log be minimal verbosity -flp:logfile=MyProjectOutput.log;verbosity=minimal
(In reply to Kyle Rabago from comment #31) > (In reply to Maciej Barć from comment #30) > > I'm pretty sure "normal" verbosity is too verbose, that still shows all the > > compiler switches and file access checks. > > > > "CoreCompile:" is probably the "worst" offender. Not surprised you wnated to > > delete log files then they would have been extremely big. > > > > And to test just how verbose "normal" is: > > > > > mkdir Tst > > > cd Tst > > > dotnet new console > > > dotnet build --verbosity normal > > what you can do is have the log be minimal verbosity > -flp:logfile=MyProjectOutput.log;verbosity=minimal if the stdout is verbose normal thats fine just have the log be minimal
(In reply to Kyle Rabago from comment #32) > (In reply to Kyle Rabago from comment #31) > > (In reply to Maciej Barć from comment #30) > > > I'm pretty sure "normal" verbosity is too verbose, that still shows all the > > > compiler switches and file access checks. > > > > > > "CoreCompile:" is probably the "worst" offender. Not surprised you wnated to > > > delete log files then they would have been extremely big. > > > > > > And to test just how verbose "normal" is: > > > > > > > mkdir Tst > > > > cd Tst > > > > dotnet new console > > > > dotnet build --verbosity normal > > > > what you can do is have the log be minimal verbosity > > -flp:logfile=MyProjectOutput.log;verbosity=minimal > > if the stdout is verbose normal thats fine just have the log be minimal also forgot one patch will add new attach one sec
Created attachment 872192 [details] Updated tarball with all patches Updated with one more needed patch
(In reply to Maciej Barć from comment #30) > I'm pretty sure "normal" verbosity is too verbose, that still shows all the > compiler switches and file access checks. > > "CoreCompile:" is probably the "worst" offender. Not surprised you wnated to > delete log files then they would have been extremely big. > > And to test just how verbose "normal" is: > > > mkdir Tst > > cd Tst > > dotnet new console > > dotnet build --verbosity normal I see now hmm I say we just leave it as normal and delete the log after the build/before the build too (since its only needed for debugging a failed build)
(In reply to Kyle Rabago from comment #35) > (In reply to Maciej Barć from comment #30) > > I'm pretty sure "normal" verbosity is too verbose, that still shows all the > > compiler switches and file access checks. > > > > "CoreCompile:" is probably the "worst" offender. Not surprised you wnated to > > delete log files then they would have been extremely big. > > > > And to test just how verbose "normal" is: > > > > > mkdir Tst > > > cd Tst > > > dotnet new console > > > dotnet build --verbosity normal > > I see now hmm I say we just leave it as normal and delete the log after the > build/before the build too (since its only needed for debugging a failed > build) besides being more verbose is always better in my book, esepecially with such a massive source like this bundle
(In reply to Kyle Rabago from comment #36) > (In reply to Kyle Rabago from comment #35) > > (In reply to Maciej Barć from comment #30) > > > I'm pretty sure "normal" verbosity is too verbose, that still shows all the > > > compiler switches and file access checks. > > > > > > "CoreCompile:" is probably the "worst" offender. Not surprised you wnated to > > > delete log files then they would have been extremely big. > > > > > > And to test just how verbose "normal" is: > > > > > > > mkdir Tst > > > > cd Tst > > > > dotnet new console > > > > dotnet build --verbosity normal > > > > I see now hmm I say we just leave it as normal and delete the log after the > > build/before the build too (since its only needed for debugging a failed > > build) > > besides being more verbose is always better in my book, esepecially with > such a massive source like this bundle also the only part of this build that I cant get to fully work is the building of the source built externals as it would require a ton more of unessary patches so those build very minimally
(In reply to Kyle Rabago from comment #37) > (In reply to Kyle Rabago from comment #36) > > (In reply to Kyle Rabago from comment #35) > > > (In reply to Maciej Barć from comment #30) > > > > I'm pretty sure "normal" verbosity is too verbose, that still shows all the > > > > compiler switches and file access checks. > > > > > > > > "CoreCompile:" is probably the "worst" offender. Not surprised you wnated to > > > > delete log files then they would have been extremely big. > > > > > > > > And to test just how verbose "normal" is: > > > > > > > > > mkdir Tst > > > > > cd Tst > > > > > dotnet new console > > > > > dotnet build --verbosity normal > > > > > > I see now hmm I say we just leave it as normal and delete the log after the > > > build/before the build too (since its only needed for debugging a failed > > > build) > > > > besides being more verbose is always better in my book, esepecially with > > such a massive source like this bundle > > also the only part of this build that I cant get to fully work is the > building of the source built externals as it would require a ton more of > unessary patches so those build very minimally in terms of verbosity that is not in terms of the overall build
After testing, version 8 sucessfully built from start to finish and the log is about 180 mb so will need to definetly be removed par each build of the sdk (and its subprojects)
(In reply to Kyle Rabago from comment #39) > After testing, version 8 sucessfully built from start to finish and the log > is about 180 mb so will need to definetly be removed par each build of the > sdk (and its subprojects) I plan on working on the patch script tomorrow (around 12 pm utc). Just to sorta restate my entire goal with this is to make debugging of this build alot easier as there are alot of moving parts in this build! Also in terms of having too much verbosity, I dont know if I would say that necessarily is a bad thing if it helps with fixing builds and finding new bugs,etc. Still even if this isnt necessarily implemented, I still thing giving the build the option (ie use/use_expand flag) could be beneficial. Also In order to make sure we dont make a possibly 200 mb log file each time, I propose removing the log file if the build/pkg installs successfully, and also making sure we remove it before starting a new build. Overall I think this is a worth while fix to make Maciej and Dotnet team's build even better. (Also btw I will make sure to look into minimal verbosity tomorrow as well but as it stands I really like normal due to its shear amount of info given to possible bug hunters/users/etc but this is definitely something that could be changed) (And while I wish I could just change one file for this, the Dir.Build.props file/s sometimes doesnt even have any args in terms of verbosity and instead use another file which can cause syntax conflicts (see v:n / -v n) so therefore its better to let the build.sh script to auto set and make sure the correct syntax is passed)
I have the patch script working, need to add places for the dir of the patches/sdk in terms of portage and it should work for v8
Created attachment 872241 [details] Script to apply patches (absolute paths as of now, change where needed to make it work with portage) In order for this to work you will need to pretty much change the locations of the patches/sdk as it hasent been modified to work with portage yet
(In reply to Kyle Rabago from comment #42) > Created attachment 872241 [details] > Script to apply patches (absolute paths as of now, change where needed to > make it work with portage) > > In order for this to work you will need to pretty much change the locations > of the patches/sdk as it hasent been modified to work with portage yet FOR VERSION 8 as of now havent looked into v7 (is next)
Today I will work on getting this script implemented into the ebuild to get it fully working at least for version 8, will upload it when done!
If needed you can just edit the dir props target file/s to remove the redirct log or something or just auto tail the log or something but as of now I dont really have an opinion on what could be done to improve it anymore without drastically changing the ebuild
