The performance of ruby-ng.eclass is pretty poor for global scope sourcing. Using 'pk' (from sys-apps/pkgcraft-tools), all the top results (after recent fixes in the tree) are Ruby packages using ruby-ng.eclass: ``` $ time pk pkg source --sort --repo ~/g/ > list $ tail -n 15 list dev-ruby/yard-0.9.34::/home/sam/g/: 80.249ms dev-ruby/whole_history_rating-0.1.2-r2::/home/sam/g/: 80.256ms dev-ruby/sinatra-3.0.5::/home/sam/g/: 80.459ms dev-ruby/multi_json-1.15.0::/home/sam/g/: 80.572ms dev-ruby/slim-4.1.0-r4::/home/sam/g/: 80.614ms dev-ruby/terminal-table-2.0.0-r1::/home/sam/g/: 80.908ms dev-ruby/webmock-3.18.1::/home/sam/g/: 81.245ms dev-ruby/sinatra-2.2.3::/home/sam/g/: 81.379ms dev-ruby/xmlrpc-0.3.2-r1::/home/sam/g/: 82.057ms dev-ruby/ruby-gettext-3.4.4::/home/sam/g/: 82.075ms dev-ruby/ruby-gettext-3.4.3::/home/sam/g/: 82.082ms dev-ruby/sprockets-4.2.0-r1::/home/sam/g/: 82.45ms dev-ruby/sinatra-3.0.5-r1::/home/sam/g/: 83.737ms dev-ruby/sinatra-3.0.6::/home/sam/g/: 84.146ms dev-ruby/yard-0.9.28::/home/sam/g/: 85.483ms ``` mgorny's suggested some ideas which should get us some easy wins without a big redesign (yet): """ [20:23:10] <@mgorny> sam_: i'd start by undoing subshells of _ruby_atoms* [20:23:38] <@mgorny> that's used by ruby_add* [20:23:57] <@sam_> thanks! [20:24:22] <@mgorny> _ruby_atoms_samelib_generic, _ruby_atoms_samelib can be made non-subshelled in place, they're not used elsewhere [20:24:38] <@mgorny> _ruby_get_all_impls is also a good candidate [20:24:52] <@mgorny> i wonder if you can set them globally [20:25:07] <@mgorny> hm, no, it uses USE_RUBY [20:25:24] <@mgorny> but i suppose you could unsubshell it too, and cache the result [20:25:55] <@mgorny> i'd also do ruby2[0-7] there [20:26:27] <+negril> is ruby2 even relevant any more? [20:26:44] <@sam_> it's a list of dead impls in this case [20:26:51] <@sam_> thanks for the ideas, much appreciated, going to try them now [20:28:30] <@sam_> _ruby_atoms_samelib is used in a few places [20:28:41] <@sam_> caching is not a bad idea """
Baseline: ``` $ time pk pkg source --repo ~/g/ 'dev-ruby/*' [...] real 0m4.159s user 0m24.702s sys 0m42.332s $ time pk pkg source --sort --repo ~/g/ > list $ tail -n 15 list dev-ruby/yard-0.9.34::/home/sam/g/: 80.249ms dev-ruby/whole_history_rating-0.1.2-r2::/home/sam/g/: 80.256ms dev-ruby/sinatra-3.0.5::/home/sam/g/: 80.459ms dev-ruby/multi_json-1.15.0::/home/sam/g/: 80.572ms dev-ruby/slim-4.1.0-r4::/home/sam/g/: 80.614ms dev-ruby/terminal-table-2.0.0-r1::/home/sam/g/: 80.908ms dev-ruby/webmock-3.18.1::/home/sam/g/: 81.245ms dev-ruby/sinatra-2.2.3::/home/sam/g/: 81.379ms dev-ruby/xmlrpc-0.3.2-r1::/home/sam/g/: 82.057ms dev-ruby/ruby-gettext-3.4.4::/home/sam/g/: 82.075ms dev-ruby/ruby-gettext-3.4.3::/home/sam/g/: 82.082ms dev-ruby/sprockets-4.2.0-r1::/home/sam/g/: 82.45ms dev-ruby/sinatra-3.0.5-r1::/home/sam/g/: 83.737ms dev-ruby/sinatra-3.0.6::/home/sam/g/: 84.146ms dev-ruby/yard-0.9.28::/home/sam/g/: 85.483ms ```
Post-fixes: ``` $ time pk pkg source --repo ~/g/ 'dev-ruby/*' [...] real 0m1.972s user 0m16.414s sys 0m15.125s $ time pk pkg source --sort --repo ~/g/ > list $ tail -n 15 list dev-ruby/sinatra-3.0.6::/home/sam/g/: 38.876ms dev-libs/xapian-bindings-1.4.22::/home/sam/g/: 38.926ms dev-python/cryptography-40.0.2::/home/sam/g/: 39.212ms dev-lang/rust-1.65.0::/home/sam/g/: 39.296ms x11-terms/wezterm-20230326.111934::/home/sam/g/: 39.344ms dev-lang/rust-1.70.0::/home/sam/g/: 40.041ms dev-lang/rust-1.66.1::/home/sam/g/: 40.781ms dev-lang/rust-1.68.2::/home/sam/g/: 41.276ms gnome-base/librsvg-2.56.0::/home/sam/g/: 42.456ms gnome-base/librsvg-2.56.0-r1::/home/sam/g/: 42.899ms dev-python/cryptography-41.0.1::/home/sam/g/: 43.414ms dev-util/maturin-1.1.0::/home/sam/g/: 45.034ms gnome-base/librsvg-2.56.1::/home/sam/g/: 45.176ms dev-util/maturin-1.0.1::/home/sam/g/: 52.503ms dev-vcs/reposurgeon-4.27::/home/sam/g/: 52.534ms ```
Nice!
The bug has been referenced in the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=8478835a7fc4226bcbf6c79475e1ba6d41d42298 commit 8478835a7fc4226bcbf6c79475e1ba6d41d42298 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-06-14 20:47:35 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-06-15 21:13:40 +0000 ruby-ng.eclass: use shopt directly, not via estack.eclass ulm points out that estack.eclass is particularly inefficient (although it'll get slightly better once https://github.com/gentoo/gentoo/pull/31437 is fixed). Let's just manually roll it like llvm.eclass does. Bug: https://bugs.gentoo.org/908465 Signed-off-by: Sam James <sam@gentoo.org> eclass/ruby-ng.eclass | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=48a0a44098b5cdfbceaf37305b7e9eac65748cd0 commit 48a0a44098b5cdfbceaf37305b7e9eac65748cd0 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-06-13 22:01:11 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-06-15 21:12:17 +0000 ruby-ng.eclass: optimize: avoid subshells for ruby_implementations_depend, ruby_get_use_targets We go from 2.5s -> 1.9s to source dev-ruby/*. Bug: https://bugs.gentoo.org/908465 Signed-off-by: Sam James <sam@gentoo.org> eclass/ruby-ng.eclass | 36 ++++++++++++++++++++++++++++-------- 1 file changed, 28 insertions(+), 8 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=676e6dc3f4a921d4fcb1e320e7d0562e2ef08856 commit 676e6dc3f4a921d4fcb1e320e7d0562e2ef08856 Author: Sam James <sam@gentoo.org> AuthorDate: 2023-06-13 21:59:47 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-06-15 21:12:16 +0000 ruby-ng.eclass: optimize: avoid subshells for _ruby_atoms_samelib* - Inline ruby_atoms_samelib (only used by one caller) - Avoid repeated (subshell) calls to _ruby_atoms_samelib_generic by using a result variable instead. We go from 3.5s -> 2.5s to source dev-ruby/*. Thanks to mgorny for the ideas here. Bug: https://bugs.gentoo.org/908465 Signed-off-by: Sam James <sam@gentoo.org> eclass/ruby-ng.eclass | 57 ++++++++++++++++++++++++++------------------------- 1 file changed, 29 insertions(+), 28 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=390ad2effe1c83a7cb25a892ad988937b53c07fe commit 390ad2effe1c83a7cb25a892ad988937b53c07fe Author: Sam James <sam@gentoo.org> AuthorDate: 2023-06-13 21:57:53 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-06-15 21:12:16 +0000 ruby-ng.eclass: optimize: avoid subshell for ruby_get_all_impls We go from ~4s -> ~3.5s for sourcing dev-ruby/*. For sinatra: ``` $ pk pkg source $(pkg) --repo ~/g/ --bench 5s # before dev-ruby/sinatra-2.2.3::/home/sam/g/: mean: 66.68ms, min: 56.938ms, max: 74.248ms, σ = 4.832ms, N = 75 dev-ruby/sinatra-3.0.6::/home/sam/g/: mean: 73.618ms, min: 60.153ms, max: 77.978ms, σ = 3.195ms, N = 68 dev-ruby/sinatra-3.0.5::/home/sam/g/: mean: 72.069ms, min: 58.736ms, max: 78.223ms, σ = 3.277ms, N = 70 dev-ruby/sinatra-3.0.5-r1::/home/sam/g/: mean: 73.265ms, min: 60.738ms, max: 81.06ms, σ = 3.227ms, N = 69 $ pk pkg source $(pkg) --repo ~/g/ --bench 5s # after dev-ruby/sinatra-3.0.5-r1::/home/sam/g/: mean: 59.677ms, min: 49.141ms, max: 63.282ms, σ = 2.511ms, N = 84 dev-ruby/sinatra-3.0.6::/home/sam/g/: mean: 59.693ms, min: 48.637ms, max: 62.862ms, σ = 2.628ms, N = 84 dev-ruby/sinatra-3.0.5::/home/sam/g/: mean: 56.697ms, min: 46.782ms, max: 60.367ms, σ = 2.822ms, N = 89 dev-ruby/sinatra-2.2.3::/home/sam/g/: mean: 54.915ms, min: 45.832ms, max: 59.513ms, σ = 3.52ms, N = 92 ``` Bug: https://bugs.gentoo.org/908465 Signed-off-by: Sam James <sam@gentoo.org> eclass/ruby-ng.eclass | 23 +++++++++++++---------- 1 file changed, 13 insertions(+), 10 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=89925fc4e6d343ce6f7ca36bc288ba7b62d1193e commit 89925fc4e6d343ce6f7ca36bc288ba7b62d1193e Author: Sam James <sam@gentoo.org> AuthorDate: 2023-06-13 21:52:12 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-06-15 21:12:15 +0000 ruby-ng.eclass: optimize: use pattern substitution We can save a little bit (consistently a few ms) by using patsubs in some obvious cases. Not really any difference globally, but for sinatra: ``` $ pk pkg source $(pkg) --repo ~/g/ --bench 5s # before dev-ruby/sinatra-3.0.5::/home/sam/g/: mean: 76.25ms, min: 59.23ms, max: 83.674ms, σ = 4.247ms, N = 66 dev-ruby/sinatra-3.0.5-r1::/home/sam/g/: mean: 77.465ms, min: 61.782ms, max: 85.127ms, σ = 3.592ms, N = 65 dev-ruby/sinatra-3.0.6::/home/sam/g/: mean: 80.192ms, min: 60.922ms, max: 84.951ms, σ = 3.899ms, N = 63 dev-ruby/sinatra-2.2.3::/home/sam/g/: mean: 80.389ms, min: 56.818ms, max: 86.915ms, σ = 4.508ms, N = 63 $ pk pkg source $(pkg) --repo ~/g/ --bench 5s # after dev-ruby/sinatra-2.2.3::/home/sam/g/: mean: 66.68ms, min: 56.938ms, max: 74.248ms, σ = 4.832ms, N = 75 dev-ruby/sinatra-3.0.6::/home/sam/g/: mean: 73.618ms, min: 60.153ms, max: 77.978ms, σ = 3.195ms, N = 68 dev-ruby/sinatra-3.0.5::/home/sam/g/: mean: 72.069ms, min: 58.736ms, max: 78.223ms, σ = 3.277ms, N = 70 dev-ruby/sinatra-3.0.5-r1::/home/sam/g/: mean: 73.265ms, min: 60.738ms, max: 81.06ms, σ = 3.227ms, N = 69 ``` Bug: https://bugs.gentoo.org/908465 Signed-off-by: Sam James <sam@gentoo.org> eclass/ruby-ng.eclass | 12 +++++------- 1 file changed, 5 insertions(+), 7 deletions(-) https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=0bb6c7f95645b618832dea16ef2d1ecdfb5ad0ed commit 0bb6c7f95645b618832dea16ef2d1ecdfb5ad0ed Author: Sam James <sam@gentoo.org> AuthorDate: 2023-06-13 21:49:57 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2023-06-15 21:12:14 +0000 ruby-ng.eclass: optimize: use pattern for old ruby impls Not much improvement on a grand scale, but for dev-ruby/sinatra at least, we get: ``` $ pk pkg source $(pkg) --repo ~/g/ --bench 5s # before dev-ruby/sinatra-3.0.5::/home/sam/g/: mean: 76.25ms, min: 59.23ms, max: 83.674ms, σ = 4.247ms, N = 66 dev-ruby/sinatra-3.0.5-r1::/home/sam/g/: mean: 77.465ms, min: 61.782ms, max: 85.127ms, σ = 3.592ms, N = 65 dev-ruby/sinatra-3.0.6::/home/sam/g/: mean: 80.192ms, min: 60.922ms, max: 84.951ms, σ = 3.899ms, N = 63 dev-ruby/sinatra-2.2.3::/home/sam/g/: mean: 80.389ms, min: 56.818ms, max: 86.915ms, σ = 4.508ms, N = 63 $ pk pkg source $(pkg) --repo ~/g/ --bench 5s # after dev-ruby/sinatra-3.0.6::/home/sam/g/: mean: 72.761ms, min: 58.627ms, max: 76.161ms, σ = 3.276ms, N = 69 dev-ruby/sinatra-3.0.5-r1::/home/sam/g/: mean: 72.967ms, min: 60.127ms, max: 76.75ms, σ = 3.176ms, N = 69 dev-ruby/sinatra-3.0.5::/home/sam/g/: mean: 69.004ms, min: 58.344ms, max: 73.661ms, σ = 3.3ms, N = 73 dev-ruby/sinatra-2.2.3::/home/sam/g/: mean: 71.061ms, min: 55.144ms, max: 74.563ms, σ = 3.848ms, N = 71 ``` A tiny, but seemingly consistent improvement. Thanks to mgorny for the idea. Bug: https://bugs.gentoo.org/908465 Signed-off-by: Sam James <sam@gentoo.org> eclass/ruby-ng.eclass | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(In reply to Hans de Graaff from comment #3) > Nice! Thanks! :)
