Possible re-occurance of metadata issues in package listings eg. https://packages.gentoo.org/packages/sys-kernel/aufs-sources . See also: https://bugs.gentoo.org/634708#c9
[Sun 2:06 am] <veremitz> antarus: this is getting silly: https://packages.gentoo.org/packages/sys-block/gparted LOL .. its obviously getting worse .. I suspect the effect is cumulative over update cycles... I'll take a look at the ruby code and see if I can decipher enough to be of help. Pointers welcomed of course ;)
I've pushed a UI patch to fix this, but the indexer is still broken. -A
(In reply to Alec Warner from comment #2) > I've pushed a UI patch to fix this, but the indexer is still broken. > > -A So reading the package_import code, its not designed to use the doc id to detect dupes. Instead its relying on some importing code: def import ... { index_v = Hash[Version.find_all_by(:package, self.name).map { |v| [v.version, v] }] model_v = Hash[ebuilds.map { |v| [v.version, v] }] index_keys = index_v.keys model_keys = model_v.keys new_v = model_keys - index_keys del_v = index_keys - model_keys ... # code to insert new versions del_v.each do |v| index_v[v].delete end So we query all versions from the index, then query all versions in the current tree, we extrude 2 maps map[version_string] = Version. So either del_v ends up being [], or index_v[v].delete isn't deleting items out of elasticsearch.
(In reply to Alec Warner from comment #3) > (In reply to Alec Warner from comment #2) > > I've pushed a UI patch to fix this, but the indexer is still broken. > > > > -A > > So reading the package_import code, its not designed to use the doc id to > detect dupes. Instead its relying on some importing code: > > def import ... { > > index_v = Hash[Version.find_all_by(:package, self.name).map { |v| > [v.version, v] }] So I believe the ES6 migration broke this query. I fixed this in master and v6.0.6: https://gitweb.gentoo.org/sites/packages.git/commit/?id=950ae1aa1bf3db94e00539f677a445e747209988 I think it works now. packagestest.gentoo.org has this (and v6.0.8, reverting the UI uniq commit) with a fresh index. v6.0.6 is deployed on packages.gentoo.org; in theory it will delete the extra version docs during the del. phase of package_import; but its possible a forced import will be required. > model_v = Hash[ebuilds.map { |v| [v.version, v] }] > > index_keys = index_v.keys > model_keys = model_v.keys > > new_v = model_keys - index_keys > del_v = index_keys - model_keys > > ... # code to insert new versions > > del_v.each do |v| > index_v[v].delete > end > > So we query all versions from the index, then query all versions in the > current tree, we extrude 2 maps map[version_string] = Version. > > So either del_v ends up being [], or index_v[v].delete isn't deleting items > out of elasticsearch. I might swap p.g.o with ptest.g.o just to get the main site updated, and do maintenance on the non-live version just to simply things. -A
Definitely looking a lot better, Alec, but there do seem to be some strange 4.14 versions sprinkled down the [sorted] listing still .. probably still a lurking gremlin there .. is it picking the right field(s) to sort on, or some-such?!
Duplicate versions should be much rarer now.
We rewrote the entire application.
