Bug 648510 – Repeated package-versions in kernel sources

• Product: Websites
• Status: RESOLVED OBSOLETE

[Description Michael 'veremitz' Everitt 2018-02-22 12:15:58 UTC]

Possible re-occurance of metadata issues in package listings eg. https://packages.gentoo.org/packages/sys-kernel/aufs-sources .

See also: https://bugs.gentoo.org/634708#c9

[Comment 1 Michael 'veremitz' Everitt 2018-02-25 11:41:44 UTC]

[Sun 2:06 am] <veremitz> antarus: this is getting silly: https://packages.gentoo.org/packages/sys-block/gparted LOL .. its obviously getting worse ..

I suspect the effect is cumulative over update cycles...

I'll take a look at the ruby code and see if I can decipher enough to be of help. Pointers welcomed of course ;)

[Comment 2 Alec Warner (RETIRED) 2018-02-26 03:36:57 UTC]

I've pushed a UI patch to fix this, but the indexer is still broken.

-A

[Comment 3 Alec Warner (RETIRED) 2018-02-26 14:31:45 UTC]

(In reply to Alec Warner from comment #2)
> I've pushed a UI patch to fix this, but the indexer is still broken.
> 
> -A

So reading the package_import code, its not designed to use the doc id to detect dupes. Instead its relying on some importing code:

def import ... {

index_v = Hash[Version.find_all_by(:package, self.name).map { |v| [v.version, v] }]
model_v = Hash[ebuilds.map { |v| [v.version, v] }]

index_keys = index_v.keys
model_keys = model_v.keys

new_v = model_keys - index_keys
del_v = index_keys - model_keys

... # code to insert new versions

del_v.each do |v|
  index_v[v].delete
end

So we query all versions from the index, then query all versions in the current tree, we extrude 2 maps map[version_string] = Version.

So either del_v ends up being [], or index_v[v].delete isn't deleting items out of elasticsearch.

[Comment 4 Alec Warner (RETIRED) 2018-02-27 04:17:30 UTC]

(In reply to Alec Warner from comment #3)
> (In reply to Alec Warner from comment #2)
> > I've pushed a UI patch to fix this, but the indexer is still broken.
> > 
> > -A
> 
> So reading the package_import code, its not designed to use the doc id to
> detect dupes. Instead its relying on some importing code:
> 
> def import ... {
> 
> index_v = Hash[Version.find_all_by(:package, self.name).map { |v|
> [v.version, v] }]

So I believe the ES6 migration broke this query.

I fixed this in master and v6.0.6:

https://gitweb.gentoo.org/sites/packages.git/commit/?id=950ae1aa1bf3db94e00539f677a445e747209988

I think it works now.

packagestest.gentoo.org has this (and v6.0.8, reverting the UI uniq commit) with a fresh index.

v6.0.6 is deployed on packages.gentoo.org; in theory it will delete the extra version docs during the del. phase of package_import; but its possible a forced import will be required.

> model_v = Hash[ebuilds.map { |v| [v.version, v] }]
> 
> index_keys = index_v.keys
> model_keys = model_v.keys
> 
> new_v = model_keys - index_keys
> del_v = index_keys - model_keys
> 
> ... # code to insert new versions
> 
> del_v.each do |v|
>   index_v[v].delete
> end
> 
> So we query all versions from the index, then query all versions in the
> current tree, we extrude 2 maps map[version_string] = Version.
> 
> So either del_v ends up being [], or index_v[v].delete isn't deleting items
> out of elasticsearch.

I might swap p.g.o with ptest.g.o just to get the main site updated, and do maintenance on the non-live version just to simply things.

-A

[Comment 5 Michael 'veremitz' Everitt 2018-02-27 23:53:34 UTC]

Definitely looking a lot better, Alec, but there do seem to be some strange 4.14 versions sprinkled down the [sorted] listing still .. probably still a lurking gremlin there .. is it picking the right field(s) to sort on, or some-such?!

[Comment 6 Alec Warner (RETIRED) 2018-03-02 02:14:17 UTC]

Duplicate versions should be much rarer now.

[Comment 7 Alec Warner (RETIRED) 2020-05-16 02:34:55 UTC]

We rewrote the entire application.