Bug 664588 – Dynamic indexing causes elasticsearch to hit field limit.

• Product: Websites
• Status: RESOLVED OBSOLETE

[Description Conrad Kostecki 2018-08-25 15:45:15 UTC]

When looking at PGO, I can see the newest commits, but the new package version is not shown by PGO.

E.G.: https://packages.gentoo.org/packages/media-sound/teamspeak-server

Reproducible: Always

[Comment 1 Alec Warner (RETIRED) 2018-08-25 16:46:13 UTC]

Yeah I think Dynamic field detection in ES is causing the PUTs to fail.

I'll see if I can turn them off.

-A

[Comment 2 Conrad Kostecki 2018-09-02 14:29:07 UTC]

Seems to be fixed, as current packages are now shown again on PGO.

[Comment 3 Alec Warner (RETIRED) 2018-09-02 16:29:26 UTC]

(In reply to Conrad Kostecki from comment #2)
> Seems to be fixed, as current packages are now shown again on PGO.

Not quite, still waiting to verify it doesn't come back.

-A

[Comment 4 Conrad Kostecki 2019-10-24 20:28:45 UTC]

Seems to be still a problem, since I see from time to time an deprecated package list on p.g.o.

[Comment 5 Conrad Kostecki 2019-12-05 17:51:01 UTC]

There seems to be currently some problems, as packages seems to be outdated.

For example: https://packages.gentoo.org/packages/games-simulation/openrct2

[Comment 6 Michael 'veremitz' Everitt 2019-12-06 09:40:11 UTC]

(In reply to Conrad Kostecki from comment #5)
> There seems to be currently some problems, as packages seems to be outdated.
> 
> For example: https://packages.gentoo.org/packages/games-simulation/openrct2

This isn't an unusual problem, but it seems the root cause(s) have never been fully traced. It needs someone with good knowledge of Ruby and ElasticSearch (or more free time, if this doesn't apply to Alec as interim maintainer).

[Comment 7 Max Magorsch (RETIRED) 2019-12-22 00:28:00 UTC]

Dynamic mapping has been completely turned off in #3c9eaa1. To make a long story short, the structure of the index had to be changed to avoid dynamic mapping. That's what happened in #ebd404d and #9fb7236 so that dynamic mapping is not needed anymore.

So the field limit should definitely not be hit anymore. Furthermore issues due to dynamic mapping should not be present anymore as dynamic mapping is turned of and an exception is thrown if new fields are detected. 

However, I am not completely sure whether this is already deployed to p.g.o and whether a fresh index has been populated already.

[Comment 8 Alec Warner (RETIRED) 2020-05-26 07:48:37 UTC]

We don't use ES anymore.

-A