dev-libs/geoip-1.6.12 # /usr/sbin/geoipupdate.sh -f Updating GeoIP databases... 2019-01-08 08:51:53 URL:https://download.maxmind.com/download/geoip/database/GeoIPv6.dat.gz [1219041/1219041] -> "geoipupdate.OecPyhpL52/GeoIPv6.dat.gz" [1] https://download.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz: 2019-01-08 08:51:53 ERROR 404: Not Found. 2019-01-08 08:51:54 URL:https://download.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz [15272147/15272147] -> "geoipupdate.OecPyhpL52/GeoLiteCityv6.dat.gz" [1] https://download.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz: 2019-01-08 08:51:54 ERROR 404: Not Found. https://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz: 2019-01-08 08:51:54 ERROR 404: Not Found. 2019-01-08 08:51:54 URL:https://download.maxmind.com/download/geoip/database/asnum/GeoIPASNumv6.dat.gz [3085152/3085152] -> "geoipupdate.OecPyhpL52/GeoIPASNumv6.dat.gz" [1] # equery belongs /usr/sbin/geoipupdate.sh * Searching for /usr/sbin/geoipupdate.sh ... dev-libs/geoip-1.6.12 (/usr/sbin/geoipupdate.sh)
/usr/sbin/geoipupdate.sh is maintained downstream. Originally introduced for infra I think.
Looks like they might have moved it to [1]. Not sure if this will work. diff --git a/dev-libs/geoip/files/geoipupdate-r6.sh b/dev-libs/geoip/files/geoipupdate-r6.sh index 83b58101663..9398bc8670c 100644 --- a/dev-libs/geoip/files/geoipupdate-r6.sh +++ b/dev-libs/geoip/files/geoipupdate-r6.sh @@ -1,6 +1,6 @@ #!/bin/sh -GEOIP_MIRROR="https://download.maxmind.com/download/geoip/database" +GEOIP_MIRROR="https://geolite.maxmind.com/download/geoip/database/" GEOIPDIR=@PREFIX@/usr/share/GeoIP TMPDIR= [1] https://geolite.maxmind.com/download/geoip/database/
# geoipupdate.sh -f Updating GeoIP databases... 2019-01-08 12:11:40 URL:https://geolite.maxmind.com/download/geoip/database//GeoIPv6.dat.gz [1219041/1219041] -> "geoipupdate.cgHEreYBKo/GeoIPv6.dat.gz" [1] https://geolite.maxmind.com/download/geoip/database//GeoLiteCity.dat.gz: 2019-01-08 12:11:40 ERROR 404: Not Found. 2019-01-08 12:11:45 URL:https://geolite.maxmind.com/download/geoip/database//GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz [15272147/15272147] -> "geoipupdate.cgHEreYBKo/GeoLiteCityv6.dat.gz" [1] https://geolite.maxmind.com/download/geoip/database//GeoLiteCountry/GeoIP.dat.gz: 2019-01-08 12:11:46 ERROR 404: Not Found. https://geolite.maxmind.com/download/geoip/database//asnum/GeoIPASNum.dat.gz: 2019-01-08 12:11:46 ERROR 404: Not Found. 2019-01-08 12:11:48 URL:https://geolite.maxmind.com/download/geoip/database//asnum/GeoIPASNumv6.dat.gz [3085152/3085152] -> "geoipupdate.cgHEreYBKo/GeoIPASNumv6.dat.gz" [1]
I think there's not much we can do: "On January 2, 2019, we discontinued our GeoLite Legacy databases. This discontinuation was first announced on January 2, 2018." https://support.maxmind.com/geolite-legacy-discontinuation-notice/ => Legacy GeoLite is dead, move to GeoLite2.
(In reply to Jeroen Roovers from comment #3) > # geoipupdate.sh -f > Updating GeoIP databases... > 2019-01-08 12:11:40 > URL:https://geolite.maxmind.com/download/geoip/database//GeoIPv6.dat.gz > [1219041/1219041] -> "geoipupdate.cgHEreYBKo/GeoIPv6.dat.gz" [1] This one still works. (In reply to Thomas Deutschmann from comment #4) > I think there's not much we can do: So why do you suggest this?
(In reply to Jeroen Roovers from comment #5) > So why do you suggest this? User is reporting that geoipupdate.sh is unable to fetch GeoIP database updates anymore. But there is nothing we can do about this because there will be _no_ updates anymore. See the upstream document from comment #4 or the following quote from https://blog.maxmind.com/2018/01/02/discontinuation-of-the-geolite-legacy-databases/: > January 2, 2019 – The remaining GeoLite Legacy builds _will be removed from our website_. GeoLite Legacy database users will need to have switched to the GeoLite2 or commercial GeoIP databases and update their integrations. (This was 2018, even existing customers cannot switch to commercial GeoIP database anymore, only v2 is available) So while I cannot tell you why "GeoIPv6.dat.gz" is still available, I can tell you for sure that GeoLite Legacy is history. Therefore the only thing we could do is finding a mirror which contains last published version but the _updater_ itself will never ever be able to pull any _updates_ because there won't be any updates for GeoLite Legacy anymore. But if we will find a new mirror for old data we would actually hide the fact that GeoIP Lite is discontinued. I.e. people relying on GeoIP data wouldn't notice that they are probably using outdated and therefore invalid data because geoipupdate.sh won't report an error (geoipupdate.sh would just do nothing because there is no new file).
So if I'm not mistaken the geoip update script in dev-libs/geoip should be removed, and a warning should be issued regarding the GeoIP Legacy DB EOL. Also, the user might be informed of the geoipv2 DB, and that the update script shipped with net-misc/geoipupdate should be used instead.
It's still not perfectly clear to me how one should fully take advantage of the free GeoIP databases according to the new v2 release being there several portage packages. This is my guess: 1) install net-misc/geoipupdate, configure /etc/GeoIP.conf accordingly (especially DatabaseDirectory), and run geoipupdate 2) install dev-libs/geoip if you need to use /usr/bin/geoiplookup. The downstream geoipupdate.sh script should either be removed, or the user should be warned that it's basically useless. However, I still have NO IDEA if the geoiplookup program will work with the new V2 database. 3) For those who make use of xtables-addons, I've run into this: # /lib/xtables-addons/xt_geoip_dl --2019-01-10 10:01:41-- http://geolite.maxmind.com/download/geoip/database/GeoIPv6.csv.gz Resolving geolite.maxmind.com... 104.16.38.47, 104.16.37.47, 2606:4700::6810:262f, ... Connecting to geolite.maxmind.com|104.16.38.47|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 1935725 (1.8M) [application/octet-stream] Saving to: 'GeoIPv6.csv.gz' GeoIPv6.csv.gz 100%[========================================================================================================================================>] 1.85M --.-KB/s in 0.08s 2019-01-10 10:01:41 (23.8 MB/s) - 'GeoIPv6.csv.gz' saved [1935725/1935725] --2019-01-10 10:01:41-- http://geolite.maxmind.com/download/geoip/database/GeoIPCountryCSV.zip Reusing existing connection to geolite.maxmind.com:80. HTTP request sent, awaiting response... 404 Not Found 2019-01-10 10:01:41 ERROR 404: Not Found. FINISHED --2019-01-10 10:01:41-- Total wall clock time: 0.1s Downloaded: 1 files, 1.8M in 0.08s (23.8 MB/s) unzip: cannot find or open GeoIPCountryCSV.zip, GeoIPCountryCSV.zip.zip or GeoIPCountryCSV.zip.ZIP. # equery belongs /lib/xtables-addons/xt_geoip_dl * Searching for /lib/xtables-addons/xt_geoip_dl ... net-firewall/xtables-addons-2.12 (/lib64/xtables-addons/xt_geoip_dl) However, I currently cannot test the latest xtables-addons ebuild in portage to see if it solves this issue and uses GeoIPv2.
I've just tested, and found that dev-libs/geoip's /usr/bin/geoiplookup does NOT work with GeoIP V2. So I guess dev-libs/geoip can be removed or patched somehow to make the lookup work with V2.
To substitute the use of dev-libs/geoip's geoiplookup, one can install dev-libs/libmaxminddb and run: # mmdblookup --file /usr/share/GeoIP/GeoLite2-City.mmdb --ip 89.16.167.134 country names en "United Kingdom" <utf8_string> # mmdblookup --file /usr/share/GeoIP/GeoLite2-City.mmdb --ip 89.16.167.134 city names en "Manchester" <utf8_string> # mmdblookup --file /usr/share/GeoIP/GeoLite2-City.mmdb --ip 89.16.167.134 postal code "M40" <utf8_string> The output is different, but at least it works. So, I gather dev-libs/geoip is obsolete unless maybe for those who might be using the Legacy commercial versions.
Also, for people previously using AS they would need to edit their GeoIP.conf file like so: # grep EditionIDs /etc/GeoIP.conf EditionIDs GeoLite2-Country GeoLite2-City GeoLite2-ASN The "GeoLite2-ASN" DB is not documented upstream as a free DB, but it is. # mmdblookup --file /usr/share/GeoIP/GeoLite2-ASN.mmdb --ip 89.16.167.134 autonomous_system_organization "Bytemark Limited" <utf8_string>
dev-libs/geoip is the legacy version which requires legacy database. Because there won't be any legacy database updates anymore, legacy geoip package will be useless for people without paid subscription for legacy database. Most users will have to migrate to geoip2 solutions. dev-libs/libmaxminddb is the successor of dev-libs/geoip. >=net-firewall/xtables-addons-3 supports GeoIP2, too.
(In reply to Thomas Deutschmann from comment #12) > dev-libs/libmaxminddb is the successor of dev-libs/geoip. I didn't know that. Now everything seems to be cleared up. Thanks
(In reply to Thomas Deutschmann from comment #12) > >=net-firewall/xtables-addons-3 supports GeoIP2, too. Yes, but >=net-firewall/xtables-addons-3 versions aren't stable yet, and they need a >=4.15 kernel, which isn't stable either. Is there a way to make the existing stable xtables geoip module and the geoip2 mmdb databases work together ?
*** Bug 678262 has been marked as a duplicate of this bug. ***
I confirm this issue. # geoipupdate.sh --force Updating GeoIP databases... https://download.maxmind.com/download/geoip/database/GeoIPv6.dat.gz: 2019-03-01 03:25:25 ERROR 404: Not Found. https://download.maxmind.com/download/geoip/database/GeoLiteCity.dat.gz: 2019-03-01 03:25:25 ERROR 404: Not Found. https://download.maxmind.com/download/geoip/database/GeoLiteCityv6-beta/GeoLiteCityv6.dat.gz: 2019-03-01 03:25:26 ERROR 404: Not Found. https://download.maxmind.com/download/geoip/database/GeoLiteCountry/GeoIP.dat.gz: 2019-03-01 03:25:27 ERROR 404: Not Found. https://download.maxmind.com/download/geoip/database/asnum/GeoIPASNum.dat.gz: 2019-03-01 03:25:27 ERROR 404: Not Found. https://download.maxmind.com/download/geoip/database/asnum/GeoIPASNumv6.dat.gz: 2019-03-01 03:25:28 ERROR 404: Not Found.
(In reply to Thomas Deutschmann from comment #12) > Most users will have to migrate to geoip2 solutions. Is there any ebuilds for geoip2?
(In reply to Viktor Levin from comment #17) > (In reply to Thomas Deutschmann from comment #12) > > Most users will have to migrate to geoip2 solutions. > > Is there any ebuilds for geoip2? dev-libs/libmaxminddb
dev-libs/geoip upstream accordingly is a commit at https://github.com/maxmind/geoip-api-c/commit/44e5ffc061c22723a0b7f858a54888f5ab96f3cb --- @@ -12,10 +12,6 @@ BuildRoot: %{_tmppath}/%{name}-%{version}-root %description GeoIP is a C library that enables the user to find geographical and network information of an IP address. - Included is a free GeoLite Country database - http://dev.maxmind.com/geoip/geolite --- Are you sure they are willing to give away some free geoip2 data? For me the only consumer is dev-libs/geoip-1.6.12 pulled in by: net-p2p/qbittorrent-9999 requires dev-libs/geoip Is it possible for qbittorrent to have a new USE to disable geoip ?
There are some unofficial databases in legacy format at, https://mailfud.org/geoip-legacy/ but I think we should rather be encouraging people to update to the newer geoip2 stuff. There are already PHP and C clients in the tree, but Perl (for SpamAssassin) is notably missing.
Debian is converting GeoLite2 csv databases to legacy csv and then buiding dat from these csv files. https://sources.debian.org/src/geoip/1.6.12-5/debian/src/ https://sources.debian.org/src/geoip/1.6.12-5/debian/rules/ https://sources.debian.org/src/geoip-database/20191108-1/debian/rules/ Script geolite2-to-legacy-csv.sh (it's actually a perl script) they are using is from here: https://github.com/mschmitt/GeoLite2xtables/
(In reply to Alexander Tsoy from comment #21) > Debian is converting GeoLite2 csv databases to legacy csv and then buiding > dat from these csv files. > > https://sources.debian.org/src/geoip/1.6.12-5/debian/src/ > https://sources.debian.org/src/geoip/1.6.12-5/debian/rules/ > https://sources.debian.org/src/geoip-database/20191108-1/debian/rules/ > > > Script geolite2-to-legacy-csv.sh (it's actually a perl script) they are > using is from here: > > https://github.com/mschmitt/GeoLite2xtables/ Thanks for the info!
dev-perl/GeoIP2 now in tree, that should help some of you.
According to https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and-using-geolite2-databases/ there is not way to get geoip databases without maxmind account anymore. geoipupdate.sh should be removed.
(In reply to Dmitry Suloev from comment #24) > According to > https://blog.maxmind.com/2019/12/18/significant-changes-to-accessing-and- > using-geolite2-databases/ there is not way to get geoip databases without > maxmind account anymore. geoipupdate.sh should be removed. Removed outright no. Converted to only run with the account key yes.
(In reply to Robin Johnson from comment #25) > Removed outright no. Converted to only run with the account key yes. What is point to keep this script when geoipupdate available in portage and can be used to download both versions of db?
(In reply to Dmitry Suloev from comment #26) > (In reply to Robin Johnson from comment #25) > > Removed outright no. Converted to only run with the account key yes. > > What is point to keep this script when geoipupdate available in portage and > can be used to download both versions of db? geoipudate in portage can only download GeoLite2 dbs.
Well, /usr/sbin/geoipupdate.sh returns 404 errors. Maybe you should remove 'ewarn' about it from ebuild?
(In reply to Andrey Aleksandrovich from comment #28) > Well, /usr/sbin/geoipupdate.sh returns 404 errors. > Maybe you should remove 'ewarn' about it from ebuild? A simple workaround would be to change GEOIP_MIRROR and DATABASE variables in geoipupdate.sh like this: GEOIP_MIRROR="https://mailfud.org/geoip-legacy/" DATABASES=" GeoIPv6 GeoIPCity GeoIPCityv6 GeoIP GeoIPASNum GeoIPASNumv6 " Works for me pretty well.
Created attachment 706869 [details] /etc/portage/env/dev-libs/geoip
(In reply to office from comment #29) > Works for me pretty well. works for me too. Just put attached file as /etc/portage/env/dev-libs/geoip
The bug has been closed via the following commit(s): https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=e257f3c300909b01309710582e8f09b3d638f864 commit e257f3c300909b01309710582e8f09b3d638f864 Author: Matt Smith <matt@offtopica.uk> AuthorDate: 2021-05-18 13:55:39 +0000 Commit: Sam James <sam@gentoo.org> CommitDate: 2021-12-08 02:03:45 +0000 dev-libs/geoip: Update geoipupdate.sh to use mirrors The free databases provided by MaxMind are no longer available and so the script doesn't work. Switch to a different mirror. Suggested-by: Alexander Berkes <office@metasoft.at> Closes: https://bugs.gentoo.org/674852 Package-Manager: Portage-3.0.18, Repoman-3.0.3 Signed-off-by: Matt Smith <matt@offtopica.uk> Closes: https://github.com/gentoo/gentoo/pull/20869 Signed-off-by: Sam James <sam@gentoo.org> dev-libs/geoip/files/geoipupdate-r7.sh | 46 +++++++++++++++++++++++++++++++ dev-libs/geoip/geoip-1.6.12-r1.ebuild | 50 ++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+)
It's more complicated than that. mailfud is not a mirror. They provide 'legacy' databases (aka *.dat), by converting the 'new format' that maxminddb provides (*.mmdb). Installing *.dat is pretty useless nowadays. Most tools expect the new format. Typically, the python tools i use.
(In reply to Thomas Capricelli from comment #33) > It's more complicated than that. > > mailfud is not a mirror. They provide 'legacy' databases (aka *.dat), by > converting the 'new format' that maxminddb provides (*.mmdb). > > Installing *.dat is pretty useless nowadays. Most tools expect the new > format. Typically, the python tools i use. In that case, they won't be using this package, they'll use geoipupdate, I would've thought? My understanding is that mmdb is out of scope for this bug. Should people migrate? Yes, but that's not the problem here. This bug is about life support for .dat while people still need it. Anyway, I might be misunderstanding you. If I am, please let me know if you have a better suggestion for how to handle this.
Er.. not sure. I might be mistaken as well. The name 'geoipupdate' or the package name 'dev-libs/geoip' make me thing of something current, not about legacy. The other big difference is that mailfud files are public, while the original/newformat are behind a account creation. paradoxically, it's easier to get legacy files, which are made from a 'protected' source.
