Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 716370 - emerge --sync and emerge-webrsync fails probably because of timestamp error
Summary: emerge --sync and emerge-webrsync fails probably because of timestamp error
Status: UNCONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core (show other bugs)
Hardware: AMD64 Linux
: Normal normal (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 650144
  Show dependency tree
 
Reported: 2020-04-05 18:50 UTC by Přemysl Šťastný
Modified: 2021-05-21 18:40 UTC (History)
3 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Přemysl Šťastný 2020-04-05 18:50:01 UTC
When I run emerge --sync, I get repeated error:
 * Refreshing keys from keyserver hkps://keys.gentoo.org ...OpenPGP keyring refresh failed:
gpg: refreshing 4 keys from hkps://keys.gentoo.org
gpg: keyserver refresh failed: Server indicated a failure


And when I run emerge-webrsync, I get 

Fetching most recent snapshot ...
 * Latest snapshot date: 20200404
 * 
 * Approximate snapshot timestamp: 1586047500
 *        Current local timestamp: 1586046901
 * 
 * The current local timestamp is possibly identical to the
 * timestamp of the latest snapshot. In order to force sync, use
 * the --revert option or remove the timestamp file located at
 * '/var/db/repos/gentoo/metadata/timestamp.x'.

When I delete /var/db/repos/gentoo/metadata/timestamp.x and run emerge-webrsync, the problem is temporary solved until the sync after sync.

Reproducible: Always




tardis /var/db/repos/gentoo/metadata # emerge --info
Portage 2.3.89 (python 3.6.10-final-0, default/linux/amd64/17.1/hardened, gcc-9.2.0, glibc-2.29-r7, 4.19.97-gentoo x86_64)
=================================================================
System uname: Linux-4.19.97-gentoo-x86_64-Intel-R-_Core-TM-_i7-4510U_CPU_@_2.00GHz-with-gentoo-2.6
KiB Mem:     8059156 total,   1086524 free
KiB Swap:          0 total,         0 free
Timestamp of repository gentoo: Sun, 05 Apr 2020 00:45:01 +0000
Head commit of repository gentoo: b3686ff7aee4ece15744169e0feda1ca0578e019
Timestamp of repository libressl: Fri, 27 Mar 2020 08:23:12 +0000
Head commit of repository libressl: c885727f9d76c38219f564d8c036dfb943f739f9

sh bash 4.4_p23-r1
ld GNU ld (Gentoo 2.33.1 p2) 2.33.1
ccache version 3.7.7 [enabled]
app-shells/bash:          4.4_p23-r1::gentoo
dev-lang/perl:            5.30.1::gentoo
dev-lang/python:          2.7.17-r2::gentoo, 3.6.10-r1::gentoo, 3.7.7-r1::gentoo
dev-util/ccache:          3.7.7-r1::gentoo
dev-util/cmake:           3.16.5::gentoo
sys-apps/baselayout:      2.6-r1::gentoo
sys-apps/openrc:          0.42.1::gentoo
sys-apps/sandbox:         2.13::gentoo
sys-devel/autoconf:       2.13-r1::gentoo, 2.69-r4::gentoo
sys-devel/automake:       1.16.1-r1::gentoo
sys-devel/binutils:       2.33.1-r1::gentoo
sys-devel/gcc:            9.2.0-r2::gentoo
sys-devel/gcc-config:     2.2.1::gentoo
sys-devel/libtool:        2.4.6-r6::gentoo
sys-devel/make:           4.2.1-r4::gentoo
sys-kernel/linux-headers: 5.4::gentoo (virtual/os-headers)
sys-libs/glibc:           2.29-r7::gentoo
Repositories:

gentoo
    location: /var/db/repos/gentoo
    sync-type: rsync
    sync-uri: rsync://rsync.gentoo.org/gentoo-portage
    priority: -1000
    sync-rsync-verify-max-age: 24
    sync-rsync-extra-opts: 
    sync-rsync-verify-jobs: 1
    sync-rsync-verify-metamanifest: yes

crossdev
    location: /var/db/repos/crossdev
    masters: gentoo

libressl
    location: /var/db/repos/libressl
    sync-type: git
    sync-uri: https://github.com/gentoo-mirror/libressl.git
    masters: gentoo

ACCEPT_KEYWORDS="amd64"
ACCEPT_LICENSE="* -@EULA"
CBUILD="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe -march=native -ggdb"
CHOST="x86_64-pc-linux-gnu"
CONFIG_PROTECT="/etc /usr/share/config /usr/share/gnupg/qualified.txt"
CONFIG_PROTECT_MASK="/etc/ca-certificates.conf /etc/dconf /etc/env.d /etc/fonts/fonts.conf /etc/gconf /etc/gentoo-release /etc/revdep-rebuild /etc/sandbox.d /etc/terminfo"
CXXFLAGS="-O2 -pipe -march=native -ggdb"
DISTDIR="/var/cache/distfiles"
EMERGE_DEFAULT_OPTS=" --complete-graph --with-bdeps=y"
ENV_UNSET="DBUS_SESSION_BUS_ADDRESS DISPLAY GOBIN PERL5LIB PERL5OPT PERLPREFIX PERL_CORE PERL_MB_OPT PERL_MM_OPT XAUTHORITY XDG_CACHE_HOME XDG_CONFIG_HOME XDG_DATA_HOME XDG_RUNTIME_DIR"
FCFLAGS="-O2 -pipe -march=native -ggdb"
FEATURES="assume-digests binpkg-docompress binpkg-dostrip binpkg-logs ccache config-protect-if-modified distlocks ebuild-locks fixlafiles ipc-sandbox merge-sync multilib-strict network-sandbox news parallel-fetch pid-sandbox preserve-libs protect-owned qa-unresolved-soname-deps sandbox sfperms strict unknown-features-warn unmerge-logs unmerge-orphans userfetch userpriv usersandbox usersync xattr"
FFLAGS="-O2 -pipe -march=native -ggdb"
GENTOO_MIRRORS="https://mirrors.tuna.tsinghua.edu.cn/gentoo ftp://ftp.fi.muni.cz/pub/linux/gentoo/ http://ftp.fi.muni.cz/pub/linux/gentoo/ rsync://ftp.fi.muni.cz/pub/linux/gentoo/ ftp://mirror.dkm.cz/gentoo/ https://mirror.dkm.cz/gentoo/ http://mirror.dkm.cz/gentoo/ rsync://mirror.dkm.cz/gentoo/ http://gentoo.mirror.web4u.cz/ ftp://gentoo.mirror.web4u.cz/"
LANG="C.utf8"
LDFLAGS="-Wl,-O1 -Wl,--as-needed"
MAKEOPTS="-j4"
PKGDIR="/var/cache/binpkgs"
PORTAGE_CONFIGROOT="/"
PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git"
PORTAGE_TMPDIR="/var/tmp"
USE="X acl alsa alsa-plugin amd64 apng bash-completion bindist bluetooth btrfs bzip2 cairo cgroups crypt cups dbus device-mapper djvu egl elogind equalizer firmware gtk gui hardened iconv icu ipv6 jpeg kde libressl libtirpc lua luajit multilib ncurses nls nptl ogg opengl openmp pam pcre pdf pie plasma png postproc printsupport pulseaudio qt5 readline script seccomp secure-delete split-usr sqlite ssl ssp svg tmux tremor unicode vim vim-syntax vorbis wayland xattr xtpax xwayland zlib" ABI_X86="64" ADA_TARGET="gnat_2018" ALSA_CARDS="ali5451 als4000 atiixp atiixp-modem bt87x ca0106 cmipci emu10k1x ens1370 ens1371 es1938 es1968 fm801 hda-intel intel8x0 intel8x0m maestro3 trident usb-audio via82xx via82xx-modem ymfpci" APACHE2_MODULES="authn_core authz_core socache_shmcb unixd actions alias auth_basic authn_alias authn_anon authn_dbm authn_default authn_file authz_dbm authz_default authz_groupfile authz_host authz_owner authz_user autoindex cache cgi cgid dav dav_fs dav_lock deflate dir disk_cache env expires ext_filter file_cache filter headers include info log_config logio mem_cache mime mime_magic negotiation rewrite setenvif speling status unique_id userdir usertrack vhost_alias" CALLIGRA_FEATURES="karbon sheets words" COLLECTD_PLUGINS="df interface irq load memory rrdtool swap syslog" CPU_FLAGS_X86="mmx mmxext sse sse2" CURL_SSL="libressl" ELIBC="glibc" GPSD_PROTOCOLS="ashtech aivdm earthmate evermore fv18 garmin garmintxt gpsclock greis isync itrax mtk3301 nmea ntrip navcom oceanserver oldstyle oncore rtcm104v2 rtcm104v3 sirf skytraq superstar2 timing tsip tripmate tnt ublox ubx" INPUT_DEVICES="mouse keyboard synaptics" KERNEL="linux" LCD_DEVICES="bayrad cfontz cfontz633 glk hd44780 lb216 lcdm001 mtxorb ncurses text" LIBREOFFICE_EXTENSIONS="presenter-console presenter-minimizer" OFFICE_IMPLEMENTATION="libreoffice" PHP_TARGETS="php7-2" POSTGRES_TARGETS="postgres10 postgres11" PYTHON_SINGLE_TARGET="python3_6" PYTHON_TARGETS="python2_7 python3_6" RUBY_TARGETS="ruby24 ruby25" USERLAND="GNU" VIDEO_CARDS="intel" XTABLES_ADDONS="quota2 psd pknock lscan length2 ipv4options ipset ipp2p iface geoip fuzzy condition tee tarpit sysrq steal rawnat logmark ipmark dhcpmac delude chaos account"
Unset:  CC, CPPFLAGS, CTARGET, CXX, INSTALL_MASK, LC_ALL, LINGUAS, PORTAGE_BINHOST, PORTAGE_BUNZIP2_COMMAND, PORTAGE_COMPRESS, PORTAGE_COMPRESS_FLAGS, PORTAGE_RSYNC_EXTRA_OPTS
Comment 1 Přemysl Šťastný 2020-05-25 13:53:59 UTC
It started working for me now. But I am not sure, if the problem is really solved.
Comment 2 Zac Medico gentoo-dev 2020-05-25 18:51:18 UTC
The the emerge --synce key refresh issue and the emerge-webrsync timestamp thing are entirely separate issues.

It's not recommended to call emerge-webrsync directly anymore, since it does not refresh keys for you automatically.

The recommended way to use emerge-webrsync is to call emerge --sync with a setting like this in repos.conf:

> sync-type = webrsync
> sync-webrsync-verify-signature = yes

The "sync-webrsync-verify-signature = yes" setting is default since bug 689506.
Comment 3 contactopublico57 2021-04-21 19:41:49 UTC
Follow up to this issue.

Recently, and for no apparent reason sudo emerge-webrsync has failed to download and populate the package tree and gives the following message: 

rsync warning: some files vanished before they could be transferred (code 24) at main.c(1330) [sender=3.2.3]
Cleaning up ...


!!! /etc/portage/make.profile is not a symlink and will probably prevent most merges.
!!! It should point into a profile within /usr/portage/profiles/
!!! (You can safely ignore this message when syncing. It's harmless.)


!!! Your current profile is invalid. If you have just changed your profile
!!! configuration, you should revert back to the previous configuration.
!!! Allowed actions are limited to --help, --info, --search, --sync, and
!!! --version.

No profiles and many builds showing zero contents.

sudo emerge --sync works but is slow:

Number of files: 146,083 (reg: 119,474, dir: 26,609)
Number of created files: 145,559 (reg: 119,474, dir: 26,085)
Number of deleted files: 6,663 (reg: 3,977, dir: 2,686)
Number of regular files transferred: 119,474
Total file size: 218.75M bytes
Total transferred file size: 218.75M bytes
Literal data: 218.75M bytes
Matched data: 0 bytes
File list size: 3.61M
File list generation time: 0.003 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 2.41M
Total bytes received: 107.63M

sent 2.41M bytes  received 107.63M bytes  29.86K bytes/sec
total size is 218.75M  speedup is 1.99

Have tried substitute mirrors, no change in problem.

Noticed an updated rsync came out recently but that did not correct the failure to populate the package build tree in /usr/portage. 

Implementing the criteria in Comment 2 of this bug report did not resolve the issue?

Please comment.
Comment 4 Zac Medico gentoo-dev 2021-04-21 19:59:13 UTC
(In reply to contactopublico57 from comment #3)
> Follow up to this issue.
> 
> Recently, and for no apparent reason sudo emerge-webrsync has failed to
> download and populate the package tree and gives the following message: 

Direct call to emerge-webrsync is deprecated, so please try emerge --sync with a setting like this in /etc/portage/repos.conf:

[gentoo]
sync-type = webrsync
Comment 5 Andrew Aladjev 2021-05-21 10:42:37 UTC
Hello everyone, I've lost yesterday evening debugging this issue and found that hkps has been blocked using DPI in my country (Belarus).

I've tried to encapsulate (mirror) hkps traffic using some servers outside Belarus and received same issue (KS_GET -- ... Server indicated a failure). I am 100% sure that hkps traffic has been blocked by state firewall. It may be similar for DPI to another blocked protocol.

Please don't waste your time debugging it if you are living in any country under the influence of post USSR or China. Low quality developers (maybe from China) implemented low quality DPI, it can ruin everything and nobody will care. Bad state governments just share it between each other and multiply bugs.

You have to use vpn today on any machine because of hkps protocol. Please be aware that vpn is not enough: you have to obfuscate vpn protocol, secure it once again and simulate regular https traffic. The only one alternative is to move physically out of countries like Belarus.

Thank you.
Comment 6 contactopublico57 2021-05-21 17:46:23 UTC
The emerge-webrsync was resolved with a configuration change.

As to the Belarus connectivity issue reported by
Andrew Aladjev 2021-05-21 10:42:37 UTC 

Go satellite
Comment 7 Andrew Aladjev 2021-05-21 18:40:44 UTC
> Go satellite

All direct satellite internet access (without bypassing through state firewall) is banned completely in Belarus. Of course we can use satellite internet as backup access: it is possible to go to the village or some forest and temporaly enable satellite transmitter. But nobody is using it permanently, it is possible to be punished.

We are using stunnel + obfs4proxy + openvpn and masking stunnel traffic as regular https traffic. This scenario comes directly from china user experience. Today we are not hiding this https traffic behind amazon/azure/etc corporate networks (known as tor meek transport) like china users, but we have already bough such frontends. We are ready for complete china firewall.

I didn't want to use vpn tunnel just on home build server, engaged in procrastination. But state firewall broke hkps protocol this month, so i had to switch to vpn anyway. This state firewall behaviour is actually a bug, i think nobody wanted to broke hkps. Please don't repeat my mistake, if you have received "gpg: keyserver refresh failed: Server indicated a failure" in Belarus - you have to replicate china user vpn installation. Don't waste time on debugging.

The organization responsible for partial China firewall installation and management in Belarus is https://belgie.by. Everyone are free to report hkps bug into this organization, i won't do it, didn't respect them.

Thank you for attention.