Summary: | metadata/timestamp*: please clean up and set a standard format | ||
---|---|---|---|
Product: | Gentoo Infrastructure | Reporter: | Michał Górny <mgorny> |
Component: | Other | Assignee: | Gentoo Infrastructure <infra-bugs> |
Status: | CONFIRMED --- | ||
Severity: | normal | CC: | dev-portage |
Priority: | Normal | ||
Version: | unspecified | ||
Hardware: | All | ||
OS: | Linux | ||
See Also: | https://bugs.gentoo.org/show_bug.cgi?id=540784 | ||
Whiteboard: | |||
Package list: | Runtime testing required: | --- | |
Bug Depends on: | |||
Bug Blocks: | 546536 |
Description
Michał Górny
2015-04-14 06:16:57 UTC
@mgorny: Infra inherited the code that does these, it was already in use in 2006, and it's actually WORSE than you think, because the patterns are used similarly in distfiles, releases and experimental. I copied the pattern to snapshots, releases, experimental, with a better format, because I wanted to scan mirrors at one point. I can tell you what some of them are however. metadata/timestamp.x is generated on the mastercvs box, in a 5-min cronjob. metadata/timestamp is generated on the masterrsync, after eclass check & egencache are successful. metadata/timestamp.chk is generated on the masterrsync, directly into the output directory, per cronjob. Based on these, changing or NOT changing, you can tell where a problem is for any mirror: If 'timestamp.x' is static, but the others are NOT, that means no new exports are happening from CVS. If 'timestamp' and 'timestamp.x' remain static, that means egencache or the eclass check are failing, but the rest of the pipeline is fine. If 'timestamp.chk' is static, then somewhere earlier in the pipeline has failed or the mirror is not syncing anymore. Here's some of the stuff that writes them: files/usr/local/bin/mastermirror/datestamp.sh: /bin/date -u '+%s %c' > ${STAGEDIR}/metadata/timestamp.x.tmp files/usr/local/bin/mastermirror/datestamp.sh: /bin/mv ${STAGEDIR}/metadata/timestamp.x.tmp ${STAGEDIR}/metadata/timestamp.x files/usr/local/bin/mastermirror/sync-experimental.sh:/bin/date -u '+%s %c' > ${FINALDIR}/timestamp.x files/usr/local/sbin/update-master-distfiles:date -u '+%s %c %Z' > $MASTER_DISTFILES/timestamp.x files/usr/local/bin/mastermirror/timestamp-rsync.sh: /bin/date -u +'%s %a, %d %b %Y %H:%M:%S %z' >"${TARGET}" where TARGET is .../snapshots-final/.timestamp-snapshots.x files/usr/local/bin/mastermirror/timestamp-releases.sh: /bin/date -u +'%s %a, %d %b %Y %H:%M:%S %z' >"${TARGET}" where TARGET is .../experimental/.timestamp-experimental. where TARGET is .../releases/.timestamp-releases.x files/usr/local/bin/mastermirror/sync-distfiles.sh:/bin/date -u '+%s' > ${DATADIR}/distfiles/timestamp.mirmon files/usr/local/bin/mastermirror/rsync-gen.sh: # timestamp.chk is generated every 5 mins in datestamp.sh script in the FINALDIR # timestamp.x is generated every 5 mins in datestamp.sh script in the STAGEDIR # We want to track the progress of STAGEDIR->FINALDIR so exclude .x file here # (so it isn't deleted) files/usr/local/bin/mastermirror/rsync-gen.sh:date -u > ${STAGEDIR}/metadata/timestamp files/usr/local/bin/mastermirror/rsync-gen.sh:date -R -u > ${STAGEDIR}/metadata/dtd/timestamp.chk files/usr/local/bin/mastermirror/rsync-gen.sh:date -R -u > ${STAGEDIR}/metadata/glsa/timestamp.chk files/usr/local/bin/mastermirror/rsync-gen.sh:date -R -u > ${STAGEDIR}/metadata/news/timestamp.chk files/usr/local/bin/mastermirror/rsync-gen.sh:# timestamp.chk is generated every 5 mins in datestamp.sh script in the FINALDIR files/usr/local/bin/mastermirror/rsync-gen.sh:# timestamp.x is generated every 5 mins in datestamp.sh script in the STAGEDIR files/usr/local/bin/mastermirror/rsync-gen.sh:rsync -Wqa --exclude=/metadata/timestamp.chk --delete ${STAGEDIR}/ ${FINALDIR}/ For moving forward, we DO need a way to track the status of different phases sanely I think, so just having a single timestamp file isn't going to cut it. But we can certainly include a "purpose" line in each timestamp file. Having the timestamp both trivially parsable and human readable is also a huge benefit, so I'd like to propose that we do the (epoch, ISO8601). For migration, wrt to Git, I think we trim it down to a few files, and come up with useful new names. .timestamp.git-commit - the timestamp of the last Git commit. .timestamp.git-export - the timestamp of the last Git export. .timestamp.mirror - the timestamp when it was mirrored. Then we announce some deprecation period to cover that old clients are going to break if we simply remove timestamps. Sounds good to me. |