Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 546520 - metadata/timestamp*: please clean up and set a standard format
Summary: metadata/timestamp*: please clean up and set a standard format
Status: CONFIRMED
Alias: None
Product: Gentoo Infrastructure
Classification: Unclassified
Component: Other (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Gentoo Infrastructure
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 546536
  Show dependency tree
 
Reported: 2015-04-14 06:16 UTC by Michał Górny
Modified: 2015-05-26 06:56 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2015-04-14 06:16:57 UTC
Right now, my Gentoo checkout has three timestamp files:

==> timestamp <==
Tue Apr 14 04:07:05 UTC 2015

==> timestamp.chk <==
Tue, 14 Apr 2015 04:15:01 +0000

==> timestamp.x <==
1428984301 Tue 14 Apr 2015 04:05:01 AM UTC

Each of them is using a different format, and neither seems to be really safe to rely on. For example, the squashfs snapshot generation script is just passing them to 'date --date=...' and hoping it'd be able to figure out what it is.

Can we please:

1. get the meaning of different timestamp files clear? I mean, we have three files which are supposed to represent something, differing by few minutes but user seeing them has no clue what this means. As I see it, having one final timestamp would be much more clear.

2. get the timestamp in some sane, machine-readable format. Like ISO8601.


Now some notes:

1. timestamp.x seems to be used by emerge-delta-webrsync and Portage's 'you haven't synced for ages'. Why? Probably someone added it specifically to have the machine-parseable unix timestamp in front of it.

2. timestamp.chk seems to be used by rsync syncing in Portage & pkgcore. And this one has the strptime() format hardcoded... It's also used by squashfs generation on Infra but it's passed to --date=, so any sane format would work.

3. I don't see any specific mention of timestamp file but it's hard to grep.

So as you can see, even a single project in Gentoo can't agree on using one file all the way. I don't know if those files are used in the proprietary infra scripts.


That said, it seems the most sane way forward would be to change metadata/timestamp to a sane format, update all the code to use it and at some point wipe the others.
Comment 1 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2015-05-26 05:03:59 UTC
@mgorny:
Infra inherited the code that does these, it was already in use in 2006, and it's actually WORSE than you think, because the patterns are used similarly in distfiles, releases and experimental. I copied the pattern to snapshots, releases, experimental, with a better format, because I wanted to scan mirrors at one point.

I can tell you what some of them are however.
metadata/timestamp.x is generated on the mastercvs box, in a 5-min cronjob.

metadata/timestamp is generated on the masterrsync, after eclass check & egencache are successful.

metadata/timestamp.chk is generated on the masterrsync, directly into the output directory, per cronjob.

Based on these, changing or NOT changing, you can tell where a problem is for any mirror:
If 'timestamp.x' is static, but the others are NOT, that means no new exports are happening from CVS.
If 'timestamp' and 'timestamp.x' remain static, that means egencache or the eclass check are failing, but the rest of the pipeline is fine.
If 'timestamp.chk' is static, then somewhere earlier in the pipeline has failed or the mirror is not syncing anymore.

Here's some of the stuff that writes them:
files/usr/local/bin/mastermirror/datestamp.sh:  /bin/date -u '+%s %c' > ${STAGEDIR}/metadata/timestamp.x.tmp
files/usr/local/bin/mastermirror/datestamp.sh:  /bin/mv ${STAGEDIR}/metadata/timestamp.x.tmp ${STAGEDIR}/metadata/timestamp.x

files/usr/local/bin/mastermirror/sync-experimental.sh:/bin/date -u '+%s %c' > ${FINALDIR}/timestamp.x
files/usr/local/sbin/update-master-distfiles:date -u '+%s %c %Z' > $MASTER_DISTFILES/timestamp.x 

files/usr/local/bin/mastermirror/timestamp-rsync.sh:
 /bin/date -u +'%s %a, %d %b %Y %H:%M:%S %z' >"${TARGET}"
  where TARGET is .../snapshots-final/.timestamp-snapshots.x

files/usr/local/bin/mastermirror/timestamp-releases.sh:
 /bin/date -u +'%s %a, %d %b %Y %H:%M:%S %z' >"${TARGET}"
  where TARGET is .../experimental/.timestamp-experimental.
  where TARGET is .../releases/.timestamp-releases.x

files/usr/local/bin/mastermirror/sync-distfiles.sh:/bin/date -u '+%s' > ${DATADIR}/distfiles/timestamp.mirmon


files/usr/local/bin/mastermirror/rsync-gen.sh:
# timestamp.chk is generated every 5 mins in datestamp.sh script in the FINALDIR
# timestamp.x is generated every 5 mins in datestamp.sh script in the STAGEDIR
# We want to track the progress of STAGEDIR->FINALDIR so exclude .x file here
# (so it isn't deleted)



files/usr/local/bin/mastermirror/rsync-gen.sh:date -u > ${STAGEDIR}/metadata/timestamp
files/usr/local/bin/mastermirror/rsync-gen.sh:date -R -u > ${STAGEDIR}/metadata/dtd/timestamp.chk
files/usr/local/bin/mastermirror/rsync-gen.sh:date -R -u > ${STAGEDIR}/metadata/glsa/timestamp.chk
files/usr/local/bin/mastermirror/rsync-gen.sh:date -R -u > ${STAGEDIR}/metadata/news/timestamp.chk
files/usr/local/bin/mastermirror/rsync-gen.sh:# timestamp.chk is generated every 5 mins in datestamp.sh script in the FINALDIR
files/usr/local/bin/mastermirror/rsync-gen.sh:# timestamp.x is generated every 5 mins in datestamp.sh script in the STAGEDIR
files/usr/local/bin/mastermirror/rsync-gen.sh:rsync -Wqa --exclude=/metadata/timestamp.chk --delete ${STAGEDIR}/ ${FINALDIR}/
Comment 2 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2015-05-26 05:08:35 UTC
For moving forward, we DO need a way to track the status of different phases sanely I think, so just having a single timestamp file isn't going to cut it. But we can certainly include a "purpose" line in each timestamp file.

Having the timestamp both trivially parsable and human readable is also a huge benefit, so I'd like to propose that we do the (epoch, ISO8601).

For migration, wrt to Git, I think we trim it down to a few files, and come up with useful new names.

.timestamp.git-commit - the timestamp of the last Git commit.
.timestamp.git-export - the timestamp of the last Git export.
.timestamp.mirror - the timestamp when it was mirrored.

Then we announce some deprecation period to cover that old clients are going to break if we simply remove timestamps.
Comment 3 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2015-05-26 06:56:32 UTC
Sounds good to me.