Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 671690 - emerge --sync should ignore .zfs directories
Summary: emerge --sync should ignore .zfs directories
Status: CONFIRMED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Core - Configuration (show other bugs)
Hardware: All Linux
: Normal normal
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 240187
  Show dependency tree
 
Reported: 2018-11-22 13:56 UTC by Erik Ohrnberger
Modified: 2019-01-01 20:58 UTC (History)
1 user (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
emerge --info text file attached (2018-11-24.emerge.info.txt,6.12 KB, text/plain)
2018-11-22 13:56 UTC, Erik Ohrnberger
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Erik Ohrnberger 2018-11-22 13:56:41 UTC
Created attachment 555922 [details]
emerge --info text file attached

If you have /usr/portage mounted as a zfs dataset,
And if zfs set snapdir=visible on that dataset

Then it appears that the rsync tree verification recurses through all the snapshots that are visible in the /usr/portage/.zfs directories, eventually exhausting the storage of that zfs dataset populating the /usr/portage/.tmp-unverified-download-quarantine directory with recursive copies of /usr/portage/.tmp-unverified-download-quarantine/.zfs/snapshot

Temporary solution is to:

    zfs set snapdir=hidden <pool>/<dataset>
    emerge --sync
    zfs set snapdir=visible <pool>/<dataset>

However, couldn't the rsync tree verification be set to ignore any .zfs directories?
Comment 1 Zac Medico gentoo-dev 2018-11-23 22:36:19 UTC
Yes, we can add --exclude=/.zfs to the default excludes, like the others from bug 666554:

https://gitweb.gentoo.org/proj/portage.git/commit/?id=b587fc874ce95064139ba85552e146da957cce9e

We could extend repos.conf to allow explicit excludes, and could also use excludes from sync-rsync-extra-opts.
Comment 2 Erik Ohrnberger 2018-11-24 01:19:17 UTC
Hmm.  Hadn't realized something until I looked at my PORTAGE_RSYNC_OPTS again.

PORTAGE_RSYNC_OPTS="--recursive --links --safe-links --perms --times --omit-dir-times --compress --force --whole-file --delete --stats --human-readable --timeout=180 --exclude=/distfiles --exclude=/local --exclude=/packages --exclude=/.git --exclude=/.zfs"

I suppose I need to change it to: --exclude=.zfs
?
Comment 3 Zac Medico gentoo-dev 2018-11-24 03:09:55 UTC
The --exclude=/.zfs option is correct, however the code seen in https://gitweb.gentoo.org/proj/portage.git/commit/?id=b587fc874ce95064139ba85552e146da957cce9e does not use PORTAGE_RSYNC_OPTS because bug 666554 was triggered by people putting --delete-excluded in PORTAGE_RSYNC_OPTS. We could scrape the --exclude settings from PORTAGE_RSYNC_OPTS, but it's probably better create a separate configuration setting for excludes.

A separate configuration setting for excludes would also be useful for the purposes of bug 657324.

There are actually 2 distinct types of excludes to deal with, though they are sometimes lumped them together:

1) Excludes for remote files that we would want to delete locally if matching files exist (which is why people put --delete-excluded in PORTAGE_RSYNC_OPTS).

2) Excludes for local files that we want to keep, including /distfiles, /local, /lost+found, /packages, and /.zfs.