Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 539418 - sys-apps/portage: support sync module option to 'git reset' before pulling
Summary: sys-apps/portage: support sync module option to 'git reset' before pulling
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Unclassified (show other bugs)
Hardware: All Linux
: Normal normal with 5 votes (vote)
Assignee: Portage team
URL:
Whiteboard:
Keywords:
Depends on: 883437 887025
Blocks:
  Show dependency tree
 
Reported: 2015-02-09 08:14 UTC by Michał Górny
Modified: 2023-12-14 08:56 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2015-02-09 08:14:59 UTC
Forum users have noticed that git -- in contrast to rsync -- does not clean up local changes from the repository before syncing. While this has its advantages, I think it could be useful to provide an optional rsync-like behavior.

I think we can possibly have three levels of cleanup:

1. git reset --hard -- to discard changes, removals and files added via 'git add',

2. git clean -dfx -- to discard locally added files,

3. git reset --hard origin/master -- to discard local commits as well.

Now, do we want all 3 options? Or maybe just a boolean that enables 1.+2. since that I would expect to be the common case. How to name it? :)
Comment 1 nE0sIghT 2015-03-08 22:23:05 UTC
Does not option 2 will destroy distfiles content? I think it must be added to .gitignore.
Comment 2 Michał Górny archtester Gentoo Infrastructure gentoo-dev Security 2015-03-08 22:59:15 UTC
Why would I care about people crazy enough to put distfiles in a repository?
Comment 3 Zac Medico gentoo-dev 2015-03-08 23:36:15 UTC
(In reply to Michał Górny from comment #2)
> Why would I care about people crazy enough to put distfiles in a repository?

Yeah, since the git sync module has nothing to do with the default rsync sync module, within this context we can consider /usr/portage/disfiles as an unsupported legacy configuration.
Comment 4 nE0sIghT 2015-03-08 23:41:36 UTC
Ok then default value for DISTDIR must be changed to something outside of /usr/portage
Comment 5 Zac Medico gentoo-dev 2015-03-08 23:49:39 UTC
(In reply to nE0sIghT from comment #4)
> Ok then default value for DISTDIR must be changed to something outside of
> /usr/portage

For bug 378603, the default DISTDIR will change along with a few others.

I don't think we should consider that a blocker for this 'git reset' feature, as long as it's not enabled in the default configuration (and git is not currently the default configuration).
Comment 6 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-12-14 08:55:48 UTC
This was done a little while ago.

commit 7a336e85052530c62acf25b5df10e2d02a17e779
Author: John Helmert III <ajak@gentoo.org>
Date:   Sat Oct 29 21:33:44 2022 -0500

    GitSync.update: clobber changes to git repo as rsync does

    Check `git diff --quiet` and if tree is dirty, do a hard reset to the
    upstream HEAD instead of simply merging. Thanks to a limitation in
    git, also `git reset --hard` if the merge fails.

    Closes: https://github.com/gentoo/portage/pull/931
    Signed-off-by: John Helmert III <ajak@gentoo.org>
    Signed-off-by: Sam James <sam@gentoo.org>

commit da8e7d193563a2b0c88a523083b2a3466ff4526e
Author: Florian Schmaus <flow@gentoo.org>
Date:   Wed Dec 14 17:09:50 2022 +0100

    sync: git: only perform shallow updates if the repository is shallow

    With the new default of shallow cloning/updating git repositories,
    added with f2207e41792d ("GitSync.update: default to "--depth 1" (bug
    824782 comment 17)"), existing *non-shallow* repositories would become
    shallow after a "emerge --sync".

    This adds a check to see if the repository is already shallow, and
    only in this case performs a shallow clone.

    Thanks to Douglas Freed for pointing out that this should consider
    whether or not the user explicitly set sync-depth.

    Thanks-to: Douglas Freed <dwfreed@mtu.edu>
    Bug: https://bugs.gentoo.org/824782
    Bug: https://bugs.gentoo.org/887025
    Closes: https://github.com/gentoo/portage/pull/960
    Signed-off-by: Sam James <sam@gentoo.org>

commit ac405c994a0cce67e11c1bc7fb37fc02bba627ae
Author: Sam James <sam@gentoo.org>
Date:   Sat Dec 17 05:19:32 2022 +0000

    sync: git: allow truncating history if repository is marked volatile

    See 54a08c9f71dc37851fdb0364576581ca19df6580 for an explanation
    of the context here.

    If the repository is marked as volatile, it's assumed that
    Portage has complete control of it, and for Portage's purposes,
    there's no use in having extended history (in fact, if anything,
    it's deterimental, as it makes a sync more likely to fail due
    to the large disk space requirements over time).

    Bug: https://bugs.gentoo.org/824782
    Bug: https://bugs.gentoo.org/887025
    Signed-off-by: Sam James <sam@gentoo.org>

commit 1696966564510e8359e3e29681dc1326c37c6e0b
Author: John Helmert III <ajak@gentoo.org>
Date:   Wed Nov 2 21:48:59 2022 -0500

    sync: git: `git clean` the repo before updating

    With a shallow clone, git may leave orphaned files if the remote
    moves from state X -> X-1. If on pulling again, state X is returned,
    git will then abort to avoid clobbering untracked files.

    Bug: https://bugs.gentoo.org/887025
    Signed-off-by: John Helmert III <ajak@gentoo.org>
    Signed-off-by: Sam James <sam@gentoo.org>

commit 92f8d27e034558edd749a08e5d61d579dc7172a8
Author: Sam James <sam@gentoo.org>
Date:   Sat Dec 17 05:51:02 2022 +0000

    sync: git: only clobber changes if repository is marked as non-volatile

    The fundamental issue with regard to Portage handling sync-type=git
    repositories is follows: whenever Portage is syncing something, our
    view was that we should prioritise successful completion of syncs,
    especially given it may be performed unattended.

    The rationale for this is that most people who use sync-type=git are doing so
    because they want quicker sync (to complete), a more reliable one
    (no Manifest race condition issues), and to get changes from upstream faster.

    Recently, Portage started doing two things:
    1. Clobbering changes to repositories with sync-type=git

    If git is pulling from a CDN, Portage may on one sync receive state X, but
    on a subsequent sync receive state X-1. This can cause an odd state
    where git wants to resolve conflicts and manual intervention is required,
    hence needing git reset.

    This situation was often worse with sync-depth=1 and would lead
    to orphaned files hence needing git clean.

    2. Defaulted to sync-depth=clone-depth=1.

    The motivation here was because of disk space growing unbounded
    otherwise. It makes sense when considered within the school of thought / motivation
    listed above (prioritising a successful sync and then making an optimisation
    based on how we achieve that).

    Further, there was additional motivation to avoid disk usage growing
    unbounded (which additionally could lead to a sync failing at some point).

    --

    To allow users to opt-out of these destructive changes, we now
    have a repository setting called 'volatile'.

    * If volatile=yes, the repository is assumed to be user-modifiable
      and Portage will NOT prioritise successful sync over preserving
      repository state.

    * If volatile=no, the repository is assumed to only be modified
      by Portage, and Portage will try its best to ensure syncs continue
      to work.

    This is a followup to 7a336e85052530c62acf25b5df10e2d02a17e779
    and 1b23d15e4c71d95c6c2520dcb85f8bf90ddf430c.

    Bug: https://bugs.gentoo.org/887025
    See: 7a336e85052530c62acf25b5df10e2d02a17e779
    See: 1b23d15e4c71d95c6c2520dcb85f8bf90ddf430c
    See: https://github.com/gentoo/portage/pull/801
    See: https://github.com/gentoo/portage/pull/931
    Closes: https://github.com/gentoo/portage/pull/939
    Closes: https://github.com/gentoo/portage/pull/960
    Signed-off-by: Sam James <sam@gentoo.org>

commit 16246ea32371ff70c8941756af75ddefb8c97f4e
Author: Florian Schmaus <flow@gentoo.org>
Date:   Sat Dec 31 11:29:02 2022 +0100

    git.py: fix "is-shallow-repository" check by stripping \n

    The output of git commands has a trailing newline character which we
    usually strip.

    Fixes: da8e7d193563a2b0c88a523083b2a3466ff4526e ("sync: git: only perform shallow updates if the repository is shallow")
    Signed-off-by: Florian Schmaus <flow@gentoo.org>
    Closes: https://github.com/gentoo/portage/pull/968
    Signed-off-by: Sam James <sam@gentoo.org>

commit f29cc2578b1c97164c1411b14a2dd8195b14ab1e
Author: Florian Schmaus <flow@gentoo.org>
Date:   Wed Feb 22 10:07:31 2023 +0100

    sync: git: fix code comment regarding clobbering the repository

    The logic the code comment stated was simply inverted, and so was the
    NEWS item. The actual implementation is correct.

    Signed-off-by: Florian Schmaus <flow@gentoo.org>
    Closes: https://github.com/gentoo/portage/pull/992
    Signed-off-by: Sam James <sam@gentoo.org>

commit 31a1cbea211633ddb162124ea42d008015f4ab5f
Author: Florian Schmaus <flow@gentoo.org>
Date:   Wed Feb 22 15:11:17 2023 +0100

    sync: git: report error if volatile repository

    Previously portage would not report a merge_cmd error if the
    repository was volatile. This could potentially be the cause of
    bug #895526, however if it unclear, at the time of writing this, if
    the repository in the bug is a volatile one.

    Bug: https://bugs.gentoo.org/895526
    Signed-off-by: Florian Schmaus <flow@gentoo.org>
    Closes: https://github.com/gentoo/portage/pull/993
    Signed-off-by: Sam James <sam@gentoo.org>