|
Description
Jeremy Wohl
2003-01-16 23:43:32 UTC
Created attachment 7390 [details]
Python script
Created attachment 7391 [details]
Config for said script
if its in python then it can easily have GUI extensions added to it yes ? there are many packages out there for this ... for instance, wxPython python-newt would be nice ;) Can we get a GPL copyright header on it please? Nicholas: Copyright added. Brandon mentioned you had diffs for the regexps? SpanKY: GUI extensions aren't on my list (I run headless/armless servers), but feel free to patch. :) Thomas: OK, I'll look at newt. This is a really nice utility! I love the backup and the ease of use. I did find a bug, though: If you press 'h', it dies. I just changed the do_help() call to self.do_help() and it now works. I also changed the command keystrokes, since I like these better: q quit, h help, n skip, m merge, d delete new, u use update Can it also be coded to support RCS if available (instead of just saving in .1, .2 etc)? That way, it uses an industry standard which is also nicely out of the way and efficient with space. I imagine the only things it needs to do for this are: # break possible lock rcs -l -M <file> # check in old version. This won't do anything if there are no changes ci -l <file> # overwrite with new file # check in new version ci -l <file> Not volunteering, mind you, too busy right now :) Created attachment 8103 [details, diff]
Improve handling of merges; tweak command chars.
I didn't like how merges overwrote the new config file, so I decided to change
this. The result is patch that should be applied to the python script in the
first attachement. It contains the following changes:
When a merge happens, the merged content is placed into a file named
._mrg####_name (i.e. "cfg" is turned into "mrg" in the new name). This merged
file is then used as the new file for the normal commands, so the utility
continues to work as before in that regard. However, there is now a "t
toggle-merge" command that allows the user to bounce back and forth between the
unchanged new file and the merged file. I also added a "l look" command to
show the user the diffs between the _new_ and the merged file (as opposed to
the current and the merged file, which are displayed by default).
I also changed the command keystrokes in order to make almost all the
non-dangerous keys be on the right side away from the only dangerous command,
"d delete". The keystrokes in my patch are:
u -- update current config with new config and continue
d -- delete new config and continue
n -- skip to next config, leave all intact
m -- interactively merge current and new configs
l -- look at diff between _new_ and merged configs (not current and merged)
t -- toggle new config between merged and pre-merged state
h -- this screen
q -- quit
Finally, the do_help() bugfix I mentioned in an earlier comment is included.
Opinions? Tests? The original vs. Wayne's updates? I'm looking to include this in 47-r3 Gimme comments... Doesn't seem like many are testing this. Created attachment 8364 [details, diff]
Updated version of my patch that fixes 'l' (look).
One bug that crept into my changes as I tweaked them is that the 'l' (look)
command is now diffing the wrong files. It is supposed to be diffing new
against merged, and it now diffs current against merged. If you've already
applied my patch, just tweak line 92 of dispatch-conf to change 'current'
to 'new'. For all others, I'm supplying a fixed version of my patch.
Nicholas: I'll do the RCS backup today. Wayne: Great you did rediff-against-original. Although there are lots of keys now. Should be a way to reduce. re keys: Looks like folks don't like my selection. :) I tried to pick ones that were well away from each other, thus some contortion of language was required. In any case, that waits on the final behavior. Re: RCS -- I've been thinking about this a lot since last night, spurred on by the bind update I just did. The changes to the /etc/bind/named.conf file were very hard because there was no way to see what changed in the original file and apply those changes to my massively different version. I'm thinking that we should use a branch in the RCS file to keep track of the original file's contents, and thus allow us to merge changes into modified config files. I'm imagining something that works like this: When a ._cfg* file is discovered, the user's current config file is immediately checked into RCS on the trunk, possibly creating the RCS file for the first time (the ci will not create a new revision if the user's file is unchanged). If the RCS file is new, we behave as we do now. If, however, there is an old 1.1.1 branch version, we do a merge of the 1.1.1 branch with the new ._cfg* version and the trunk (which we justed updated), generating a ._mrg* file with our suggested merge of what changed in the packaged version. If there are no changes in the packaged version, we delete the update files and we're done. At the end of the user-update session, we check in the ._cfg*/._mrg* version on the 1.1.1 branch of the RCS file, thus preserving the last-packaged version for the next update (we do this even if the user chooses to delete the update and keep what is there now). Since the auto-generated merge file can contain conflicts, we either auto- edit the file if there are conflicts, or we add an edit command to let the user tweak the merge file. I'm in favor of the latter, since it would have made my bind update easier (I have two "view"s in my named.conf file, and I needed to edit the "pri/127" change into both views). It would also be nice to have a way to display a diff of what changed between the last 1.1.1 branch version and the ._cfg* file, perhaps attached to the 'p' key (for package diff). Finally, I was wondering if the RCS support should be optional or not? Or should we just require rcs to be installed to use dispatch-conf? Just to be clear, my text on updating the 1.1.1 branch should not have mentioned the ._mrg* file. It should have said: At the end of the user-update session, we check in the ._cfg* version on the 1.1.1 branch of the RCS file I've got 47-r3 coming out shortly... Are you integrating Wayne's patch? It doesn't apply cleanly. Chuck 3 is rejected from the cvs 1.13 I grabbed. I'll include it without unless you get back to me fairly quickly . Created attachment 8575 [details, diff]
Added RCS archiving to my previous changes.
I've implemented the RCS archiving I mentioned in my previous comments (rcs
processing is not optional in this patch). As mentioned above, this will
save off the users config files on the trunk and the default config files on
the 1.1.1 branch, allowing for simpler merges of updates (e.g. my much-modified
make.conf file now updates very easily). This script has only been lightly
tested so far, so please let me know if you discover any bugs.
I added an 'e' (edit) command so that it is easy to edit the result of the
rcsmerge when there are conflicts.
Note that I also moved the 'd' (delete) command to 'z' (zap) just to keep
it away from the 'e' key.
The return status of the RCS commands is not currently checked for failure.
My actual implementation differs slightly from the algorithm described in my
prior comments. I have the code go ahead and check in both the user's current
config and the new system config at the start of the processing. This will
generate a ._mrg* file for any configs that have changes, and the script uses
the presence of this file to avoid duplicating the rcs processing if the user
quits and restarts.
We should write a script that saves off a config file before it is edited.
If the unchanged config file is put on the branch from the start, the first
merge will be assisted. If the user waits to let the first update create
the branch, only the second merge will be assisted.
An option could easily be added to auto-update files where the trunk and the
branch are identical (i.e. where the user hasn't modified the config file).
I used the following simple shell script to transform my .1/.2/etc. files
into rcs files. You may wish to archive these files somewhere before running
this, just to be safe.
#!/bin/sh
cd /etc/config-archive || exit
for fn in `find . -type f | sed -n -e '/\.[0-9][0-9]*$/p'`; do
bn=`echo $fn | sed 's/\.[0-9][0-9]*$//'`
echo "#$fn# to #$bn#"
mv $fn $bn
if [ -f "$bn,v" ]; then
rcs -M -l $bn
fi
ci -t-"Archived config file." -m"dispatch-conf update." $bn
done
Created attachment 8589 [details, diff]
Auto-merge unmodified and force branch creation.
This patch adds support for a "replace-unmodified" option that causes
dispatch-conf to use the newly-arrived config file if it is unmodified
from the distributed version. Add these lines to your dispatch-conf.conf
file:
# Automerge files that the user hasn't modified
# (yes or no)
replace-unmodified=yes
Also included is a bug-fix that ensures that we create the branch on the
first commit by passing the -f option to ci (but only when creating the
branch).
Created attachment 8593 [details]
New script to archive a file into the /etc/config-archive dir.
This new script, archive-conf, can be used prior to editing a config file
to save off a copy of the current version into the rcs archive used by
dispatch-conf. The script checks to see if the specified file(s) match
the current package version's MD5, and if so it also stores the file onto
the branch. For example:
cd /etc/init.d
archive-conf bootmisc distccd fam halt.sh localmount modules named net.eth0
This created the branch version of all these config files (since they are
unmodified on my system), and thus the next emerge that changes one of
these scripts will auto-merge (because I have replace-unmodified set).
Created attachment 8602 [details, diff]
My latest patch for dispatch-conf.
Fixed a bug where the script might try to unlink the merge file when it didn't
exist.
I hope someone is finding these patches useful...
CVS 1.13 is in portage-2.0.47-r3. Jeremy: Are you going to merge Wayne's patches? Wayne: Can you make your changes modular/selectable? Nicholas: I'm in the middle of moving, and despite some progress, have been unable to finish up. If someone else wants to take ownership, that would be fine. However, it seems to me this script isn't ready for prime time. There are some good suggestions out there to play with (i.e. I don't think they're quite right for the average user -- we're only getting suggestions/testing here from advanced folks), and features on my TODO list to at least bring it par with etc-update. I can look at things again, middle of next week -- which means it doesn't go in -r3. Let me know. On second thought, adding CVS 1.13 alone makes sense -- we can get wider feedback before integrating these thornier features. Thoughts? (I may be making hay over a minor side utility, but everything has its elegance. :) Created attachment 8624 [details]
Wayne's latest script (not a patch)
The patch for this new version was almost as big as the script, so I'm just
attaching the script this time. This version is based on Jeremy's CVS 1.13
version, and now lets you choose to use rcs or not based on a new config
setting:
# Use rcs for storing files in the archive directory?
# (yes or no)
use-rcs=no
Both versions have support for merging in the user's changes based on
the diff3 merging of the user's config file, the last-installed packaged
version, and the new packaged version.
Do we want an option to control the running of this rcsmerge/diff3?
Created attachment 8625 [details]
Wayne's dispatch-conf.conf
Here's my config file that has the new config items included in it.
Concerning what version to release in portage 2.0.47-r3: I'd recommend at least including the ._mrg* enhancements since they make it safer -- you don't lose the new config file when merging, and can thus toggle it away and re-merge if you mess up. Also, what do people think about the command bindings? If there are going to be changes from the CVS 1.13 version, I'd like to see the bindings be updated sooner rather than later. For instance, if we like the 'z' (zap) binding of my latest version better than the 'd' (delete) binding (because it will be farther away from the 'e' (edit) keystroke, then I could generate a version that combines the 1.13 CVS version with my 8364 patch and the 'z' binding change. Created attachment 8633 [details]
Wayne's latest dispatch-conf script (not a patch)
Fixed the auto-updating of files that the user hasn't changed (the new version
was being discarded instead of installed).
I've done some more work on dispatch-conf and archive-conf. I've separated the
code that the two have in common into a library (dispatch_conf.py) and installed
it in /usr/lib/python2.2/site-packages. Rather than post 4 new attachements,
I'll just point you at an index page that has links to the 4 files:
http://www.blorf.net/dispatch-conf/
Changes in these scripts (besides the library module changes):
I've changed the way the non-rcs files are named. The distibuted config file
for "foo" is now saved as foo.dist so that the most recent user config file can
always be named "foo". Older user config files now have increasing .# suffixes
(like the log files in /var/log) with a maximum of .9 (10 old user configs) per
file. I'm currently using this archive format since rcs has started to seem
like it is really overkill for this situation.
I also fixed the permissions and owner/group of merged files so that they match
the ._cfg* file.
I had the editor hard-wired to "vim" in my last release, so I changed it to
access the environment variable "EDITOR" like it should.
Finally, all the just-installed changes are written out in diff format into
the file /tmp/dispatch-conf.changes (with one old version saved to that name
with .old appended). This lets a user very easily review what just got
installed (which is particularly handy if you have all the auto-install
options turned on).
I just ran across this submission, and was wondering if anyone following this one has seen the patch I posted to http://bugs.gentoo.org/show_bug.cgi?id=4709, and whether that kind of automated merge functionality would be acceptable in Portage? I find it to be exceptionally convenient both for config files I don't mess with as well as with those I modify. Having Portage save config files it installs for 3-way merges is also very helpful if automerging is not desired, and also in the case where merge conflicts occur. I've been using dispatch-conf instead of etc-update ever since it became
available. I finally got around looking closer at the code after hitting a
merge problem yet again. I haven't looked at Wayne's patches yet so my
concerns may have already been addressed, but as it stands now, this utility
has two grave shortcomings:
1) creation of a merged file uses a very predictable name which is susceptable
to symlink attacks.
2) merging doesn't work when /tmp is on a separate filesystem from /etc.
The first problem may be addressed by changing the line:
merged = '/tmp/dispatch-conf.merged.%i' % (os.getpid (),)
to something like:
MKTEMP = 'mktemp /tmp/dispatch-conf.merged.%i.XXXXXX'
merged = commands.getoutput (MKTEMP % (os.getpid ()))
Hopefully this part will be easier when we switch to python2.3 which has
tempfile.mkstemp() function (I'm not sure of the specifics though).
The second problem needs to be solved by doing copy/unlink instead of using
rename(). This, again, will be easier when python2.3 comes around which has
shutil.move() function.
I'll attach a patch for both of these just to give an idea what I'm talking
about.
Created attachment 15984 [details, diff]
dispatch-conf-2.0.49_pre17-merge_fixes.patch
merge fixes for vulnurable name creation and /tmp being a separate filesystem
Max: My version of dispatch-conf doesn't have either of these problems because it creates the merge file in the same directory as the ._cfg file (by substituting "mrg" for "cfg" in the filename). It also fixes a few other deficiencies in the merge process, such as not overwriting the system's cfg version (allowing you to fall back to the unmerged version or to diff the two versions) and the ability to do an automatic 3-way merge and present that to you as a default merged version. I personally consider the current code's cfg-overwriting behavior to be major deficiency, since it makes it too easy for a single merge mistake to lose some of the new data supplied by the package. In my version, you can notice the mistake and try again. Wayne: Good to hear! I would love to see your changes go in. Going to try them here and report back. Any reason why your attachments are marked obsolete? My old attached files are marked as obsolete because I switched over to just putting my latest versions on my web server at this URL (since I was making so many attachments):
http://www.blorf.net/dispatch-conf/
I could attach a tar of these files to this bug, if desired -- that would probably be less confusing now that the changes have subsided.
Created attachment 16059 [details]
Wayne's proposed version.
Here's a tar.bz2 file of my proposed version of dispatch-conf and an adjunct
script called archive-conf (which saves off a config file before you modify
it).
Hi,
archive-conf seems broken here. I've tried both the one from your
website and the one from recent portage, and here is what I get:
gromit /root # archive-conf /etc/make.conf
File "/usr/sbin/archive-conf", line 90
dispatch_conf.rcs_archive_post_process(archive):
^
SyntaxError: invalid syntax
Am I wrong or should this whole line (and its non-rcs counterpart) be removed? I don't see way there should be some .dis/.dist.new files to rename. Or is there something I've not understood in the way it's suppsed to be used?
Thanks for any bugfixes or explanations.
There is a trailing colon on that line (and the one 3 lines below it) that needs to be removed. The calls themselves need to be there, for the following reason: The code checks if the current config file (that it is saving off) matches the md5 of the distributed config file, and if it does, it saves the file as the ancestor file for doing 3-ways diffs in the future. In the rcs version, this means committing the FILE.dist.new file onto the branch (after renaming it to just "FILE"). Okay, thanks for your quick answer and fix. Created attachment 17243 [details, diff] Fix a few bugs This patch fixes a few problems in the version in 16059 (that was included in the last portage update): - Removed a couple superfluous colons from archive-conf (fixing a syntax error). - Changed where we log the changes we made from /tmp/dispatch-conf.changes to /var/log/dispatch-conf.changes (fixing a security problem with root writing a known tmp-file name). - Changed where we create the temporary merge-output file from a file in the /tmp dir to a file in the same dir as the config file (it is named with a "tmp" in place of the "mrg"). [Max: this fixes the security problem you reported in comment #30 that I erroneously said was not present in my version -- my apologies for not checking that better before.] - Added a missing post_process() call to the 'z' (zap new config) command. This makes sure that the latest distributed version get archived when using the 'z' command. I've tested these changes and they work fine. Created attachment 17602 [details]
dialog interface to dispatch-conf
A very simple dialog interface to dispatch-conf, works rather nicely.
Known Issues:
-Is known to break when using really small terminal window size, that should be
solvable, though
Created attachment 17603 [details]
Pythondialog with some extensions by me needed to use dispatch-conf-dialog
put this in the same directory you put dispatch-conf-dialog (it is not the
standard Pythondialog in Portage)
We've had two portage releases since my patch in attachment 17243 [details, diff] and it
has still not been applied. This patch fixes 2 security problems, a syntax
error (in archive-conf) and fixes the archiving of the ancestor config file
when using the 'z' command. This patch needs to be applied right away.
Included for 49-r17 dialog stuff is renamed to 'dcdialog' and that doesn't affect anything as you 'import *' it. @Wayne: you may think I'm a bit slow, but in fact I completely forgot (sorry about that) to reply here in september when I realized the following (while re-testing archive-conf after my comment #38): If you use archive conf to simply make a backup of a modified config file (that doesn't match any vanilla file md5) then "md5_match_hash[conf]" is an empty string, no ".dist.new" file is created in "dispatch_conf.{rcs,file}_archive", and it makes the "{rcs,file}_archive_post_process" useless. Actually, they break in that situation: Traceback (most recent call last): File "/usr/sbin/archive-conf", line 99, in ? archive_conf() File "/usr/sbin/archive-conf", line 95, in archive_conf dispatch_conf.file_archive_post_process(archive) File "/usr/lib/portage/pym/dispatch_conf.py", line 150, in file_archive_post_process os.rename(archive + '.dist.new', archive + '.dist') OSError: [Errno 2] No such file or directory I suggest adding a test for this case before the call to the post_process function. I will attach the patch I use in general for dispatch-conf (which include also your other fixes from attachment #17243 [details, diff]). @Carpaski: BTW, your comment #47 makes me think you must have forgot this attachment #17243 [details, diff], no? Because it is neither included in -r18 nor in cvs. Or you were only talking about the dialog stuff? Created attachment 21970 [details, diff]
dispatch-conf-fixes.patch
what I suggest as a replacement for attachement #17243
|
