Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 543814 - app-portage/mirrorselect ignores continuation lines (backslash) in make.conf
Summary: app-portage/mirrorselect ignores continuation lines (backslash) in make.conf
Status: RESOLVED FIXED
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Tools (show other bugs)
Hardware: All Linux
: Normal normal (vote)
Assignee: Portage Tools Team
URL:
Whiteboard:
Keywords: InVCS
Depends on:
Blocks:
 
Reported: 2015-03-19 18:24 UTC by Jack
Modified: 2023-07-06 16:46 UTC (History)
0 users

See Also:
Package list:
Runtime testing required: ---


Attachments
Use shlex to support multi-line GENTOO_MIRRORS assignments (write_make_conf-support-multi-line-GENTOO_MIRRORS-bu.patch,5.69 KB, patch)
2019-05-27 07:42 UTC, Zac Medico
Details | Diff
GENTOO_MIRROPRS patch for multiline mirror list (backslah.patch,454 bytes, patch)
2022-05-31 08:15 UTC, Brian Dolbec (RETIRED)
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Jack 2015-03-19 18:24:10 UTC
If GENTOO_MIRRORS in make.conf is split over more than one line, using a backslash to indicate continuation, mirrorselect only replaces the first line, putting the new GENTOO_MIRRORS at the end of the file, but leaving all the previous continuation lines intact, leaving make.conf in an invalid state.  (I'm using 2.2.2)

Reproducible: Always

Steps to Reproduce:
1. use backslash and continuation lines for GENTOO_MIRRORS in make.conf
2. run mirrorselect (-s5 in my example)
3.
Actual Results:  
The new GENTOO_MIRRORS is placed at the end of the file, all on one line.  The original GENTOO_MIRRORS is removed, but only the first line, not the continuation lines.

Expected Results:  
All lines making up GENTOO_MIRRORS are removed.  Optionally, I would like to see the new value also use continuation lines with one server per line, but I'm willing to let that be a separate enhancement request.

I'm calling this a normal bug, but it does leave make.conf in an invalid state.  Since I caught the problem before running emerge, I don't know what problems it would cause - hopefully only warning messages.
Comment 1 Jack 2017-08-18 18:15:59 UTC
I just confirmed the problem is still present with versions 2.2.2-r2 and 2.2.3.
Comment 2 Jack 2019-05-26 23:38:19 UTC
Problem still present in mirrorselect-2.2.4-r1
Comment 3 Zac Medico gentoo-dev 2019-05-27 03:34:25 UTC
The faulty code is located at https://gitweb.gentoo.org/proj/mirrorselect.git/tree/mirrorselect/configs.py?h=2.2.4#n78:

>	regex = re.compile('^%s=.*' % var)
>	for line in lines:
>		if regex.match(line):
>			lines.remove(line)
Comment 4 Zac Medico gentoo-dev 2019-05-27 05:01:11 UTC
We can use a shlex instance to detect the line number where the GENTOO_MIRRORS setting ends. This is the same approach that portage uses to read make.conf, and mirrorselect already uses this approach in the get_filesystem_mirrors function here:

https://gitweb.gentoo.org/proj/mirrorselect.git/tree/mirrorselect/configs.py?h=2.2.4#n153
Comment 5 Zac Medico gentoo-dev 2019-05-27 07:42:10 UTC
Created attachment 577868 [details, diff]
Use shlex to support multi-line GENTOO_MIRRORS assignments
Comment 6 Larry the Git Cow gentoo-dev 2019-05-27 16:42:45 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/proj/mirrorselect.git/commit/?id=42238f4ff13ae6c75bc1925f366429a35c26317d

commit 42238f4ff13ae6c75bc1925f366429a35c26317d
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-05-27 06:11:28 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-05-27 07:39:39 +0000

    write_make_conf: support multi-line GENTOO_MIRRORS (bug 543814)
    
    Use shlex to support multi-line GENTOO_MIRRORS assignments.
    
    Bug: https://bugs.gentoo.org/543814
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 .gitignore                    |  1 -
 mirrorselect/configs.py       | 33 +++++++++++++++++++++++++++------
 setup.py                      | 37 ++++++++++++++++++-------------------
 tests/__init__.py             |  0
 tests/test_write_make_conf.py | 41 +++++++++++++++++++++++++++++++++++++++++
 5 files changed, 86 insertions(+), 26 deletions(-)
Comment 7 Larry the Git Cow gentoo-dev 2019-05-27 18:11:06 UTC
The bug has been referenced in the following commit(s):

https://gitweb.gentoo.org/repo/gentoo.git/commit/?id=82bbe253084208ac9216f8a5a822e8e15361d7f4

commit 82bbe253084208ac9216f8a5a822e8e15361d7f4
Author:     Zac Medico <zmedico@gentoo.org>
AuthorDate: 2019-05-27 17:47:08 +0000
Commit:     Zac Medico <zmedico@gentoo.org>
CommitDate: 2019-05-27 18:10:57 +0000

    app-portage/mirrorselect: version bump to 2.2.5
    
    Bug: https://bugs.gentoo.org/543814
    Bug: https://bugs.gentoo.org/582508
    Package-Manager: Portage-2.3.67, Repoman-2.3.13
    Signed-off-by: Zac Medico <zmedico@gentoo.org>

 app-portage/mirrorselect/Manifest                  |  1 +
 app-portage/mirrorselect/mirrorselect-2.2.5.ebuild | 39 ++++++++++++++++++++++
 2 files changed, 40 insertions(+)
Comment 8 Jack 2020-05-20 21:29:35 UTC
I confirm 2.2.6-r1 fixes the problem.  Is there any reasonable way to get it to use backslashes and one mirror per line, especially if the previous version had it that way?  (I agree it may not be worth the effort it would take, but ask just in case.)
Comment 9 Brian Dolbec (RETIRED) gentoo-dev 2022-05-31 08:15:08 UTC
Created attachment 781601 [details, diff]
GENTOO_MIRROPRS patch for multiline mirror list

possible backslah patch for 1 mirror per line.
Comment 10 Jack 2022-08-16 22:15:40 UTC
Something is still strange.  I just tried "mirrorselect -R 'North America' -s 5" (with my current make.conf having four mirrors on separate linet) which replace the current selection with four servers on one line.  I recompiled with the patch from Comment #8, increasing to '-s 6' and then "-s 10" and it put three servers (but correctly on separate lines.)

Note that in all cases, it recognized the -s parameter saying "Using netselect to choose the top 6 mirrors...Done." with the appropriate number.

Why would it not provide the number of mirrors it said it was going to?  Is it related, or a separate bug?
Comment 11 Jack 2023-06-01 16:20:41 UTC
My comment #10 still applies (without needing that patch) with mirrorselect-2.3.0-r1 - whether I use -s 6 or -s 10, I only get three mirrors.
In addition 'mirrorselect -a -R "North America"' gives me "ERROR: Choose at most one of -s or -a".  Should I close this as FIXED and open a new bug, or can someone look at this issue?
Comment 12 Jack 2023-06-01 16:27:04 UTC
I see there is already bug 872218 for --all not working, so I suppose this can be closed as fixed, but it should probably refer to the appropriate commit.
Comment 13 Sam James archtester Gentoo Infrastructure gentoo-dev Security 2023-07-06 08:01:41 UTC
(In reply to Jack from comment #12)
> I see there is already bug 872218 for --all not working, so I suppose this
> can be closed as fixed, but it should probably refer to the appropriate
> commit.

Indeed, https://gitweb.gentoo.org/proj/mirrorselect.git/commit/?id=a09b4d302e4cee79254d3ff4a5ac9f080b958acf in 2.3.0.

Would you mind filing a new bug if there's any outstanding issues (aside from the aforementioned --all issue)? Thanks.
Comment 14 Jack 2023-07-06 16:46:29 UTC
Confirmed this does now work correctly.  Thanks.