Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 51704 - MySQL cache backend for portage
Summary: MySQL cache backend for portage
Status: VERIFIED LATER
Alias: None
Product: Portage Development
Classification: Unclassified
Component: Enhancement/Feature Requests (show other bugs)
Hardware: All Linux
: High enhancement (vote)
Assignee: Portage team
URL: http://oneofone.limitlessfx.com/phpwi...
Whiteboard:
Keywords: Inclusion
: 35103 (view as bug list)
Depends on:
Blocks:
 
Reported: 2004-05-21 16:01 UTC by Roman Gaufman
Modified: 2007-01-11 13:08 UTC (History)
12 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments
/etc/portage/modules (modules,151 bytes, text/plain)
2004-05-21 16:03 UTC, Roman Gaufman
Details
portage-init-db.sql (portage-init-db.sql,852 bytes, text/plain)
2004-05-21 16:04 UTC, Roman Gaufman
Details
/usr/lib/portage/pym/portage_db_mysql.py (portage_db_mysql.py,7.65 KB, text/plain)
2004-05-21 16:04 UTC, Roman Gaufman
Details
new version (portage_db_mysql.py,7.04 KB, text/plain)
2004-06-04 22:29 UTC, Ahmed Farid
Details
init-portage-db.sql (init-portage-db.sql,831 bytes, text/plain)
2004-06-04 22:32 UTC, Ahmed Farid
Details
new module (portage_db_mysql.py,10.82 KB, text/plain)
2004-08-22 01:36 UTC, Ahmed Farid
Details
simple search script to replace esearch (myportage-search.py,3.21 KB, text/plain)
2004-08-22 01:40 UTC, Ahmed Farid
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Roman Gaufman 2004-05-21 16:01:20 UTC
Yes, this is critical, this is a hanging issue, on my slower systems, waiting for responce from portage takes up to 20 minutes in some cases, this is not acceptable and easily fixed with the solution bellow:

I've come across this improvement to portage, the steps are explained on this site, and I summarize them bellow:

the 3 required files are to follow with attachments.

1) emerge mysql-python
2) /etc/init.d/mysql start
3) mysql -u root -p < portage-init-db.sql
4) create: /usr/lib/portage/pym/portage_db_mysql.py
5) create: /etc/portage/modules
6) emerge regen

Reproducible: Always
Steps to Reproduce:
1. emerge something
2. hang yourself after waiting for response for over 10 minutes
3.



Expected Results:  
fast responce, would be very nice if this is (atleast optionally) implemented 
into portage. Have everything needed for used to just specify USE_MYSQL=yes 
in /etc/make.conf or something.
Comment 1 Roman Gaufman 2004-05-21 16:03:28 UTC
Created attachment 31813 [details]
/etc/portage/modules
Comment 2 Roman Gaufman 2004-05-21 16:04:19 UTC
Created attachment 31814 [details]
portage-init-db.sql
Comment 3 Roman Gaufman 2004-05-21 16:04:53 UTC
Created attachment 31815 [details]
/usr/lib/portage/pym/portage_db_mysql.py
Comment 4 Brian Harring (RETIRED) gentoo-dev 2004-05-24 11:57:51 UTC
*** Bug 35103 has been marked as a duplicate of this bug. ***
Comment 5 Tuan Van (RETIRED) gentoo-dev 2004-06-01 14:03:12 UTC
`equery list package` `ecat -v package` do not work with this method. gentoolkit-0.2.0_pre8

# equery list -i -o -p squirrelmail
Searching for package 'squirrelmail' in all categories among:
 * installed packages
 * Portage tree (/usr/portage)
 * overlay tree (/usr/local/portage)
[-P-] [ ~] mail-client/squirrelmail-1.4.3-r1 (1.4.3-r1)
[-P-] [  ] mail-client/squirrelmail-1.4.3 (1)
[-P-] [  ] mail-client/squirrelmail-1.4.3_rc1 (1)
[-P-] [ ~] mail-client/squirrelmail-1.4.3_rc1-r1 (1.4.3_rc1-r1)
Exception exceptions.AttributeError: "'NoneType' object has no attribute 'close'" in <bound method Cursor.__del__ of <MySQLdb.cursors.Cursor instance at 0x40a94aac>> ignored

# etcat -v squirrelmail
[ Results for search key           : squirrelmail ]
[ Candidate applications found : 4 ]

 Only printing found installed programs.

*  mail-client/squirrelmail :
        [   ] 1.4.3_rc1 (1)
        [ ~ ] 1.4.3_rc1-r1 (1.4.3_rc1-r1)
        [   ] 1.4.3 (1)
        [ ~ ] 1.4.3-r1 (1.4.3-r1)
Exception exceptions.AttributeError: "'NoneType' object has no attribute 'close'" in <bound method Cursor.__del__ of <MySQLdb.cursors.Cursor instance at 0x409f202c>> ignored
Comment 6 Ahmed Farid 2004-06-04 22:29:51 UTC
Created attachment 32679 [details]
new version

new version to fix the bugs reported before.
Comment 7 Ahmed Farid 2004-06-04 22:32:22 UTC
Created attachment 32680 [details]
init-portage-db.sql

the new sql file needed, please drop the old database first then 
mysql -u root < init-portage.db.sql
Comment 8 Nicholas Jones (RETIRED) gentoo-dev 2004-06-05 17:18:19 UTC
Requirements:
The module needs to be self-contained. It need to be able to create or
alter the tables itself without user intervention.

If you post changes to something PATCH IT please. I don't like spending
time to see what was done.
Comment 9 Ahmed Farid 2004-08-22 01:36:02 UTC
Created attachment 37914 [details]
new module

This is a new version, uses a sep. table for package info.
It also handle creating tables.
Just drop all the tables in portage db, install this module and emerge
sync/regen.
please note that you must have the user/database created already or it won't
work.
This works only on .51*
Comment 10 Ahmed Farid 2004-08-22 01:40:30 UTC
Created attachment 37916 [details]
simple search script to replace esearch

this a simple script to replace esearch, with support for searching USE,
provides, keywords (~x86, etc),categories, licenses.
chmod +x it then ./myportage-search.py -h for more details
Comment 11 Jeff Smelser 2004-10-27 10:44:35 UTC
I am getting errors.. I deleted the tables.. and let the portage user create them again.. Here is what I get:

Regenerating cache entries...
processing app-accessibility/SphinxTrain

  error processing app-accessibility/SphinxTrain-0.9.1-r1, continuing... (Error inserting/updating the database ('UNUSED_01').)
processing app-accessibility/brltty
processing app-accessibility/festival

  error processing app-accessibility/festival-1.4.3-r1, continuing... (Error inserting/updating the database ('UNUSED_01').)

  error processing app-accessibility/festival-1.4.3-r2, continuing... (Error inserting/updating the database ('UNUSED_01').)
processing app-accessibility/flite
Caught signal 2 in pid 26020
{'keys': 0, 'set': 9, 'del': 0, 'get': 21}
{'keys': 0, 'set': 9, 'del': 0, 'get': 21}
Exiting due to signal

The grants for mysql are:
+-------------------------------------------------------------------------------------------------------+
| Grants for portage@%.smelser.org                                                                      |
+-------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON *.* TO 'portage'@'%.smelser.org'                                                       |
| GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, LOCK TABLES ON `portage`.* TO 'portage'@'%.smelser.org'
Comment 12 G.K.MacGregor 2004-11-08 13:52:05 UTC
In the new version I got those
"Error inserting/updating the database ('UNUSED_01')" flooding my
console as well, even though the MySQL grants were OK.
Comment 13 Jeff Smelser 2004-11-08 13:58:26 UTC
I had to revert back to http://bugs.gentoo.org/show_bug.cgi?id=51704#c6.. Seems to work fine on .51
Comment 14 Jason Stubbs (RETIRED) gentoo-dev 2005-07-28 07:25:32 UTC
Putting a hold on feature requests for portage as they are drowning out the 
bugs. Most of these features should be available in the next major version of 
portage. But for the time being, they are just drowning out the major bugs and 
delaying the next version's progress. 
 
Any bugs that contain patches and any bugs for etc-update or dispatch-conf can 
be reopened. Sorry, I'm just not good enough with bugzilla. ;) 
Comment 15 Marius Mauch (RETIRED) gentoo-dev 2007-01-11 13:08:00 UTC
Closing due to old age (doesn't work with portage-2.1).