It's really not funny anymore. I sometimes wait up to a minute for a response. Don't know, if this may have something to do with the recent outage, if bug whining is so expensive or whatever.
This is IMHO not jforman's fault really... :) From what I've been told, the DB server gets hammered by planet.g.o. pretty often (on a side note, I'll file a separate bug about planet.g.o. exhibiting broken behaviour lately). Also, these issues have been lasting for ~10 days, yesterday bugzilla was completely unreachable from ~8 to ~10 UTC. Really annoying, yeah.
P.S. The above comment took 3 timeouts and one Internal Server Error to get thru. ~20 minutes altogether. :=(
+1, Please fix this.
So I'm sitting at LWE now and wolf31o2 suggested that I turn off whining and see how that improves performance, so that's what I'll do. See -core email. -Jeffrey
I have to say this is bad PR for gentoo. We need a working bugzilla, I hope there is enough hardware around to get this running smoothly again.
(In reply to comment #5) > I have to say this is bad PR for gentoo. PR is the last problem, imho. From my POV it's not as worse as before disabling the whining feature. But bugzilla responses were never really quick and having to twiddele your thumbs (or better doing something else), until it has processed requests, is annoying and while not the regular case, it's not that it happens rarely. If there is really a problem with the db server load, shut down planet.g.o.
Last I heard, planet's b2evo installation was not causing any load problems. Randomly killing websites probably won't help :)
Daniel: I wrote "if" in reference to Jakub's comment. Of course I'm not the one being able to gather the data to find out, what the problem is.
Bugzilla is very picky about locking tables compared to phpBB. I can at least explain the the massive slowdown between 0800-0900 UTC every day. That is the time when the OSL is running the mysqldump for the backup which will cause locks on the tables. We are looking at different solutions to the problem, one of which is to finally move to a new mysql cluster the OSL has been working on. Jeff is currently working on testing the next bugzilla upgrade on that cluster, and if things go well, we'll most likely move bugzilla there and see how things go. It sounds like we should be able to run the dump from the slave machine which should hopefully take away the locking issues during backups. Now for the random issues during the day I suspect its some run away query that locks a bunch of tables. This is either a bug in bugzilla itself, or something else is going on. From the graphs of the all the machines involved, I don't see any load issues causing the problem. I'm not entirely convinced that the planet issues are related to the bugs issues yet. I'm hoping we can get access to the slow query log so we can maybe sort out this problem.
We've identified a possible oom condition that could be causing stale locks and other general nastiness. More to come soon.
Ok, so... ??? Lately, my estimate is that total downtime might be ~2-3 hours every day (~1 hour for daily backup + 1-2 hours randomly dead). Just imagine CVS/SVN was similarly broken, I'd say we'd have a small revolution soon. Any ETA when this will be fixed?
I will talk to Robert (r2d2) about his query limiting implementation and maybe we can schedule some downtime for early next week.
guess still no progress huh ?
Our main contact at the OSL in regards to the new replicated bugzilla setup has been sick this week and kinda MIA. Along with the fact that I am graduating this weekend from syr.edu, so it's been kind of hectic. I'm planning on doing the upgrade next week sometime, just need to do some last minute testing.
Ping... any progress here wrt the migration to new setup?
I'm still waiting on the cluster setup to be declared stable. In the osl infra channel the other day, i saw the machine going up and down, so it's clearly not production-ready just yet. I will ping them again today and see what the deal is. Believe me, I want clustered goodness as much as you guys. -Jeffrey
???
(In reply to comment #17) > ??? Is that a question or a statement? Kind of hard to figure out what you want with those characters ;-).
Hmmm... <KingTaco> chizu, it wasn't clear last night, are you still working on the mysql cluster for OSL? <chizu> Nope <chizu> I'm not sure who's taken over that. Soooo... Since everything else failed, I'll request this to be on next council meeting agenda unless the situation gets solved by then. Just a preemptive notice.
(In reply to comment #19) > Hmmm... > > <KingTaco> chizu, it wasn't clear last night, are you still working on the > mysql cluster for OSL? > <chizu> Nope > <chizu> I'm not sure who's taken over that. > > Soooo... Since everything else failed, I'll request this to be on next council > meeting agenda unless the situation gets solved by then. Just a preemptive > notice. Thanks for taking that short excerpt out of context. Chizu is doing an internship with Mozilla this summer, so he's not at the OSL right now. But that doesn't mean that work on the db cluster has stopped. Kingtaco has been working on getting something going on those grounds. I don't appreciate threats as we are still working on the problem and haven't abandoned it. I also have another route we could take if it looks like the OSL is going to take much longer. I need to get in contact with the sponsor to see how long it would take for us to get that hardware.
(In reply to comment #20) > I don't appreciate threats as we > are still working on the problem and haven't abandoned it. I also have another > route we could take if it looks like the OSL is going to take much longer. I > need to get in contact with the sponsor to see how long it would take for us to > get that hardware. Threats? I'm telling you that I want the council to discuss this, as it's been taking frickin' 4 months now without any visible progress. If you see this as a threat, oh well that's not my problem really. Wrt the hardware, we've been offered that a couple of times already, but it's been declined b/c of HW "consolidation". Shrug.
(In reply to comment #21) > Threats? I'm telling you that I want the council to discuss this, as it's been > taking frickin' 4 months now without any visible progress. If you see this as a > threat, oh well that's not my problem really. How do you define visible progress? If you're talking about the end result, then yes, you wouldn't see much progress. But there is much progress going on in the background that you seem to ignore or don't appreciate. > Wrt the hardware, we've been offered that a couple of times already, but it's > been declined b/c of HW "consolidation". Shrug. That was one reason, and also because it seemed as though the hardware at the OSL for the cluster was getting available soon. They just got two T2000s and wanted to work on that now. I can't wait any longer so I'm looking at those options now as possible temporary solutions. I just emailed the sponsor this morning about getting three blade servers that he offered me a few weeks back. I just don't appreciate the negative tone you have on this whole issue. It makes all of us involved not to want to work on it at all. I have asked in the past if anyone knew of a mysql guru that could help us out. I have yet to see any of you follow up on that. Thanks for proving again about the beauties of our thankless 'job' we have at Gentoo.
(In reply to comment #22) > How do you define visible progress? If you're talking about the end result, > then yes, you wouldn't see much progress. But there is much progress going on > in the background that you seem to ignore or don't appreciate. Well, I define visible progress as in "this damned simple search won't timeout on me ten times in a row" - pretty simple and measurable and all. So no, there has not been really any progress at all. A background diplomacy won't make my search faster and won't save me from wasting my time because the box is unresponsive or dead a couple of hours every day... > I just don't appreciate the negative tone you have on this whole issue. It > makes all of us involved not to want to work on it at all. I have asked in the > past if anyone knew of a mysql guru that could help us out. I have yet to see > any of you follow up on that. > > Thanks for proving again about the beauties of our thankless 'job' we have at > Gentoo. Remarkable to see so many projects use bugzilla w/o having any such issues, or at least without having 3+ hours downtime every day. Maybe they employ some magicians, ask them. Otherwise - well, not really my business - if it makes you "not to want to work on it at all", then quit. Won't make much difference wrt bugzilla state, quite frankly said - and that's what this bug is all about. If the only outcome after 4 months is that we even don't know who (and if anyone) is working on this... hmmm.
Jakub, Apparently you missed what Lance said. It's your attitude that's making him feel this way and this approach that you appear to have that you are ungrateful for the fact that we get our current bugzilla setup for FREE thanks to the gracious people at the OSUOSL. As such, it may take longer for some of these problems to get fixed than other projects or organizations with these lightning fast bugzilla implementations you reference. These are conditions we will probably have to deal with for some time unless we can get some donated datacenter/gateway space and bandwidth or the Foundation can afford to get that space on a regular basis. Also, if you're feeling emotionally charged when you post (like you were in comment #21 and comment #23 for example), please walk away from the keyboard for a few minutes. These kinds of comments don't really help anyone, and you help perpetuate the feeling expressed by some developers lately of this project going to hell in a handbasket because of poor personal interaction.
Jakub, seriously chill out, for f*ck's sake. It *is* a pain in the ass, I agree, jeff agrees and even Lance agrees. We can get all confrontational with each other, which achieves exactly nothing, or we can try and work as a team. I don't think confrontational is the best approach. So let's leave out talk of the council here, and also talk of devrel, shall we?
(In reply to comment #25) > Jakub, seriously chill out, for f*ck's sake. It *is* a pain in the ass, I > agree, jeff agrees and even Lance agrees. We can get all confrontational with > each other, which achieves exactly nothing, or we can try and work as a team. > I don't think confrontational is the best approach. So let's leave out talk of > the council here, and also talk of devrel, shall we? Right, so just make infra do something then, so far Lance is just moaning how thankless people are. I've been damn silent and polite here for 3 months, then someone gets pissed off when I say that I'll ask the council to discuss an issue that's been a huge PITA for months and sees this as a threat? Whom should I ask then? Just would really love to see some progress here, it's wasting hours of my time every day, hard to understand? All this talking about "oh, we are just waiting for Mr. X to ask Mr. Y to tell Mr. Z. that the situation sucks, be patient meanwhile" doesn't get this much further.
I know I have absolutley nothing to do with this bug but... jakub, ramereth, kingtaco and jforman (and others) have been working overtime trying to get this setup. It's not there fault that chizu has this wonderful chance to work for mozilla this summer. If you hang out in #gentoo-infra you will see that there *is* visible work occuring on this issue. We all know how important bugs.g.o is to the Gentoo project and it is, and always has been, one of our number one priorities. Please give the team working on this a little while longer before you ask the council to discuss it. Maybe just a CC of the council members to this bug should be enough to inform them of the situation and they can ask the team if they have any questions. Thanks
not pointing fingers, but would infra please consider asking the mysql team for mysql assistance? it's not like we maintain it for nothing. I do have MySQL certification, and I've been an upstream developer of phpMyAdmin since long before I joined Gentoo ;-).
Doh, never thought about asking you guys :). Guess I figured someone within Gentoo would pipe up eventually. Do you think you guys could help us out with possibly fixing the table locking issues we're having? If you could get with Kingtaco that would be best. Thanks!
(In reply to comment #29) > Do you think you guys could help us out with > possibly fixing the table locking issues we're having? Hi there maybe checkout this blog post : http://blogs.gnome.org/view/ovitters/2006/08/12/0 and this bug number for the patch : https://bugzilla.mozilla.org/show_bug.cgi?id=57350 The patch is already backported for bugzie 2.20. I spotted setup replication, slow queries and table locking in the blog post so it should help until the new blade server is ready to go. Cheers and keep on the good work guys !
We're days away from getting the mysql stuff up on the new cluster, at which time I will work my hardest to get the cluster up with the new setup. Just please sit tight, this is a big change and won't happen overnight.
To continue, the recent mention of this patch to magically speed it up, will not be applied. Since we're moving installations from one center to another, I would rather keep the applications of changes to a minimum. Thanks for understanding.
(In reply to comment #32) > To continue, the recent mention of this patch to magically speed it up, will > not be applied. Since we're moving installations from one center to another, I > would rather keep the applications of changes to a minimum. Not to butt in here, but if its a fairly easy patch, we could apply it on the current bugstest.g.o and people could hammer that. However, I'm not sure what state that version of the bugs instance is and Jeff would have to comment on it. I really don't see a problem if we at least patch it on that instance while we wait on kingtaco/robbat2 to get the database server up. I am curious too if this particular patch helps us in the short run. (if not, a little) I would just hate to see progress on the new setup slowed because of this patch. Cheers-
(In reply to comment #33) > (In reply to comment #32) > > To continue, the recent mention of this patch to magically speed it up, will > > not be applied. Since we're moving installations from one center to another, I > > would rather keep the applications of changes to a minimum. > > Not to butt in here, but if its a fairly easy patch, we could apply it on the > current bugstest.g.o and people could hammer that. However, I'm not sure what > state that version of the bugs instance is and Jeff would have to comment on > it. I really don't see a problem if we at least patch it on that instance while > we wait on kingtaco/robbat2 to get the database server up. > > I am curious too if this particular patch helps us in the short run. (if not, a > little) I would just hate to see progress on the new setup slowed because of > this patch. > I will try to get to this tonight (8/15) but I make no promises with other personal issues going on right now.
(In reply to comment #34) > I will try to get to this tonight (8/15) but I make no promises with other > personal issues going on right now. Thanks, if you need help feel free to abuse/use Kingtaco or anyone else you feel comfortable helping you with bugzilla.
attempting to do patch: nuthatch Bugzilla # patch -p2 < bugzilla-patch-57350-v4 patching file DB.pm Hunk #1 FAILED at 218. Hunk #2 FAILED at 468. Hunk #3 succeeded at 915 with fuzz 1 (offset -1736 lines). 2 out of 3 hunks FAILED -- saving rejects to file DB.pm.rej ------------------------------------ nuthatch Bugzilla # cat DB.pm.rej *************** *** 218,229 **** _recrypt_plaintext_passwords(); - # 2001-06-06 justdave@syndicomm.com: - # There was no index on the 'who' column in the long descriptions table. - # This caused queries by who posted comments to take a LONG time. - # http://bugzilla.mozilla.org/show_bug.cgi?id=57350 - $dbh->bz_add_index('longdescs', 'longdescs_who_idx', [qw(who)]); - # 2001-06-15 kiko@async.com.br - Change bug:version size to avoid # truncates re http://bugzilla.mozilla.org/show_bug.cgi?id=9352 $dbh->bz_alter_column('bugs', 'version', --- 218,223 ---- _recrypt_plaintext_passwords(); # 2001-06-15 kiko@async.com.br - Change bug:version size to avoid # truncates re http://bugzilla.mozilla.org/show_bug.cgi?id=9352 $dbh->bz_alter_column('bugs', 'version', *************** *** 474,479 **** # 2006-08-19 LpSolit@gmail.com - Bug 87795 $dbh->bz_alter_column('tokens', 'userid', {TYPE => 'INT3'}); ################################################################ # New --TABLE-- changes should go *** A B O V E *** this point # ################################################################ --- 468,475 ---- # 2006-08-19 LpSolit@gmail.com - Bug 87795 $dbh->bz_alter_column('tokens', 'userid', {TYPE => 'INT3'}); + _update_longdescs_who_index(); + ################################################################ # New --TABLE-- changes should go *** A B O V E *** this point # ################################################################ nuthatch Bugzilla #
Any progress on this?
Moving open bugzilla bugs to the new bugzilla group (because I'm about to stab lots of these bugs).
the new bugzilla boxes are online now. I did apply the index change mentioned here as well (directly to the DB actually)
