Gentoo Websites Logo
Go to: Gentoo Home Documentation Forums Lists Bugs Planet Store Wiki Get Gentoo!
Bug 317793 - rsync5.us.gentoo.org has gone off the rails (disk problems?)
Summary: rsync5.us.gentoo.org has gone off the rails (disk problems?)
Status: RESOLVED FIXED
Alias: None
Product: Mirrors
Classification: Unclassified
Component: Server Problem (show other bugs)
Hardware: All Linux
: High normal (vote)
Assignee: Mirror Admins
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-04-29 16:28 UTC by Matt Summers (RETIRED)
Modified: 2010-05-07 10:53 UTC (History)
2 users (show)

See Also:
Package list:
Runtime testing required: ---


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Matt Summers (RETIRED) gentoo-dev 2010-04-29 16:28:10 UTC
emerge --sync produces insane output, many many errors.

Reproducible: Always

Steps to Reproduce:
1.SYNC="rsync://129.21.171.98/gentoo-portage #rsync5.us.gentoo.org
2.# emerge --sync
3.

Actual Results:  
I see many many errors like the one below, for all sorts of pkgs. Ebuild & patch verification failures a' plenty

ERROR: metadata/cache/sys-kernel/git-sources-2.6.34_rc5-r3 failed verification -- update discarded.                          
rsync: read errors mapping "/metadata/cache/sys-kernel/git-sources-2.6.34_rc5-r3" (in gentoo-portage): Input/output error (5)


Expected Results:  
Clean sync

Not sure what Severity to give it so I'll leave that to the wranglers. Thanks!
Comment 1 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-04-29 16:33:04 UTC
Thanks, disabled from rsync.us and rsync.namerica rotations for now.

Paul, can you please look at this when time allows. Seems like disk problems?
Comment 2 Jeremy Olexa (darkside) (RETIRED) archtester gentoo-dev Security 2010-05-06 03:23:56 UTC
(In reply to comment #1)
> Thanks, disabled from rsync.us and rsync.namerica rotations for now.
> 
> Paul, can you please look at this when time allows. Seems like disk problems?
> 

Appears fine now. re-enabled.
Comment 3 Paul Mezzanini 2010-05-06 11:27:11 UTC
The 3ware raid controller in this machine is, frankly, a piece of crap.  Any very heavy IO and the machine does a KP.    Supermicro blames 3ware, 3ware blames Supermicro.  

The new ubuntu came out and that put it over the edge.
Comment 4 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2010-05-06 22:45:39 UTC
(In reply to comment #3)
> The 3ware raid controller in this machine is, frankly, a piece of crap.
Just wondering what 3ware, and if the card is maybe faulty? I've had a lot of 3ware gear in my personal and work boxes over the years, and push a lot of traffic through them (mostly on DB servers).

I have seen a number of misconfiguration issues with kernels/3ware also cause problems however.
Comment 5 Paul Mezzanini 2010-05-07 01:10:00 UTC
(In reply to comment #4)
> (In reply to comment #3)
> > The 3ware raid controller in this machine is, frankly, a piece of crap.
> Just wondering what 3ware, and if the card is maybe faulty? I've had a lot of
> 3ware gear in my personal and work boxes over the years, and push a lot of
> traffic through them (mostly on DB servers).
> 
> I have seen a number of misconfiguration issues with kernels/3ware also cause
> problems however.
> 
I've mainly used 3ware cards in the past but lately I have had issues.  This particular machine is using a 9650SE-8LP with Fedora 11.  The same problem existed with Centos 5.4 and Gentoo.  I've also had issues with the 9690 series.  This card should be out of warranty so I'm SOL with it.

I just purchased an LSI card to replace the 3ware.  I will be running software RAID once it shows up.  
Comment 6 Robin Johnson archtester Gentoo Infrastructure gentoo-dev Security 2010-05-07 02:35:23 UTC
(In reply to comment #5)
> I've mainly used 3ware cards in the past but lately I have had issues.  This
> particular machine is using a 9650SE-8LP with Fedora 11.  The same problem
> existed with Centos 5.4 and Gentoo.  I've also had issues with the 9690 series.
>  This card should be out of warranty so I'm SOL with it.
Weird. You mentioned SuperMicro also passing the blame, was it something with the SAS backplane there? All my work DBs are using 9690's, and aside from issues with firmware in the past, they've been rock solid. I've got an 9650SE-8LPML in my desktop that used to be in a fileserver before it got upgraded, it's got one weird BIOS boot order issue, but actual performance is solid.

> I just purchased an LSI card to replace the 3ware.  I will be running software
> RAID once it shows up.  
Lots of newest LSI cards are very similar to 3ware since 3ware got bought out by LSI. Right down to the same tw_cli tool.
Comment 7 Paul Mezzanini 2010-05-07 10:53:48 UTC
(In reply to comment #6)

> Weird. You mentioned SuperMicro also passing the blame, was it something with
> the SAS backplane there? 

It is actually a SATA backplane running SATA drives.  The twin to this machine is my backuppc box with the 9650 running the internal drive bays and a 9690 running two external SAS chassis.  I beat the crap out of that machine so I'm pretty sure I just have a dud card.  I still have the bad 9690 in a box.  3ware never told me where to mail it so I never sent it.   

> 
> > I just purchased an LSI card to replace the 3ware.  I will be running software
> > RAID once it shows up.  
> Lots of newest LSI cards are very similar to 3ware since 3ware got bought out
> by LSI. Right down to the same tw_cli tool.
> 
True, but this card predates the 3ware/LSI merger.  It uses the LSI SAS1068E chip.  Worst case, it actually is a motherboard issue and I need to find a new box to host mirrors.