Bugzilla Bug 149327

OK i think this is big enough for a bug of its own.  When pbs_server starts it
tries to read the .keep file as it thinks it is a queue file.  This shouldn't
be a problem -- the server will error with a 'que_recov: read error', which
_should_ just be an annoying error message.

However, i had intermittent crashing issues with pbs_server when i was
modifying my queues (mainly when trying to delete recoverd jobs with qdel)..

A search through the code seems to indicate that there's an error in que_recov,
as when it allocates memory to load the queue data it adds that structure into
a linked list--but on error it just free's the structure and leaves the linked
list pointing at invalid memory.

I've sent a message upstream to find out if this is actually a bug.

Created an attachment (id=98250) [edit]
Patch to fix que_recov

The following patch fixed the issue on my system -- i changed the direct free()
calls in que_recov to que_free() (the counterpart of que_alloc(), which is how
the structure is made).

Please reopen when you get a resolution from the upstream guys. I try to avoid
adding patches to our stuff until upstream's looked them over, in cases where
upstream is responsive.

Created an attachment (id=98277) [edit]
upstream patch to skip dot-filenames

Upstream hasn't responded on the memory free'ing issuee yet, but they did patch
the load function to skip dot-filenames (as i mentioned in bug 149226) which is
what caused the issue for me in the first place.

I attached the patch they provided for convenience.

Ian, thanks for your quick response and effort on this! For reference,
http://www.clusterresources.com/pipermail/torquedev/2006-September/000314.html
is the post. I added the dot-file skip patch to 2.1.2-r2 since the problem's
likely to affect many people.

The 'Patch to fix que_recov' above has been added upstream now too.
http://www.clusterresources.com/pipermail/torquedev/2006-September/000316.html

I guess we can close this one?

Is there any reason for us to add the original patch, now that we've got the
second one?

Depends -- if que_recov() fails on one queue file (couldn't read the queue
file, file descriptor is bad) but then succeeds on another, then memory
corruption will probably occur (and the pbs_server will crash).  But, as
mentioned upstream, if this happens then you have bigger problems.  

And with the second patch (skipping dot-filenames) already applied, there isnt
any reason that loading the queue file will fail (assuming nobody plays with
the contents of server_priv/queues)..

I suppose we might as well add it, but I won't do a revision bump this time.

Applied the que_recov patch as well. Thanks!