[GE users] New update to 6.2u4; Jobs will start randomly, but most sit in queue forever.

m0zes adam.tygart at gmail.com
Wed Jan 6 17:26:25 GMT 2010


Hello Everyone,

I have been using Grid Engine for roughly 7 months now, and for the
most part it has been very stable.

I have now run into an issue that could be a deal breaker if I cannot
get it resolved.

First things first, this is a cluster of 122 nodes and 1000 cores. It
is used for scientific research at an academic institution, so we have
a variety of needs. I run 4 queues (batch.q, long.q, highmem.q, and
long-highmem.q). This had been running fairly well up until this
point.

Over the weekend I migrated to 6.2u4 (from 6.2u3), and things have
gone to hell in a hand basket.

It seems I cannot get new jobs to start. A few of them started just
after the upgrade, and now nothing seems to.

Here is from the qmaster messages file:

01/06/2010 11:05:57|  main|athena|I|starting up GE 6.2u4 (lx24-amd64)
01/06/2010 11:05:57|schedu|athena|E|element "scout71.beocat" does not exist
01/06/2010 11:05:57|schedu|athena|E|callback function for event "1.
EVENT MOD EXECHOST scout71.beocat" failed
01/06/2010 11:06:05|worker|athena|E|There is no job registered for the
following users: mozes
01/06/2010 11:06:07|schedu|athena|E|could not find job "161897" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "51.
EVENT JOB 161897.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161898" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "52.
EVENT JOB 161898.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161901" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "53.
EVENT JOB 161901.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161902" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "54.
EVENT JOB 161902.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161905" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "55.
EVENT JOB 161905.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161906" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "56.
EVENT JOB 161906.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161850" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "57.
EVENT JOB 161850.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161854" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "58.
EVENT JOB 161854.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161847" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "59.
EVENT JOB 161847.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161849" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "60.
EVENT JOB 161849.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161874" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "61.
EVENT JOB 161874.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161952" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "172.
EVENT JOB 161952.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161953" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "173.
EVENT JOB 161953.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161799" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "179.
EVENT JOB 161799.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161842" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "180.
EVENT JOB 161842.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161899" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "181.
EVENT JOB 161899.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161900" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "182.
EVENT JOB 161900.1 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161960" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "183.
EVENT JOB 161960.1 task 3.scout55 USAGE" failed
01/06/2010 11:06:07|schedu|athena|E|could not find job "161960" in master list
01/06/2010 11:06:07|schedu|athena|E|callback function for event "184.
EVENT JOB 161960.1 USAGE" failed
01/06/2010 11:06:17|schedu|athena|E|could not find job "161845" in master list
01/06/2010 11:06:17|schedu|athena|E|callback function for event "215.
EVENT JOB 161845.1 USAGE" failed
01/06/2010 11:06:17|schedu|athena|E|could not find job "161846" in master list
01/06/2010 11:06:17|schedu|athena|E|callback function for event "216.
EVENT JOB 161846.1 USAGE" failed
01/06/2010 11:06:17|schedu|athena|E|could not find job "161765" in master list
01/06/2010 11:06:17|schedu|athena|E|callback function for event "252.
EVENT JOB 161765.1 task 3.scout75 USAGE" failed
01/06/2010 11:06:17|schedu|athena|E|could not find job "161765" in master list
01/06/2010 11:06:17|schedu|athena|E|callback function for event "253.
EVENT JOB 161765.1 USAGE" failed
01/06/2010 11:06:17|schedu|athena|E|could not find job "161744" in master list
01/06/2010 11:06:17|schedu|athena|E|callback function for event "279.
EVENT JOB 161744.1 task 3.scout78 USAGE" failed
01/06/2010 11:06:17|schedu|athena|E|could not find job "161744" in master list
01/06/2010 11:06:17|schedu|athena|E|callback function for event "280.
EVENT JOB 161744.1 USAGE" failed
01/06/2010 11:06:27|schedu|athena|E|could not find job "161903" in master list
01/06/2010 11:06:27|schedu|athena|E|callback function for event "521.
EVENT JOB 161903.1 USAGE" failed
01/06/2010 11:06:27|schedu|athena|E|could not find job "161904" in master list
01/06/2010 11:06:27|schedu|athena|E|callback function for event "522.
EVENT JOB 161904.1 USAGE" failed
01/06/2010 11:06:27|schedu|athena|E|could not find job "161743" in master list
01/06/2010 11:06:27|schedu|athena|E|callback function for event "548.
EVENT JOB 161743.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161948" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "619.
EVENT JOB 161948.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161949" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "620.
EVENT JOB 161949.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161954" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "626.
EVENT JOB 161954.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161955" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "627.
EVENT JOB 161955.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161713" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "628.
EVENT JOB 161713.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161808" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "629.
EVENT JOB 161808.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161767" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "630.
EVENT JOB 161767.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161768" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "631.
EVENT JOB 161768.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161769" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "633.
EVENT JOB 161769.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161770" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "634.
EVENT JOB 161770.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161950" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "635.
EVENT JOB 161950.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161951" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "636.
EVENT JOB 161951.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161800" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "637.
EVENT JOB 161800.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161801" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "638.
EVENT JOB 161801.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161802" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "639.
EVENT JOB 161802.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161843" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "640.
EVENT JOB 161843.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161897" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "641.
EVENT JOB 161897.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161898" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "642.
EVENT JOB 161898.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161956" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "643.
EVENT JOB 161956.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161957" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "644.
EVENT JOB 161957.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161852" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "645.
EVENT JOB 161852.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161869" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "646.
EVENT JOB 161869.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161850" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "647.
EVENT JOB 161850.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161854" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "648.
EVENT JOB 161854.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161905" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "649.
EVENT JOB 161905.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161906" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "650.
EVENT JOB 161906.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161847" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "651.
EVENT JOB 161847.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161849" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "652.
EVENT JOB 161849.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161901" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "654.
EVENT JOB 161901.1 USAGE" failed
01/06/2010 11:06:37|schedu|athena|E|could not find job "161902" in master list
01/06/2010 11:06:37|schedu|athena|E|callback function for event "655.
EVENT JOB 161902.1 USAGE" failed
01/06/2010 11:06:47|schedu|athena|E|could not find job "161874" in master list
01/06/2010 11:06:47|schedu|athena|E|callback function for event "656.
EVENT JOB 161874.1 USAGE" failed
01/06/2010 11:06:47|schedu|athena|E|could not find job "161875" in master list
01/06/2010 11:06:47|schedu|athena|E|callback function for event "657.
EVENT JOB 161875.1 USAGE" failed
01/06/2010 11:06:47|schedu|athena|E|could not find job "161899" in master list
01/06/2010 11:06:47|schedu|athena|E|callback function for event "662.
EVENT JOB 161899.1 USAGE" failed
01/06/2010 11:06:47|schedu|athena|E|could not find job "161900" in master list
01/06/2010 11:06:47|schedu|athena|E|callback function for event "663.
EVENT JOB 161900.1 USAGE" failed
01/06/2010 11:06:47|schedu|athena|E|could not find job "161952" in master list
01/06/2010 11:06:47|schedu|athena|E|callback function for event "664.
EVENT JOB 161952.1 USAGE" failed
01/06/2010 11:06:47|schedu|athena|E|could not find job "161953" in master list
01/06/2010 11:06:47|schedu|athena|E|callback function for event "665.
EVENT JOB 161953.1 USAGE" failed
01/06/2010 11:06:47|schedu|athena|E|could not find job "161799" in master list
01/06/2010 11:06:47|schedu|athena|E|callback function for event "666.
EVENT JOB 161799.1 USAGE" failed
01/06/2010 11:06:47|schedu|athena|E|could not find job "161842" in master list
01/06/2010 11:06:47|schedu|athena|E|callback function for event "667.
EVENT JOB 161842.1 USAGE" failed
01/06/2010 11:06:47|schedu|athena|E|could not find job "161960" in master list
01/06/2010 11:06:47|schedu|athena|E|callback function for event "669.
EVENT JOB 161960.1 task 3.scout55 USAGE" failed
01/06/2010 11:06:47|schedu|athena|E|could not find job "161960" in master list
01/06/2010 11:06:47|schedu|athena|E|callback function for event "670.
EVENT JOB 161960.1 USAGE" failed
01/06/2010 11:06:57|schedu|athena|E|could not find job "161845" in master list
01/06/2010 11:06:57|schedu|athena|E|callback function for event "672.
EVENT JOB 161845.1 USAGE" failed
01/06/2010 11:06:57|schedu|athena|E|could not find job "161846" in master list
01/06/2010 11:06:57|schedu|athena|E|callback function for event "673.
EVENT JOB 161846.1 USAGE" failed
01/06/2010 11:06:57|schedu|athena|E|could not find job "161765" in master list
01/06/2010 11:06:57|schedu|athena|E|callback function for event "676.
EVENT JOB 161765.1 task 3.scout75 USAGE" failed
01/06/2010 11:06:57|schedu|athena|E|could not find job "161765" in master list
01/06/2010 11:06:57|schedu|athena|E|callback function for event "677.
EVENT JOB 161765.1 USAGE" failed
01/06/2010 11:06:57|schedu|athena|E|could not find job "161744" in master list
01/06/2010 11:06:57|schedu|athena|E|callback function for event "680.
EVENT JOB 161744.1 task 3.scout78 USAGE" failed
01/06/2010 11:06:57|schedu|athena|E|could not find job "161744" in master list
01/06/2010 11:06:57|schedu|athena|E|callback function for event "681.
EVENT JOB 161744.1 USAGE" failed
01/06/2010 11:07:07|schedu|athena|E|could not find job "161903" in master list
01/06/2010 11:07:07|schedu|athena|E|callback function for event "686.
EVENT JOB 161903.1 USAGE" failed
01/06/2010 11:07:07|schedu|athena|E|could not find job "161904" in master list
01/06/2010 11:07:07|schedu|athena|E|callback function for event "687.
EVENT JOB 161904.1 USAGE" failed
01/06/2010 11:07:07|schedu|athena|E|could not find job "161743" in master list
01/06/2010 11:07:07|schedu|athena|E|callback function for event "690.
EVENT JOB 161743.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161948" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "694.
EVENT JOB 161948.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161949" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "695.
EVENT JOB 161949.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161713" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "698.
EVENT JOB 161713.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161808" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "699.
EVENT JOB 161808.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161954" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "701.
EVENT JOB 161954.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161955" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "702.
EVENT JOB 161955.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161767" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "714.
EVENT JOB 161767.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161768" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "715.
EVENT JOB 161768.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161897" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "721.
EVENT JOB 161897.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161898" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "722.
EVENT JOB 161898.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161800" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "724.
EVENT JOB 161800.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161801" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "725.
EVENT JOB 161801.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161956" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "728.
EVENT JOB 161956.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161957" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "729.
EVENT JOB 161957.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161769" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "731.
EVENT JOB 161769.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161770" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "732.
EVENT JOB 161770.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161802" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "734.
EVENT JOB 161802.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161843" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "735.
EVENT JOB 161843.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161950" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "737.
EVENT JOB 161950.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161951" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "738.
EVENT JOB 161951.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161850" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "740.
EVENT JOB 161850.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161854" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "741.
EVENT JOB 161854.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161901" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "743.
EVENT JOB 161901.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161902" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "744.
EVENT JOB 161902.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161905" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "746.
EVENT JOB 161905.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161906" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "747.
EVENT JOB 161906.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161847" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "749.
EVENT JOB 161847.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161849" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "750.
EVENT JOB 161849.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161852" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "752.
EVENT JOB 161852.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161869" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "753.
EVENT JOB 161869.1 USAGE" failed
01/06/2010 11:07:17|schedu|athena|E|could not find job "161874" in master list
01/06/2010 11:07:17|schedu|athena|E|callback function for event "757.
EVENT JOB 161874.1 USAGE" failed
01/06/2010 11:07:27|schedu|athena|E|could not find job "161875" in master list
01/06/2010 11:07:27|schedu|athena|E|callback function for event "760.
EVENT JOB 161875.1 USAGE" failed
01/06/2010 11:07:27|schedu|athena|E|could not find job "161799" in master list
01/06/2010 11:07:27|schedu|athena|E|callback function for event "774.
EVENT JOB 161799.1 USAGE" failed
01/06/2010 11:07:27|schedu|athena|E|could not find job "161842" in master list
01/06/2010 11:07:27|schedu|athena|E|callback function for event "775.
EVENT JOB 161842.1 USAGE" failed
01/06/2010 11:07:27|schedu|athena|E|could not find job "161952" in master list
01/06/2010 11:07:27|schedu|athena|E|callback function for event "777.
EVENT JOB 161952.1 USAGE" failed
01/06/2010 11:07:27|schedu|athena|E|could not find job "161953" in master list
01/06/2010 11:07:27|schedu|athena|E|callback function for event "778.
EVENT JOB 161953.1 USAGE" failed
01/06/2010 11:07:27|schedu|athena|E|could not find job "161899" in master list
01/06/2010 11:07:27|schedu|athena|E|callback function for event "780.
EVENT JOB 161899.1 USAGE" failed
01/06/2010 11:07:27|schedu|athena|E|could not find job "161900" in master list
01/06/2010 11:07:27|schedu|athena|E|callback function for event "781.
EVENT JOB 161900.1 USAGE" failed
01/06/2010 11:07:27|schedu|athena|E|could not find job "161960" in master list
01/06/2010 11:07:27|schedu|athena|E|callback function for event "785.
EVENT JOB 161960.1 task 3.scout55 USAGE" failed
01/06/2010 11:07:27|schedu|athena|E|could not find job "161960" in master list
01/06/2010 11:07:27|schedu|athena|E|callback function for event "786.
EVENT JOB 161960.1 USAGE" failed
01/06/2010 11:07:37|schedu|athena|E|could not find job "161845" in master list
01/06/2010 11:07:37|schedu|athena|E|callback function for event "792.
EVENT JOB 161845.1 USAGE" failed
01/06/2010 11:07:37|schedu|athena|E|could not find job "161846" in master list
01/06/2010 11:07:37|schedu|athena|E|callback function for event "793.
EVENT JOB 161846.1 USAGE" failed
01/06/2010 11:07:37|schedu|athena|E|could not find job "161765" in master list
01/06/2010 11:07:37|schedu|athena|E|callback function for event "797.
EVENT JOB 161765.1 task 3.scout75 USAGE" failed
01/06/2010 11:07:37|schedu|athena|E|could not find job "161765" in master list
01/06/2010 11:07:37|schedu|athena|E|callback function for event "798.
EVENT JOB 161765.1 USAGE" failed
01/06/2010 11:07:37|schedu|athena|E|could not find job "161744" in master list
01/06/2010 11:07:37|schedu|athena|E|callback function for event "806.
EVENT JOB 161744.1 task 3.scout78 USAGE" failed
01/06/2010 11:07:37|schedu|athena|E|could not find job "161744" in master list
01/06/2010 11:07:37|schedu|athena|E|callback function for event "807.
EVENT JOB 161744.1 USAGE" failed
01/06/2010 11:07:47|schedu|athena|E|could not find job "161903" in master list
01/06/2010 11:07:47|schedu|athena|E|callback function for event "853.
EVENT JOB 161903.1 USAGE" failed
01/06/2010 11:07:47|schedu|athena|E|could not find job "161904" in master list
01/06/2010 11:07:47|schedu|athena|E|callback function for event "854.
EVENT JOB 161904.1 USAGE" failed
01/06/2010 11:07:47|schedu|athena|E|could not find job "161743" in master list
01/06/2010 11:07:47|schedu|athena|E|callback function for event "860.
EVENT JOB 161743.1 USAGE" failed
01/06/2010 11:07:58|schedu|athena|E|could not find job "161948" in master list
01/06/2010 11:07:58|schedu|athena|E|callback function for event "863.
EVENT JOB 161948.1 USAGE" failed
01/06/2010 11:07:59|schedu|athena|E|could not find job "161949" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "864.
EVENT JOB 161949.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161713" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "866.
EVENT JOB 161713.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161808" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "867.
EVENT JOB 161808.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161954" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "869.
EVENT JOB 161954.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161955" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "870.
EVENT JOB 161955.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161767" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "880.
EVENT JOB 161767.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161768" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "881.
EVENT JOB 161768.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161874" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "885.
EVENT JOB 161874.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161897" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "887.
EVENT JOB 161897.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161898" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "888.
EVENT JOB 161898.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161901" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "890.
EVENT JOB 161901.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161902" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "891.
EVENT JOB 161902.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161800" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "893.
EVENT JOB 161800.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161801" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "894.
EVENT JOB 161801.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161956" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "896.
EVENT JOB 161956.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161957" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "897.
EVENT JOB 161957.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161769" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "900.
EVENT JOB 161769.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161770" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "901.
EVENT JOB 161770.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161802" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "903.
EVENT JOB 161802.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161843" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "904.
EVENT JOB 161843.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161950" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "906.
EVENT JOB 161950.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161951" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "907.
EVENT JOB 161951.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161850" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "909.
EVENT JOB 161850.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161854" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "910.
EVENT JOB 161854.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161905" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "912.
EVENT JOB 161905.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161906" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "913.
EVENT JOB 161906.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161847" in master list
01/06/2010 11:08:00|schedu|athena|E|callback function for event "915.
EVENT JOB 161847.1 USAGE" failed
01/06/2010 11:08:00|schedu|athena|E|could not find job "161849" in master list
01/06/2010 11:08:01|schedu|athena|E|callback function for event "916.
EVENT JOB 161849.1 USAGE" failed
01/06/2010 11:08:01|schedu|athena|E|could not find job "161852" in master list
01/06/2010 11:08:01|schedu|athena|E|callback function for event "918.
EVENT JOB 161852.1 USAGE" failed
01/06/2010 11:08:01|schedu|athena|E|could not find job "161869" in master list
01/06/2010 11:08:01|schedu|athena|E|callback function for event "919.
EVENT JOB 161869.1 USAGE" failed
01/06/2010 11:08:07|schedu|athena|E|could not find job "161875" in master list
01/06/2010 11:08:07|schedu|athena|E|callback function for event "927.
EVENT JOB 161875.1 USAGE" failed
01/06/2010 11:08:07|schedu|athena|E|could not find job "161799" in master list
01/06/2010 11:08:07|schedu|athena|E|callback function for event "937.
EVENT JOB 161799.1 USAGE" failed
01/06/2010 11:08:07|schedu|athena|E|could not find job "161842" in master list
01/06/2010 11:08:07|schedu|athena|E|callback function for event "938.
EVENT JOB 161842.1 USAGE" failed
01/06/2010 11:08:07|schedu|athena|E|could not find job "161952" in master list
01/06/2010 11:08:07|schedu|athena|E|callback function for event "940.
EVENT JOB 161952.1 USAGE" failed
01/06/2010 11:08:07|schedu|athena|E|could not find job "161953" in master list
01/06/2010 11:08:07|schedu|athena|E|callback function for event "941.
EVENT JOB 161953.1 USAGE" failed
01/06/2010 11:08:07|schedu|athena|E|could not find job "161899" in master list
01/06/2010 11:08:07|schedu|athena|E|callback function for event "943.
EVENT JOB 161899.1 USAGE" failed
01/06/2010 11:08:07|schedu|athena|E|could not find job "161900" in master list
01/06/2010 11:08:07|schedu|athena|E|callback function for event "944.
EVENT JOB 161900.1 USAGE" failed
01/06/2010 11:08:07|schedu|athena|E|could not find job "161960" in master list
01/06/2010 11:08:07|schedu|athena|E|callback function for event "948.
EVENT JOB 161960.1 task 3.scout55 USAGE" failed
01/06/2010 11:08:07|schedu|athena|E|could not find job "161960" in master list
01/06/2010 11:08:07|schedu|athena|E|callback function for event "949.
EVENT JOB 161960.1 USAGE" failed
01/06/2010 11:08:17|schedu|athena|E|could not find job "161845" in master list
01/06/2010 11:08:17|schedu|athena|E|callback function for event "957.
EVENT JOB 161845.1 USAGE" failed
01/06/2010 11:08:17|schedu|athena|E|could not find job "161846" in master list
01/06/2010 11:08:17|schedu|athena|E|callback function for event "958.
EVENT JOB 161846.1 USAGE" failed
01/06/2010 11:08:17|schedu|athena|E|could not find job "161765" in master list
01/06/2010 11:08:17|schedu|athena|E|callback function for event "960.
EVENT JOB 161765.1 task 3.scout75 USAGE" failed
01/06/2010 11:08:17|schedu|athena|E|could not find job "161765" in master list
01/06/2010 11:08:17|schedu|athena|E|callback function for event "961.
EVENT JOB 161765.1 USAGE" failed
01/06/2010 11:08:17|schedu|athena|E|could not find job "161744" in master list
01/06/2010 11:08:17|schedu|athena|E|callback function for event "966.
EVENT JOB 161744.1 task 3.scout78 USAGE" failed
01/06/2010 11:08:17|schedu|athena|E|could not find job "161744" in master list
01/06/2010 11:08:17|schedu|athena|E|callback function for event "967.
EVENT JOB 161744.1 USAGE" failed
01/06/2010 11:08:27|schedu|athena|E|could not find job "161903" in master list
01/06/2010 11:08:27|schedu|athena|E|callback function for event "1018.
EVENT JOB 161903.1 USAGE" failed
01/06/2010 11:08:27|schedu|athena|E|could not find job "161904" in master list
01/06/2010 11:08:27|schedu|athena|E|callback function for event "1019.
EVENT JOB 161904.1 USAGE" failed
01/06/2010 11:08:27|schedu|athena|E|could not find job "161743" in master list
01/06/2010 11:08:27|schedu|athena|E|callback function for event "1021.
EVENT JOB 161743.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161948" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1027.
EVENT JOB 161948.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161949" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1028.
EVENT JOB 161949.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161713" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1031.
EVENT JOB 161713.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161808" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1032.
EVENT JOB 161808.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161954" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1034.
EVENT JOB 161954.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161955" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1035.
EVENT JOB 161955.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161767" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1045.
EVENT JOB 161767.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161768" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1046.
EVENT JOB 161768.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161901" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1051.
EVENT JOB 161901.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161902" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1052.
EVENT JOB 161902.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161956" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1054.
EVENT JOB 161956.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161957" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1055.
EVENT JOB 161957.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161769" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1057.
EVENT JOB 161769.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161770" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1058.
EVENT JOB 161770.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161852" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1060.
EVENT JOB 161852.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161869" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1061.
EVENT JOB 161869.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161802" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1063.
EVENT JOB 161802.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161843" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1064.
EVENT JOB 161843.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161950" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1066.
EVENT JOB 161950.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161951" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1067.
EVENT JOB 161951.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161850" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1069.
EVENT JOB 161850.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161854" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1070.
EVENT JOB 161854.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161897" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1072.
EVENT JOB 161897.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161898" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1073.
EVENT JOB 161898.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161800" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1075.
EVENT JOB 161800.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161801" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1076.
EVENT JOB 161801.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161847" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1078.
EVENT JOB 161847.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161849" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1079.
EVENT JOB 161849.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161905" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1081.
EVENT JOB 161905.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161906" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1082.
EVENT JOB 161906.1 USAGE" failed
01/06/2010 11:08:37|schedu|athena|E|could not find job "161874" in master list
01/06/2010 11:08:37|schedu|athena|E|callback function for event "1084.
EVENT JOB 161874.1 USAGE" failed

These are the jobs that are queued:

161658 0.00000 wesamRun   welshamy     qw    12/29/2009 06:40:53
                             1
 161668 0.00000 out.sh     ssandeep     qw    12/29/2009 18:30:37
                              1
 161975 0.00000 kbhrex     debabani     qw    01/05/2010 10:55:10
                              8
 161996 0.00000 g-T        ttduyle      qw    01/05/2010 11:49:36
                              8
 162123 0.00000 fliv.T900  jiangao      qw    01/06/2010 08:54:44
                              8
 162204 0.00000 wa23       jiangao      qw    01/06/2010 10:39:14
                              8
 162225 0.00000 --titans-m mozes        qw    01/06/2010 11:06:35
                              1
 162226 0.00000 --scouts-m mozes        qw    01/06/2010 11:06:36
                              1
 162227 0.00000 --fiends-m mozes        qw    01/06/2010 11:06:38
                              1
 162228 0.00000 --brutes-m mozes        qw    01/06/2010 11:06:40
                              1
 162229 0.00000 --rogues-m mozes        qw    01/06/2010 11:06:41
                              1
 162230 0.00000 --titans-m mozes        qw    01/06/2010 11:06:43
                              1
 162231 0.00000 --scouts-m mozes        qw    01/06/2010 11:06:47
                              1
 162232 0.00000 --fiends-m mozes        qw    01/06/2010 11:06:49
                              1
 162233 0.00000 --brutes-m mozes        qw    01/06/2010 11:06:51
                              1
 162234 0.00000 --rogues-m mozes        qw    01/06/2010 11:06:53
                              1
 162235 0.00000 --titans-m mozes        qw    01/06/2010 11:06:57
                              1
 162236 0.00000 --scouts-m mozes        qw    01/06/2010 11:06:59
                              1
 162237 0.00000 --fiends-m mozes        qw    01/06/2010 11:07:01
                              1
 162238 0.00000 --brutes-m mozes        qw    01/06/2010 11:07:02
                              1
 162239 0.00000 --rogues-m mozes        qw    01/06/2010 11:07:04
                              1
 162240 0.00000 --titans-m mozes        qw    01/06/2010 11:07:09
                              1
 162241 0.00000 --scouts-m mozes        qw    01/06/2010 11:07:10
                              1
 162242 0.00000 --fiends-m mozes        qw    01/06/2010 11:07:12
                              1
 162243 0.00000 --brutes-m mozes        qw    01/06/2010 11:07:14
                              1
 162244 0.00000 --rogues-m mozes        qw    01/06/2010 11:07:15
                              1

qstat -j gives no output (in reference to why a job won't start).

Attached is the debug output from SGE qmaster

If anyone has any thoughts it would be greatly appreciated.

--
Adam Tygart
Beocat Sysadmin

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=236869

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

######################################################################
ISS has detected a compressed file attached to this message.
Please note that compressed files can be used to spread computer viruses.
If you were not expecting this file you should not open the attachment
even if you know that the sender is genuine.

ISS Helpdesk
helpdesk at leeds.ac.uk
+44 113 343 3333
######################################################################




    [ Part 2, "Debug_Output_L2.tar.gz"  Application/X-GZIP (Name: ]
    [ "Debug_Output_L2.tar.gz") 2.9 MB. ]
    [ Unable to print this part. ]



More information about the gridengine-users mailing list