[GE users] Resource Reservation Issue

[Bradford, Matthew]

>-----Original Message-----
>From: Reuti [mailto:reuti at staff.uni-marburg.de] 
>Sent: 16 September 2008 12:42
>To: users at gridengine.sunsource.net
>Subject: Re: [GE users] Resource Reservation Issue
>
>Hi Mat,
>
>Am 14.09.2008 um 13:01 schrieb Bradford, Matthew:
>
>> Reuti,
>>
>> Also noticed this effect with Resource Reservation:
>>
>> I have 2 nodes, and 2 queues. Each node has a parallel queue and a 
>> serial queue. The queues subordinate each other to prevent 
>the running 
>> of serial jobs and parallel jobs at the same time on the same node.
>>
>> If I submit a serial job to comp1 node and another serial 
>job to comp2 
>> node, they both happily start executing, and the parallel queues on 
>> both nodes are suspended due to subordination.
>>
>> I can then submit a parallel job, such as
>>
>> Qsub -R y -pe my_pe 2 mpi_job.sh
>>
>> If I look in the schedule file, I can see that the scheduler is not 
>> making any reservations, I'm assuming because the two 
>parallel queues 
>> are suspended, and therefore not considered.
>>
>> If I kill one of the serial jobs, then the queue state of the 
>> parallel.q on that node becomes available again, as the 
>subordinating 
>> queue doesn't have any jobs running in it.
>>
>> If I now look at the schedule file, I would expect the scheduler to 
>> reserve that node to be used for the parallel job. This doesn't 
>> however appear to be the case. There is no scheduling being observed 
>> in the schedule file.
>
>No reservation at all?

That's correct. I thought it would show that it has reserved the free
node, but it doesn't.

>
>> Is this because the scheduler will only start reserving nodes for a 
>> specific job if there are enough active (non-suspended) 
>queues at that 
>> moment in time that can meet the job requirements, and the 
>only reason 
>> the job can't run is due to other jobs already running in those 
>> queues?
>> The scheduler does not seem to make build up the reservation 
>pool for 
>> a job as slots become available, but will only activate the 
>> reservation when all slots are active. If this is the case, then it 
>> looks like subordination between queues causes problems.
>
>Yes, I noticed the same. But I have no solution for it. 
>Subordinated queues are ignored, and even a reservation may 
>become invalid when they get subordinated.

Isn't this a bit of a flaw if the normal usage of one part of SGE
(subordinated queues) prevents another part (Resource Reservation) from
functioning correctly.

Does anybody know if this is/will be fixed at any point?

Cheers,

Mat
>
>-- Reuti
>
>
>> If, as in the test case
>> described, there is another job in the pending queue requesting a 
>> single slot in the serial queue, it will be able to run as 
>SGE has not 
>> placed a reservation on the compute node. This means that it is 
>> possible that the parallel job may never get enough resource 
>available 
>> to start.
>>
>> Any thoughts?
>>
>> Cheers,
>>
>> Mat
>>
>>
>>> -----Original Message-----
>>> From: Bradford, Matthew [mailto:matthew.bradford at eds.com]
>>> Sent: 14 September 2008 11:22
>>> To: users at gridengine.sunsource.net
>>> Subject: RE: [GE users] Resource Reservation Issue
>>>
>>> Reuti,
>>>
>>> No, we used qalter once the jobs were running, we manually reduced 
>>> there h_rt time to see whether this changes  which nodes the 
>>> scheduler thinks will become available first, and therefore 
>use these 
>>> in the reservation for the reserving job.
>>>
>>> I think my understanding of the use of h_rt was wrong. If I now 
>>> understand correctly, it is part of the request stating 
>that the job 
>>> requires this much run time rather than a attribute of the job 
>>> itself.
>>> The scheduler will then use the h_rt value to select the 
>appropriate 
>>> queue. Once the job has started, I assume that altering this value 
>>> has no effect.
>>>
>>>
>>> Maybe that isn't the best way of testing this. We have tried a 
>>> similar test, with a cluster full of jobs, and a pending job 
>>> requesting resource reservation of 4 slots. Even killing 
>some of the 
>>> running jobs in the cluster to free up 4 slots, the scheduler still 
>>> doesn't alter the reservation to use them, and if jobs 
>lower down in 
>>> the pending queue are present, they can jump in and start using the 
>>> newly available slots.
>>>
>>> Basically, we have users complaining that their jobs, which are 
>>> sitting at the top of the pending queue, with reservations set, are 
>>> not the next jobs to execute.
>>>
>>> Any thoughts would be most helpful.
>>>
>>> Cheers,
>>>
>>> Mat
>>>
>>>
>>>> -----Original Message-----
>>>> From: Reuti [mailto:reuti at staff.uni-marburg.de]
>>>> Sent: 12 September 2008 23:02
>>>> To: users at gridengine.sunsource.net
>>>> Subject: Re: [GE users] Resource Reservation Issue
>>>>
>>>> Hi,
>>>>
>>>> Am 11.09.2008 um 13:21 schrieb Bradford, Matthew:
>>>>
>>>>> We are running SGE 6.0u8 and have a problem with how resource 
>>>>> reservation works.
>>>>>
>>>>> For example:
>>>>>
>>>>> We have a full cluster running mainly parallel jobs of
>>> various sizes
>>>>> from 1 node to 16 nodes.
>>>>> We allow only 2 jobs to be run with a Reserve flag on them.
>>>>> Users aren't specifying an h_rt, and the default runtime is
>>>> set to 480
>>>>> hours.
>>>>> Occasionally, we set important jobs with a reserve flag 
>to prevent 
>>>>> resource starvation, and we push the job to the top of 
>the pending 
>>>>> queue.
>>>>>
>>>>> What we appear to get when we switch on monitoring is the 
>scheduler 
>>>>> selecting the nodes for the reserved job, but then not
>>> amending that
>>>>> selection even when there are free nodes in the system.
>>>>>
>>>>> During testing of this we noticed no difference when we
>>>> explicitly set
>>>>> the h_rt to 480 hours for all jobs, and then reduce the h_rt for 
>>>>> specific jobs. We thought that the scheduler would 
>recalculate and 
>>>>> select nodes where it knows that jobs are nearly finished.
>>>>>
>>>> what do you mean by "reducing" - the waiting jobs? With 
>qalter while 
>>>> they are waiting?
>>>>
>>>> -- Reuti
>>>>
>>>> 
>--------------------------------------------------------------------
>>>> -
>>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>>> For additional commands, e-mail: 
>users-help at gridengine.sunsource.net
>>>>
>>>>
>>>
>>> 
>---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>>
>>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>> For additional commands, e-mail: users-help at gridengine.sunsource.net
>>
>
>
>---------------------------------------------------------------------
>To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>For additional commands, e-mail: users-help at gridengine.sunsource.net
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net