Opened 11 years ago

Last modified 9 years ago

#908 new defect

IZ606: MaxPendingJobsSLO produces needs for held pending jobs

Reported by: rhierlmeier Owned by:
Priority: normal Milestone:
Component: hedeby Version: 1.0u1
Severity: Keywords: Sun gridengine_adapter
Cc:

Description

[Imported from gridengine issuezilla http://gridengine.sunsource.net/issues/show_bug.cgi?id=606]

        Issue #:      606                      Platform:     Sun      Reporter: rhierlmeier (rhierlmeier)
       Component:     hedeby                      OS:        All
     Subcomponent:    gridengine_adapter       Version:      1.0u1       CC:    None defined
        Status:       REOPENED                 Priority:     P3
      Resolution:                             Issue type:    DEFECT
                                           Target milestone: 1.0u5
      Assigned to:    afisch (afisch)
      QA Contact:     rhierlmeier
          URL:
       * Summary:     MaxPendingJobsSLO produces needs for held pending jobs
   Status whiteboard:
      Attachments:
                      Date/filename:                                   Description:              Submitted by:
                      Thu Dec 11 05:30:00 -0700 2008: review.txt.patch Review paper (text/plain) afisch


     Issue 606 blocks:
   Votes for issue 606:  10    Vote for this issue


   Opened: Thu Dec 4 02:22:00 -0700 2008 
------------------------


   Description

   GE allows the submission of jobs on hold. Scheduler will not schedule this jobs
   until the hold flag is deleted.  If a GE service has a MaxPendingJobsSLO and
   there are held jobs in the GE cluster the MaxPendingJobsSLO produces resource
   requests.  However the jobs are not scheduled and after reaching the
   maxWaitTimeForJobs timeout the  MaxPendingJobsSLO does no longer give usage to
   the new resources. They will be moved back into the spare_pool.

   This leads to the situation that the GE service has a constant need for new
   resources, however this resources are not needed. We have an oscillating
   system.

   Evaluation

   Pending or waiting pending jobs must not be considered for the calculation
   of needs by the MaxPendingJobsSLO.

   Suggested Fix/Work Around

   There is not work around possible for this problem.

   Analysis

   The GE service runs periodically a qstat -f over jgdi. The MaxPendingJobsSLO
   uses the result of the qstat to perform the SLO calculation.  In the current
   implementation it counts the number of pending jobs and if this number is
   higher than the configured limit it produces a resource request.  This
   algorithm does not distinguish between pending jobs waiting for being scheduled
   or pending jobs which are on hold.

   The following snippet shows the implemented the algorithm:

   class MaxPendingJobsSLO ... {
       ...
       public SLOState update(SLOContext ctx) throws GrmException {
           ...
           if (jobFilter == null) {
               count = geCtx.getPendingJobs().size();
           } else {
               JobHardRequestFilterAdapter filterSource = new
   JobHardRequestFilterAdapter();
               for (JobSummary job: geCtx.getPendingJobs()) {
                   filterSource.setFilterObject(job);
                   if (jobFilter.matches(filterSource)) {
                       count++;
                   }
                   // If the SLOContext has been invalided => stop the SLO calcuation
                   ctx.validate();
                }
           }
       }
   }

   If the MaxPendingJobsSLO has no job filter it simple counts the number of
   pending jobs. If there is a job filter it counts the number of pending jobs
   matching the job filter. Unfortunately with job filter it is not possible to
   filter the state of a job.

   We can fix the problem in two ways:

   1. Hard code that jobs in hold or waiting state are not considered.

   2. Enhance the JobHardRequestFilterAdapter that filtering of the state of
      a job is possible. The default job filter would look as follows:

      <common:slo name="maxPendingJobs" ...>
          <ge_adapter:jobFilter> !(state matches ".*[hw].*") </ge_adapter:jobFilter>
          ...
      </common:slo>

   Solution 2 is more flexible, however it has some drawbacks:

   o users can override the job filter, if the state is not correctly filtered we
     will run into the same issue due to an configuration error.

   o If in the cluster a complex 'state' is defined we have a naming collision
     because the filter does not know should the information be taken from the
     complex 'state' or is the state of the jobs meant.


   How to test

   Test scenario:

   o Choose one GE service for the test
   o Shutdown all other services
   o Setup a MaxPendingJobsSLO for the GE service (max attribute should be 1)
   o submit one jobs with 'qsub -h' into the cluster
   o Check the the MaxPendingJobsSLO does not produce a resource request
     (with 'sdmadm sslo' or 'sdmadm srr')

   For automated testing enhance the junit test MaxPendingJobsSLOTest. Add a new
   test case which tests the described scenario.


   ATC: 0.5 PD
   ETC: 3PD
               ------- Additional comments from afisch Mon Dec 8 07:34:30 -0700 2008 -------
   afisch started this issue
               ------- Additional comments from afisch Mon Dec 8 07:45:04 -0700 2008 -------
   changed to afisch
               ------- Additional comments from afisch Mon Dec 8 07:55:35 -0700 2008 -------
   After discussion with Richard we agreed to implement the first solution ("Hard
   coded"). The second solution ("Enhance JobHardRequestFilterAdapter") describes
   additionally to the fix a new feature that can be addressed with a separate
   issue.
               ------- Additional comments from afisch Thu Dec 11 05:23:02 -0700 2008 -------
   Updated Wiki entries for MaxPendingSLO
               ------- Additional comments from afisch Thu Dec 11 05:24:23 -0700 2008 -------
   fixed
               ------- Additional comments from afisch Thu Dec 11 05:30:19 -0700 2008 -------
   Created an attachment (id=49)
   Review paper

               ------- Additional comments from rhierlmeier Wed Nov 25 07:17:06 -0700 2009 -------
   Milestone changed
               ------- Additional comments from rpetrovski Mon Apr 12 03:48:04 -0700 2010 -------
   The jobs that exit with code 100 are marked by sge as state E and kept in the
   queue. The MaxPendingJobsSLO counts such jobs as pending and produces a need for
   resources. As these jobs are not executed by sge regardless of the resource
   availability, we have an oscillating system again.

Attachments (1)

49 (2.4 KB) - added by trac 9 years ago.

Download all attachments as: .zip

Change History (1)

Changed 9 years ago by trac

Note: See TracTickets for help on using tickets.