[GE users] Job submission speed through DRMAA api.

Daniel Templeton Dan.Templeton at Sun.COM
Thu Jan 20 12:48:33 GMT 2005


Thank you for digging into this.  Clearly DRMAA job submission is a bit 
slower.  I think the next rational step is to profile your code sample 
to see where the majority of that time is spent.  I suspect it's spent 
playing with the job template.  The job template functions are... how 
shall I put this... not well optimized.  There's potentially a lot of 
extra memory allocations and string comparisons.  qsub builds the job 
structure directly, without using a job template.
At the moment, I'm rather busy on other projects.  There's a chance that 
I will have some time next week to dig into it.  If you'd rather 
instrument the code and dig deeper yourself, that would be a great help.

Daniel

Kevin Ruland wrote:

> 
> All,
> 
> Ron:  I understand the difference you are looking for.  I am calling 
> submission speed the time from drmaa_init, to the return from 
> drmaa_run_job.
> 
> Alas there appears to be a significant difference between qsub and drmaa 
> in job submission.  I haven't tried to dermine if there is a difference 
> in dispatching.
> 
>  From the command line I'm doing this:
> 
> qsub -p 0 -b yes -shell no -w e sleeper.sh
> 
> the command line time tool tells me it returns in < 0.1 second.
> 
> I then compiled howto2.c and its complete run too about 6 seconds.  
> Recall that howto.c does the following:
>  drmaa_init
>  drmaa_allocate_job_template
>  drmaa_set_attribute - remote command
>  drmaa_run_job
>  drmaa_delete_job_template
>  drmaa_exit
> 
> This takes between 4 and 6 seconds to execute.
> 
> I notices that some time happened between the printf with the jobid and 
> the program termination, so I cut out the delete_job_template and exit 
> calls and the time is at about 3.5seconds.
> 
> I've the copy of the c source which doesn't shut down drmaa gracefully.
> 
> Kevin
> 
> Kevin Ruland wrote:
> 
>>
>> All,
>>
>> My situation is a little more complex than I first let on. First I'm 
>> not using DRMAA directly, but rather through python swig wrappers.  So 
>> I need to figure out what I'm doing in there, recode in straight C and 
>> see if there is any significant difference.  So I don't have a nice 
>> test case yet.
>>
>> I also don't have hard numbers.  It's all a feeling now.
>>
>> Kevin
>>
>> Ron Chen wrote:
>>
>>> Kevin,
>>>
>>> Do you have a testcase?
>>>
>>> Also, is it the submission speed that is slow, or the
>>> job dispatch speed that is slow? They are very
>>> different, so may be you can tell us more about how
>>> you discovered the speed problem.
>>>
>>> -Ron
>>>
>>>
>>> --- Stephan Grell <stephan.grell at sun.com> wrote:
>>>  
>>
>>
>>
> 
> ------------------------------------------------------------------------
> 
> #include <stdio.h>
> #include "drmaa.h"
> 
> int main (int argc, char **argv) {
>   char error[DRMAA_ERROR_STRING_BUFFER];
>   int errnum = 0;
>   drmaa_job_template_t *jt = NULL;
>   
>   errnum = drmaa_init (NULL, error, DRMAA_ERROR_STRING_BUFFER);
> 
>   if (errnum != DRMAA_ERRNO_SUCCESS) {
>     fprintf (stderr, "Could not initialize the DRMAA library: %s\n", error);
>     return 1;
>   }
> 
>   errnum = drmaa_allocate_job_template (&jt, error, DRMAA_ERROR_STRING_BUFFER);
> 
>   if (errnum != DRMAA_ERRNO_SUCCESS) {
>     fprintf (stderr, "Could not create job template: %s\n", error);
>     return 1;
>   }
> 
>   errnum = drmaa_set_attribute (jt, DRMAA_REMOTE_COMMAND, "sleeper.sh",
> 				error, DRMAA_ERROR_STRING_BUFFER);
> 
>   if (errnum != DRMAA_ERRNO_SUCCESS) {
>     fprintf (stderr, "Could not set attribute \"%s\": %s\n",
> 	     DRMAA_REMOTE_COMMAND, error);
>     return 1;
>   }
> 
>   char jobid[DRMAA_JOBNAME_BUFFER];
> 
>   errnum = drmaa_run_job (jobid, DRMAA_JOBNAME_BUFFER, jt, error,
> 			  DRMAA_ERROR_STRING_BUFFER);
> 
>   if (errnum != DRMAA_ERRNO_SUCCESS) {
>     fprintf (stderr, "Could not submit job: %s\n", error);
>     return 1;
>   }
>   printf ("Your job has been submitted with id %s\n", jobid);
> 
>   return 0;
> }
> 
> 
> 
> ------------------------------------------------------------------------
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list