[GE users] Job submission speed through DRMAA api.

Kevin Ruland kruland at ku.edu
Fri Jan 21 16:07:01 GMT 2005


    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]


Please take these numbers as approximate.  They are from a single run, 
and include some console io, etc.  This is a very rough breakdown of the 
times:

drmaa_init = 2 sec
drmaa_allocate_job = 0 sec (to 4 places)
drmaa_set_attribute ( script name ) = 0.04 sec
drmaa_run_job = 0.04 sec
drmaa_synchronize == not reporting since it doesn't mean much.
drmaa_exit = 1.0 sec.

Andreas is correct in that the drmaa_init time is negligable if 
amortized over multiple jobs.  But, suppose I know before hand that I 
don't need to synchronize the jobs, would there be some way to forego 
the init?

Thanks much for looking into this.

Kevin

Andreas Haas wrote:

>Irrespective of possible inefficiencies in implementation you
>could add in case of DRMAA synchronization at session begin is
>always necessary to guarantee drmaa_wait() reliably works. For
>that reason a drmaa_init() delay is unavoidable even though
>I fully agree there should possibilities to lessen drmaa_init()
>delay.
>
>Though I don't know the overall set-up used in that case but as
>long as the python script issues a number of drmaa_run_job()
>calls the session init delay should be negligible. Also the
>drmaa_init() overhead performance-wise will yield a sensible
>profit as soon as DRMAA capability to synchronize with jobs
>finish is utilized compared to qstat/qacct be used for that
>purpose.
>
>HTH,
>Andreas
>
>On Fri, 21 Jan 2005, Daniel Templeton wrote:
>
>  
>
>>Duh.  I should have known that.  I wrote it. ;)
>>
>>The reason for the difference is that qsub is single-threaded unless a
>>-sync or -now option is used.  DRMAA is always mutli-threaded.  At the
>>creation of the second thread, the second thread requests a current list
>>of running jobs from the qmaster.  It and the main thread block until
>>that list is received.
>>
>>OK, so we've accounted for 2 of 3.5 seconds.  The other 1.5 must be in
>>the job templates and the drmaa_run_job() call.  The next interesting
>>test would be to time how long the drmaa_run_job() call takes.  I
>>suspect it's less than 0.1 seconds.
>>
>>Daniel
>>
>>    
>>


---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list