[GE users] Re: Matlab integration

Andreas Haas Andreas.Haas at Sun.COM
Tue Mar 14 18:34:41 GMT 2006


Hi Bernd,

I hope there is no reason not to talk about this in
public. Please find my replies below.

Regards,
Andreas

On Tue, 14 Mar 2006, Bernd Dammann wrote:

> Dear Andreas,
>
> > have you been able meanwhile to get Matlab integrated with N1GE?
> >
> Yes - we have made some progress to get the Matlab Distributed
> Computing Engine/Toolbox to interact with our SGE setup.

Good!

> We have added a new scheduler type to the toolbox, and the basic
> functionality is there now.  However, the student doing the work
> discovered some problems, some related to the DCT, some related to
> SGE.  Here are two questions regarding SGE:
>
> 1)
> When requesting the state of a specific job with a given jobID, it is not
> visible in the 'qstat -j jobID'. One instead has to issue 'qstat' and
> deduce the state from this list causing a much larger overhead.
>
> Right now we are using something like 'qstat | grep jobID', which
> gives a substantial overhead, because this is called frequently from
> Matlab to check the status of the tasks.

Generally there are means to lessen the overhead. I'm curious
however what kind of synchronization that is? Possibly it is
related to the "status files" you mention below?

> 2)
> qsub only supports a single range of task IDs of the format:
>
>     qsub -t 1-10
>
> What is missing is the option to define multiple ranges and or values of
> tasks IDs, e.g.
>
>     qsub -t 1-10,15,20-25
>
> The latter is important, because a user can delete tasks from the list
> before submitting a job to SGE.

Qsub as-is does not support multiple ranges. Though that limitation
is only due to client-side qsub parsing code, but nevertheless the
limitation exists. Possibly one can work around that by submitting
an array job for each range?

> We are still using SGE 5.3p5, so those problems might be solved in SGE
> 6 already?

No.

>
> Another problem (not for us, but maybe at other places) is the fact, that
> our current version only works if all the cluster nodes have access to a
> shared filesystems.

I understand this makes it somewhat uncomfortable. Nevertheless
having it working at first with shared file system constraint
only may be already usable for others.

> Is there anything in SGE, that allows to move input
> files to the execution node on startup and vice versa when a task ends?

Please see

   http://gridengine.sunsource.net/howto/filestaging/

resp. support for input/output/error file staging

   http://gridengine.sunsource.net/howto/filestaging/filestaging6.html

if 6.0 DRMAA is used for job submission.

> I am not even sure, if this is supported by the DCT, because there are
> some 'status files' that are checked on a regular basis during job
> execution.  So this is a question for MathWorks as well. :-)
>
> I tried to summarize some of our findings on the Wiki last week, but
> there seems to have been a problem with some google-analytics.com
> links, so the pages never finished loading.

Never had problems with GridWiki.

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list