[GE users] Comments please: Proposal for JobQueue::GridEngine and JobQueue::ForkingLayer modules

Joe Landman landman at scalableinformatics.com
Tue May 30 15:34:07 BST 2006


    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Hi Alex

Alex Beamish wrote:

[...]

> As I understand it, naming a Perl module is a subject not to be taken
> lightly, since the community will have to live with the choice (right
> or wrong) forever.

Not true.  You can change the name by creating a new module in that 
name, submitting it, and then deleting the old one.  I have done this, 
as have many others.

What they don't want is yet another top level module namespace.  I might 
suggest you look at the Schedule:: name space.  Tim Harsch has his 
Schedule::DRMAAc there.  I might suggest Schedule::GridEngine as the top 
branch of your work.

> Alternative choices for module names deal either
> specifically with coderefs (Parallel::Queue), work within a larger
> infrastructure (POE::Component::JobQueue), simply provide a queueing
> subsystem (Queue::Base, Heap::Priority, IPC::DirQueue, Data::RefQueue,
> Thread::Queue::Multiplex), or implement a specific job
> (Net::Download::Queue). I still think JobQueue is the best name
> choice.
> 
> Interestingly, the last named module (Net::Download::Queue) is similar
> to one of the things that I do with my existing module -- I rsync
> files to a remote server, something quite suited to parallel queues,
> since many queues are better than one (think bank tellers) and because
> of the time cost of the setup/teardown of an rsync command, providing
> more efficient use of the available bandwidth.
> 
> 
> Part of the existing man page is as follows: as I have not yet
> received formal approval to submit this code to CPAN, I have redacted
> some of the comments that don't relate to the JobQueue module.
> 
> 
> ----------------------------------
> 
> NAME
>    JobQueue - Job Queue API
> 
> SYNPOSIS
>        use JobQueue;
>        $engine = JobQueue->new('SGE:/test/sge');
>        defined ( $engine ) or
>          die "Unable to access grid engine: $!";
> 
>        if ( $queue->isAcceptingJobs ) {
> 
>          if ( $queue->submitJob (
>            {id=>$jobId, file=>$shellScript,
>             stdout=>outputFile, stderr=>errorFile} ) ) {
> 
>            .. Job was submitted, all is well.
> 
>          } else {
> 
>            .. There was a problem, the job did not start.
> 
>          }
>        }
> 
>        my @listOfFinishedJobs = $queue->retrieveFinishedJobs;
>        foreach my $thisJob ( @listOfFinishedJobs ) {
> 
>          .. Handle this finished job.
> 
>        }
> 
> DESCRIPTION
>    As a pre-cursor to making a multi-session rsyncer, this abstraction
>    layer will allow us to make a generic job queue, for use with either the
>    Sun Grid Engine or the new Forking Layer.
> 
> METHODS
>    new
>      Creates the JobQueue object and initializes it.
> 
>    status
>      Prods the JobQueue engine to make sure that it's alive. This may at
>      some point in the future produce a report on what jobs are currently
>      running.
> 
>    isAcceptingJobs
>      This is a test to see whether the JobQueue is accepting jobs.
> 
>    submitJob ( { id => [job identifier], directory => [working directory],
>    file => [shell script], stdout => [stdout file], stderr => [stderr
> file]    )
>      Submit a job to the JobQueue. Arguments are mandatory. The arguments
>      are
> 
>      id
>        Job id, as supplied by the caller
> 
>      directory
>        Directory where the files (below) are located, and where the job
>        runs.
> 
>      file
>        Shell script containing the commands to be run.
> 
>      stdout
>        File designated for stdout stream; if not specified, defaults to
>        stdout.qsub.
> 
>      stderr
>        File designated for stderr stream; if not specified, defaults to
>        stderr.qsub.
> 
>      The method returns the run identifier or undef if the job submission
>      failed.
> 
>    retrieveFinishedJobs
>      Return an array of finished jobs. This array may be empty is there are
>      no finished jobs.
> 
>      The array elements are in the following form:
> 
>        { jobId => [job identifier], runId => [run identifier],
>          directory => [working directory], status => [return status],
>          response => [response] }
> 
>      [redacted]
> 
>    term
>      This terminates the JobQueue.
> 
>      At this point there is no provision for terminating the child
>      processes in any kind of safe way; however, if (for example) an rsync
>      process is KILLed, it will return and this bad status should be
>      returned [redacted]. Testing will prove that this works. Or not.
> 
> Copyright
>    (c) 2005, 2006 by Tata Infotech Ltd.
>    (c) 2006 by Tata Consultancy Services Ltd.
> 
> ----------------------------------
> 
> I look forward to your feedback. Thanks!
> 

-- 
Joseph Landman, Ph.D
Founder and CEO
Scalable Informatics LLC,
email: landman at scalableinformatics.com
web  : http://www.scalableinformatics.com
phone: +1 734 786 8423
fax  : +1 734 786 8452
cell : +1 734 612 4615

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list