[GE users] Comments please: Proposal for JobQueue::GridEngine and JobQueue::ForkingLayer modules

Alex Beamish talexb at gmail.com
Tue May 30 15:20:20 BST 2006

    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]


I am looking for feedback with regard to how to name two fraternal
Perl modules I have written that provide a unified API to either the
Sun Grid Engine or a forking layer (fork/wait subprocess mechanism). I
proposed the current internal names, JobQueue, JobQueue::GridEngine
and JobQueue::ForkingLayer on comp.lang.perl.modules ("New module
namespace", Wed, May 17), and got only a single message back from the

As I understand it, naming a Perl module is a subject not to be taken
lightly, since the community will have to live with the choice (right
or wrong) forever. Alternative choices for module names deal either
specifically with coderefs (Parallel::Queue), work within a larger
infrastructure (POE::Component::JobQueue), simply provide a queueing
subsystem (Queue::Base, Heap::Priority, IPC::DirQueue, Data::RefQueue,
Thread::Queue::Multiplex), or implement a specific job
(Net::Download::Queue). I still think JobQueue is the best name

Interestingly, the last named module (Net::Download::Queue) is similar
to one of the things that I do with my existing module -- I rsync
files to a remote server, something quite suited to parallel queues,
since many queues are better than one (think bank tellers) and because
of the time cost of the setup/teardown of an rsync command, providing
more efficient use of the available bandwidth.

Part of the existing man page is as follows: as I have not yet
received formal approval to submit this code to CPAN, I have redacted
some of the comments that don't relate to the JobQueue module.


    JobQueue - Job Queue API

        use JobQueue;
        $engine = JobQueue->new('SGE:/test/sge');
        defined ( $engine ) or
          die "Unable to access grid engine: $!";

        if ( $queue->isAcceptingJobs ) {

          if ( $queue->submitJob (
            {id=>$jobId, file=>$shellScript,
             stdout=>outputFile, stderr=>errorFile} ) ) {

            .. Job was submitted, all is well.

          } else {

            .. There was a problem, the job did not start.


        my @listOfFinishedJobs = $queue->retrieveFinishedJobs;
        foreach my $thisJob ( @listOfFinishedJobs ) {

          .. Handle this finished job.


    As a pre-cursor to making a multi-session rsyncer, this abstraction
    layer will allow us to make a generic job queue, for use with either the
    Sun Grid Engine or the new Forking Layer.

      Creates the JobQueue object and initializes it.

      Prods the JobQueue engine to make sure that it's alive. This may at
      some point in the future produce a report on what jobs are currently

      This is a test to see whether the JobQueue is accepting jobs.

    submitJob ( { id => [job identifier], directory => [working directory],
    file => [shell script], stdout => [stdout file], stderr => [stderr
file]    )
      Submit a job to the JobQueue. Arguments are mandatory. The arguments

        Job id, as supplied by the caller

        Directory where the files (below) are located, and where the job

        Shell script containing the commands to be run.

        File designated for stdout stream; if not specified, defaults to

        File designated for stderr stream; if not specified, defaults to

      The method returns the run identifier or undef if the job submission

      Return an array of finished jobs. This array may be empty is there are
      no finished jobs.

      The array elements are in the following form:

        { jobId => [job identifier], runId => [run identifier],
          directory => [working directory], status => [return status],
          response => [response] }


      This terminates the JobQueue.

      At this point there is no provision for terminating the child
      processes in any kind of safe way; however, if (for example) an rsync
      process is KILLed, it will return and this bad status should be
      returned [redacted]. Testing will prove that this works. Or not.

    (c) 2005, 2006 by Tata Infotech Ltd.
    (c) 2006 by Tata Consultancy Services Ltd.


I look forward to your feedback. Thanks!

Alex Beamish (talexb)
Toronto, Ontario

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list