[GE users] Comments please: Proposal for JobQueue::GridEngine and JobQueue::ForkingLayer modules

Alex Beamish talexb at gmail.com
Tue May 30 15:20:20 BST 2006


    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Greetings,

I am looking for feedback with regard to how to name two fraternal
Perl modules I have written that provide a unified API to either the
Sun Grid Engine or a forking layer (fork/wait subprocess mechanism). I
proposed the current internal names, JobQueue, JobQueue::GridEngine
and JobQueue::ForkingLayer on comp.lang.perl.modules ("New module
namespace", Wed, May 17), and got only a single message back from the
list.

As I understand it, naming a Perl module is a subject not to be taken
lightly, since the community will have to live with the choice (right
or wrong) forever. Alternative choices for module names deal either
specifically with coderefs (Parallel::Queue), work within a larger
infrastructure (POE::Component::JobQueue), simply provide a queueing
subsystem (Queue::Base, Heap::Priority, IPC::DirQueue, Data::RefQueue,
Thread::Queue::Multiplex), or implement a specific job
(Net::Download::Queue). I still think JobQueue is the best name
choice.

Interestingly, the last named module (Net::Download::Queue) is similar
to one of the things that I do with my existing module -- I rsync
files to a remote server, something quite suited to parallel queues,
since many queues are better than one (think bank tellers) and because
of the time cost of the setup/teardown of an rsync command, providing
more efficient use of the available bandwidth.


Part of the existing man page is as follows: as I have not yet
received formal approval to submit this code to CPAN, I have redacted
some of the comments that don't relate to the JobQueue module.


----------------------------------

NAME
    JobQueue - Job Queue API

SYNPOSIS
        use JobQueue;
        $engine = JobQueue->new('SGE:/test/sge');
        defined ( $engine ) or
          die "Unable to access grid engine: $!";

        if ( $queue->isAcceptingJobs ) {

          if ( $queue->submitJob (
            {id=>$jobId, file=>$shellScript,
             stdout=>outputFile, stderr=>errorFile} ) ) {

            .. Job was submitted, all is well.

          } else {

            .. There was a problem, the job did not start.

          }
        }

        my @listOfFinishedJobs = $queue->retrieveFinishedJobs;
        foreach my $thisJob ( @listOfFinishedJobs ) {

          .. Handle this finished job.

        }

DESCRIPTION
    As a pre-cursor to making a multi-session rsyncer, this abstraction
    layer will allow us to make a generic job queue, for use with either the
    Sun Grid Engine or the new Forking Layer.

METHODS
    new
      Creates the JobQueue object and initializes it.

    status
      Prods the JobQueue engine to make sure that it's alive. This may at
      some point in the future produce a report on what jobs are currently
      running.

    isAcceptingJobs
      This is a test to see whether the JobQueue is accepting jobs.

    submitJob ( { id => [job identifier], directory => [working directory],
    file => [shell script], stdout => [stdout file], stderr => [stderr
file]    )
      Submit a job to the JobQueue. Arguments are mandatory. The arguments
      are

      id
        Job id, as supplied by the caller

      directory
        Directory where the files (below) are located, and where the job
        runs.

      file
        Shell script containing the commands to be run.

      stdout
        File designated for stdout stream; if not specified, defaults to
        stdout.qsub.

      stderr
        File designated for stderr stream; if not specified, defaults to
        stderr.qsub.

      The method returns the run identifier or undef if the job submission
      failed.

    retrieveFinishedJobs
      Return an array of finished jobs. This array may be empty is there are
      no finished jobs.

      The array elements are in the following form:

        { jobId => [job identifier], runId => [run identifier],
          directory => [working directory], status => [return status],
          response => [response] }

      [redacted]

    term
      This terminates the JobQueue.

      At this point there is no provision for terminating the child
      processes in any kind of safe way; however, if (for example) an rsync
      process is KILLed, it will return and this bad status should be
      returned [redacted]. Testing will prove that this works. Or not.

Copyright
    (c) 2005, 2006 by Tata Infotech Ltd.
    (c) 2006 by Tata Consultancy Services Ltd.

----------------------------------

I look forward to your feedback. Thanks!

-- 
Alex Beamish (talexb)
Toronto, Ontario

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list