[GE users] a few questions...

paul_simpson paul at realisestudio.com
Tue Feb 2 15:40:39 GMT 2010


    [ The following text is in the "iso-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

thanks both of you for taking the time to reply at length.  it's good to hear i'm not totally alone.  i do know that axyz in toronto is using sge.  as are/were the people who made the ant bully in texas according to one of my friends who worked there for some time.

i also figured out about the retry switch - so all good with resubmitting tasks. :)

i think i may initially go with ganglia - which looks quite sweet.  and also some custom reporting/control guis too - which will be web based (ajax i would imagine).  so, quite some work to do - but looking forward to it.

it would be good for all the animation/fx sge users to start talking.  i would be up for open-sourcing my setup once i get it working/presentable. i think that much of what we do has a lot of overlap...

will report back once i've had some success (or failure).

again thanks and regards to all,

paul

On 29 January 2010 20:22, juby <justin.ottley at gmail.com<mailto:justin.ottley at gmail.com>> wrote:
Hey -

reuti wrote:
> Hi,
>
> Am 29.01.2010 um 17:10 schrieb juby:
>
>
>> Hi Paul,
>>
>> Las year I deployed an SGE 6.1u4 cluster for our renderfarm (animation
>> and visual effects studio). We run array jobs of a bunch of the usual
>> suspects - maya, houdini, prman, fusion, nuke, shake, etc.
>>
>> paul_simpson wrote:
>>
>>> hello all,
>>>
>>> this is my first post to this list.  i've just installed sge6.2u5
>>> - which i'll be using to calculate simulations, render animation
>>> and other animation related batch jobs.  so far so good - sge may
>>> be ugly on the (gui) surface - but i do appreciate the beauty and
>>> power beneath.  ie, the parallel environment is great for
>>> controlling cpu slot allocation, as are reporting options and user/
>>> queue config options.  the choice is almost daunting at first!
>>>
>>> i've a few simple questions if thats ok:
>>> 1/ is it possible to restart a task of an array job (ie, a render
>>> frame)?  ie, if a machine bugs out - i just want to reassign the
>>> failed frame again rather than rerun the complete job.  so far
>>> qresub only works at the job granularity from what i can see...
>>>
>>>
>> yes, take a look at qmod.
>> Particularly, qmod -rj (or -r or -rq).
>>
>
> yep, but only when the task is still in the system. Once it left it,
> it can't be rescheduled this way although other tasks of this array
> jobs are still in the system as a template.
>
Yes, thats a good point, thanks for clarifying that :)
Also, I suppose its worth mentioning that using reschedule_unknown is
another potential option.
> I think, you could enter an RFE to allow the -t option to qresub. If
> no -t option is specified, use the one which was used in the original
> qsub, otherwise the specified one. So, instead of specifying the the
> task id to the jobnumber, you would need to apply -t with the correct
> number.
>
>
>
>>> 2/ are there any mature gui's/front ends that i can leverage?  i
>>> see that there's a xml output and a java gui (xml-qstat).  before
>>> i install any more new software - it would be great if anyone
>>> could give me a quick heads up on status.
>>>
>>>
>> I wrote a custom job view GUI for our cluster, since production
>> required
>> more sophisticated (workflow specific) features than I found with qmon
>> or xml-qstat.
>> Id be interested to hear about anything you find though!
>>
>
> Which information do you need from the gui's/front ends in detail?
>
> -- Reuti
>
>
Well IMHO its a matter of both the information users need and the
presentation of the information..

Information wise, the gui shows workflow-specific information that jobs
carry such as "shot id", "job type", "frame range", "order", "scene",
"task", (etc) that are extracted from the job and presented in columns,
then there is progress information that is interpreted for job progress
bars, there is other frame information that is interpreted to show what
frame range a job (or job-task) is running, then there is stuff like
queue time, run time, priority (a workflow specific interpretation of
the qsub -p value).. There is colour-coding for job states (running,
rerunning, queued, delayed, error, killed, completed, etc)..

Also a very important part of the gui is transparent use of the database
(that is used behind ARco), for example for job history (jobs
themselves, finding paths to logs, job run times, etc) and user history;
another critical part is display of job logs; we have a relatively
sophisticated system of logging that does annotation on the job output,
which allows detection of known problems in a render and differentition
of "important" output vs "debug" output vs <whatever> output... so via
the gui, the job logs are filterable, paginated, known errors are
highlighted, can be streamed in real-time as the job is running (w/o NFS
usage).. these features are very useful for debugging renders (but again
are site specific).

Presentation wise, not sure i can describe in detail at the moment, but
there are some sophisticated widgets such as a separate torrent-style
progress summary for the whole array job (mouse sensitive.. its
pretty... artist like it..). Without something like a screenshot it
might not be so easy to identify what is going on vs something like
xml-qstat, but i think its hard to compare a workflow specific solution
(intended for a certain kind of user) vs. something intended to be
workflow independent (like xml-qstat, which is great in itself, and
qmon, which we still use heavily for maintainance and administration)
since they are kind of solving different problems and have a different
set of requirements.

-justin

>>> 3/ i've also read with great interest at ganglia and other
>>> monitoring software.  again - are there any up-to date summaries?
>>> i'm even thinking maybe i could use this as a first stab at job
>>> monitoring.. (as well as grid health monitor)
>>>
>>>
>> We use ARco here (although ganglia looks very nice), and we are pretty
>> happy with it so far. We use it for data and graphs for frame render
>> times, license usage, slot usage, blade health, server load, etc. They
>> are all handcrafted queries since we have a particular workflow for
>> our
>> jobs (just some simple tricks to get job data into the database) but
>> nothing really crazy. There are a couple UI things I would like to
>> improve, but all in all its been good.
>>
>>> 4/ are there many people using sge within the animation context?
>>> if so, are there any specific mailing lists that i should be on?
>>>
>> Ive seen one or two people on this list (besides you), but im not
>> sure..
>> if you do find any specific mailing lists id be interested to know!
>>
>>> i'm also keen to hear any good/bad experiences before i leap in
>>> with both feet first.
>>>
>>>
>> As for the rest of our installation, roughly:
>> - SGE 6.1u4
>> - BDB RPC
>> - ARCo
>> - local execd spool
>> ~ 800 CPUs up (linux, OS X, windows)
>>
>> checking some stats now, we run somewhere between 10,000 - 25,000
>> array
>> jobs a day.
>>
>> Our qmaster server is not the "latest" hardware by any means, and runs
>> at < 0.5 np_load_avg, same with our ARco server. Our BDB server
>> load is
>> negligible.
>> I chose BDB-RPC for qmaster failover, which has come in handy in the
>> past. Of course the BDB single point isnt ideal.. I had a prototype
>> solution for BDB replication/failover but never saw it through far
>> enough to know if it worked in practice (so for now I just protect the
>> bdb server from the outside world).
>>
>> I have a few small peeves about the XML output, for example 6.1u4
>> qstat
>> -j doesnt give you a state code for the job (r, Rr, qw, hqw, etc),
>> invalid XML in some edge cases (later versions are better afaik)...
>>
>> -justin
>>
>>
>>> many thanks in advance.  regards,
>>>
>>> paul
>>>
>>> ------------------------------------------------------
>>> http://gridengine.sunsource.net/ds/viewMessage.do?
>>> dsForumId=38&dsMessageId=241633
>>>
>>> To unsubscribe from this discussion, e-mail: [users-
>>> unsubscribe at gridengine.sunsource.net<mailto:unsubscribe at gridengine.sunsource.net>].
>>>
>>>
>>>
>>>
>> ------------------------------------------------------
>> http://gridengine.sunsource.net/ds/viewMessage.do?
>> dsForumId=38&dsMessageId=241756
>>
>> To unsubscribe from this discussion, e-mail: [users-
>> unsubscribe at gridengine.sunsource.net<mailto:unsubscribe at gridengine.sunsource.net>].
>>
>
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=241775
>
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net<mailto:users-unsubscribe at gridengine.sunsource.net>].
>
>
>

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=241793

To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net<mailto:users-unsubscribe at gridengine.sunsource.net>].




More information about the gridengine-users mailing list