[GE users] xml-qacct?

Heywood, Todd heywood at cshl.edu
Thu Mar 1 16:26:18 GMT 2007


Hi Andreas,

That looks like it could be quite useful. Could a -d option analogous to
the -u option be added to produce a department table?

The hardest thing is making sense of "timesteps". They are in terms of
seconds since epoch format, right (as per the accounting man page),
right? Would it be possible to specify a time range like you do with
qaact, i.e. yymmddhh?

If I were to ask for the moon :-) I'd like to use the -h option with a
"total available hosts" argument, with the hosts table also producing a
sum of the time columns, and also a "cluster utilization" percentage
(i.e. a division of those column sums by total the available hosts times
the time range being analyzed). And if I were to ask for the stars, I
could combine this with either -u or -d to break down the total cluster
utilization by user or department. But this is way too much to ask! I
might try it myself over time, but I would need to learn some Ruby.

Thanks!

Todd

-----Original Message-----
From: Andreas.Haas at Sun.COM [mailto:Andreas.Haas at Sun.COM] 
Sent: Thursday, March 01, 2007 9:35 AM
To: users at gridengine.sunsource.net
Subject: Re: [GE users] xml-qacct?

Hi Todd,

I wrote for my own purposes a small ruby script which does lots of what 
you're asking:

    > ./analyze.rb
    usage: analyze.rb <options> accounting_file
            -help
            -r                                records table
            -u                                users table
            -h                                hosts table
            -q                                queues table
            -p                                projects table
            -c                                categories table
            -ts                               timesteps table
            -ts_c                             categories per timestep
            -ts_j                             jobs per timestep
            -t "first"|<first> "last"|<last>  full analysis, but print
these timesteps only

it gets you

(1) a summarized view of cpu/mem utilization (net+gross) for each of the
table views 
(2) selection of accounting entries with a time range
(3) time-step analysis on a job and job category basiss as required for
visualizing
     utilization over time

like this

    > analyze.rb -r $SGE_ROOT/default/common/accounting
    ###### Table with 503 job runs ######
    job      task  n  user    pending wallclock       cpu   maxvmem
maxrss submit       start        ended        category
    1        0     1  rd14130       6        47         0         0
0 1170834784   1170834790   1170834837   "NONE"
    6        0     1  aa11408      67       233         0   2555904
0 1172678589   1172678656   1172678889   "NONE"
    7        0     1  aa11408      67       233         0   2555904
0 1172678590   1172678657   1172678890   "NONE"
    8        1     1  aa11408       6        61         0   4636672
0 1172679130   1172679136   1172679197   "NONE"
    8        2     1  aa11408       6        66         0   4636672
0 1172679130   1172679136   1172679202   "NONE"

most recent version is availble under

    http://gridengine.sunsource.net/files/documents/7/82/analyze.rb.gz

please let me know it it fits your needs. Until now it's home is simply

    http://gridengine.sunsource.net/

but I could also move it to

    http://drmaa4ruby.sunsource.net/

which is a by far smaller OS-project.

Regards,
Andreas

On Wed, 28 Feb 2007, Heywood, Todd wrote:

> Hi,
>
>
>
> I have found the xml-qstat tool nice and useful. HA anyone tried this
> approach, or thought about it, for qacct? .
>
>
>
> Ideally this would allow you to specify a date range for the
accounting
> info, options to summarize by user and department, and be able to show
> cluster utilization (total wallclock time used divided by total
> processors times "real" elapsed time).
>
>
>
> I have found ARCo rather cumbersome to use, and the database and
> reporting file really eat up too much disk space for what ARCo
provides.
>
>
>
> Todd Heywood
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net




More information about the gridengine-users mailing list