[GE dev] Another qacct performance improvement

rayson rayrayson at gmail.com
Wed May 13 03:29:12 BST 2009


A much smaller performance improvement than Roland's fix for issue
2510. The changes are:

1) In sge_read_rusage(), the static buffer szLine[] is removed.
Instead, the caller is responsible for allocating a buffer and passing
it to sge_read_rusage(). While it is not a big issue for now, this
(plus if we use strtok_r()) will enable multiple threads to parse the
accounting file in the future.

2) Further defer allocating and copying the fields (the sge_strdup()
block at the end of
sge_read_rusage()). Currently, when a line is of interest,
sge_read_rusage() calls sge_strdup() to duplicate the fields. And then
the strings are freed in the next iteration. This means parsing the
accounting file can cause a lot of allocate and free. The new way of
doing is to keep all the fields in the buffer passed in from the
caller.

I use the extreme case, which is when every line matches the condition
set by the user, the performance can improve by 30%:

Latest CVS:
% /usr/bin/time qacct
Total System Usage
    WALLCLOCK         UTIME         STIME           CPU
MEMORY                 IO                IOW
================================================================================================================
       215775     86310.000     21577.500      9000.000
9000.000              0.000              0.000

0.82user 0.00system 0:01.55elapsed 53%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+622minor)pagefaults 0swaps

With my change:
% /usr/bin/time qacct
Total System Usage
    WALLCLOCK         UTIME         STIME           CPU
MEMORY                 IO                IOW
================================================================================================================
       215775     86310.000     21577.500      9000.000
9000.000              0.000              0.000
0.64user 0.00system 0:01.16elapsed 56%CPU (0avgtext+0avgdata 0maxresident)k
0inputs+0outputs (0major+623minor)pagefaults 0swaps

I've uploaded my version of qacct.c in issue 2510.

Rayson

------------------------------------------------------
http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=39&dsMessageId=194862

To unsubscribe from this discussion, e-mail: [dev-unsubscribe at gridengine.sunsource.net].



More information about the gridengine-users mailing list