[GE users] Accounting script

Olivier Blondel olivier.blondel at cea.fr
Fri May 11 09:41:30 BST 2007


    [ The following text is in the "ISO-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

Hello all,
I am new to this list, SGE, and cluster administration in general.
One off my first problem has been the fair share of the resources, and 
be able to prove each user that  he receive the  amount he deserve.
So I looked to accounting but 'qacct' doesn't fit precisely my needs and 
was not straight forward enough for average user. Then I found on 
grindengine.info website, a perl script from 'Joe at Scalabale 
Informatics' which was closer from what I was looking for. After some 
test I start to modify the script 'usage.pl' and then completely rewrite 
it. Since it has been a very useful starting point for me, I think it's, 
now, time to share it back to the community.

The output of my script 'userstat' (or maybe 'n1userstat' or 'quserstat' 
?) look similar to the one from usage.pl.
It differs on the implementation since it doesn't use the 'qacct' 
command through the 'run' module.
You can ask for last day|week|month|year|or any number of day stat.
You can sort any column on any order
You can ask group stat instead of userstat (since credits which paid the 
machine were group based)
You can ask for the stats of a queue or a pool of nodes.

Try 'userstat --help'

Rem: if you simlink the script with a name which contain 'group' (like 
'groupstat') you got the bygroup usage.

Hope this will help someone...


Olivier Blondel from CEA.



    [ Part 2: "Attached Text" ]

#!/usr/bin/perl
use strict;
use Getopt::Long;
use File::Basename;

use constant {
	sec	=> 1,
	min	=> 60,
	hour	=> 3600,
	day	=> 86400,
	week	=> 86400*7,
	month	=> 86400*365.25/12,
	year	=> 86400*365.25,
};
use constant {
  	limit	=> 38,
	qname	=> 0,
	hostname=> 1,
	group   => 2,
	owner   => 3,
	start_t	=> 9,
	end_t	=> 10,
	wall_t	=> 13,
	utime_t	=> 14,
	stime_t	=> 15,
	failed	=> 11,
	status	=> 12,
	cpu	=> 36,
	mem	=> 37,
};

my $acctfn	= "/opt/sge/default/common/accounting";	# path to acctounting file
my $bname	= basename($0);

my (@line,$hosts,$rc,$sorthelper);
my ($units,$totals,$users,$user,$col,$start,$bygrp,$label);
my ($queue,$grp,$unit,$last,$sort,$reverse,$bygroup,$byuser,$help);

$unit		= "hour"; 	# report usage in units of hours
$grp		= "x2200_ETH";
$col		= "wall";
$last		= "week";

$rc = GetOptions(
		"queue=s" => \$queue,
		"nodes=s" => \$grp,
		"unit=s"  => \$unit,
		"last=s"  => \$last,
		"sort=s"  => \$sort,
		"reverse" => \$reverse,
		"bygroup" => \$bygroup,
		"byuser"  => \$byuser,
		"help"    => \$help
		);

if ($help) {
  printf "Usage: $bname\n";
  printf "	--queue='queuename'\n";
  printf "	--nodes={x2200_ETH|x2200_IB|x2200|kalis|}\n";
  printf "	--unit={second|minute|hour|day|week|month|year}\n";
  printf "	--last={day|week|month|year|'number'}\n";  
  printf "	--sort={wall|user|system|cpu|mem}\n";
  printf "	--reverse\n";
  printf "	--bygroup\n";
  printf "ex: $bname --grp=kalis --units=min --sort=mem\n";
  exit;
}
$bygrp	= ($bname =~m/group/);
$bygrp	= 1		if ($bygroup);
$bygrp	= 0 		if ($byuser);
$col	= "wall"	if (($sort eq "wall") or ($sort eq "wallclock"));
$col	= "utime"	if (($sort eq "user") or ($sort eq "utime"));
$col	= "stime"	if (($sort eq "sys") or ($sort eq "system") or ($sort eq "stime"));
$col	= "cpu"		if (($sort eq "cpu") or ($sort eq "ctime"));
$col	= "mem"		if (($sort eq "mem") or ($sort eq "memory"));
$units	= sec		if ($unit eq "second");
$units	= min		if ($unit eq "minute");
$units	= hour		if ($unit eq "hour");
$units	= day		if ($unit eq "day");
$units	= week		if ($unit eq "week");
$units	= month		if ($unit eq "month");
$units	= year		if ($unit eq "year");
$start	= time - day	if ($last eq "day");
$start	= time - week	if ($last eq "week");
$start	= time - month	if ($last eq "month");
$start	= time - year	if ($last eq "year");
$start	= time - $last * day	if ($last !~/\D/);
$hosts	= "n(0|1[0-6])"		if (($grp eq "x2200_ETH") or ($grp eq "ETH"));
$hosts	= "n(1[7-9]|2|3)"	if (($grp eq "x2200_IB" ) or ($grp eq "IB" ));
$hosts	= "n[0-3]"		if (($grp eq "x2200") or ($grp eq "arrakis"));
$hosts	= "kali[1-4]"		if ($grp eq "kalis");


sub Asc{ $users->{$a}->{$col} <=> $users->{$b}->{$col} }
sub Dsc{ $users->{$b}->{$col} <=> $users->{$a}->{$col} }

if ($reverse) 	{$sorthelper = "Asc"}
else 		{$sorthelper = "Dsc"}


open IF, $acctfn or die "Impossible d'ouvrir: $acctfn";
#get total and per user usage
while (<IF>) {
  @line=split (/:/, $_, limit);
  if ($line[start_t] < $start) {next}
  if ( $queue and ($line[qname] !~ m/^$queue/) ) {next}
  if (!$queue and ($line[hostname] !~ m/$hosts/)) {next}
  
  $totals->{wall}		+= $line[wall_t];
  $totals->{utime}		+= $line[utime_t];
  $totals->{stime}		+= $line[stime_t];
  $totals->{cpu}		+= $line[cpu];
  $totals->{mem}		+= $line[mem];

  if ($bygrp)	{ $user = $line[group]; $label = "Group"}
  else		{ $user = $line[owner]; $label = "User" }
 
  $users->{$user}->{wall}	+= $line[wall_t];
  $users->{$user}->{utime}	+= $line[utime_t];
  $users->{$user}->{stime}	+= $line[stime_t];
  $users->{$user}->{cpu}	+= $line[cpu];
  $users->{$user}->{mem}	+= $line[mem]; 
}
close IF;

# now report totals ...
printf "Total usage in units of %s\n",$unit;
if ($queue)	{ printf "Last $last user stats for queue $queue\n" }
else		{ printf "Last $last user stats for nodes $grp\n" }
#printf "For the last %d days\n", (time - $start) / day;
printf "\twallclock  :\t%12.2f \n",$totals->{wall}/$units,;
printf "\tuser time  :\t%12.2f  [%.2f\%]\n",$totals->{utime}/$units,100*$totals->{utime}/$totals->{wall};
printf "\tsystem time:\t%12.2f  [%.2f\%]\n",$totals->{stime}/$units,100*$totals->{stime}/$totals->{wall};
printf "\tcpu time   :\t%12.2f  [%.2f\%]\n",$totals->{cpu} / $units,100*$totals->{cpu} / $totals->{wall};

printf "\nBy $label usage sorted by $col\n";
printf "%-12s%-4s%-10s%-10s%-10s%-10s%-16s\n",$label."name","%","wall","user","system","cpu","memory";
#and loop over users ...
foreach $user (sort $sorthelper (keys (%{$users}))) {  
  printf "%-12s%-4.0f%-10.0f%-10.0f%-10.0f%-10.0f%-16.0f\n",
  $user,
  100*$users->{$user}->{$col}/$totals->{$col},
  $users->{$user}->{wall}/$units,
  $users->{$user}->{utime}/$units,
  $users->{$user}->{stime}/$units,
  $users->{$user}->{cpu}/$units,
  $users->{$user}->{mem};
}



    [ Part 3: "Attached Text" ]

    [ The following text is in the "iso-8859-1" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some special characters may be displayed incorrectly. ]

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net



More information about the gridengine-users mailing list