[GE users] inserting large accounting file into ARCo database

mhanby mhanby at uab.edu
Wed Apr 14 16:45:13 BST 2010


Bill,

Regarding " I thought that the reporting is generated from the accounting file. I will not have any reporting file until I install arco..."

The reporting file is not generated by ARCo, it is simply used as input by ARCo dbwriter. The reporting file is enabled in the global configuration by setting reporting=true :

$ qconf -sconf|grep reporting_params
reporting_params             accounting=true reporting=true \

FYI, the reporting file grows much faster than the accounting file. For example, our accounting file is 360M and the corresponding reporting file is 4G.

I believe that dbwriter will erase entries from the reporting files as they are added to the database. This could be good or bad, depending on your view point. If you need to keep the file intact, for use with other scripts, an archiving of the file prior to executing dbwriter should work.

Mike

From: Jana.Olivova at sun.com [mailto:Jana.Olivova at sun.com]
Sent: Thursday, April 08, 2010 3:58 AM
To: users at gridengine.sunsource.net
Cc: Daniel Templeton
Subject: Re: [GE users] inserting large accounting file into ARCo database

Here is copy of the email in regards of how to change the script: http://arco.sunsource.net/arco_faq.html.

Yes, the script is old, however it is very simple script and since the format of the accounting file or the reporting file has not changed until 6.2, it can be used without change for 6.1 ARCo /6.1 accounting file. As I said it would only need to be changed if you would want to import 6.1 accounting file into 6.2 ARCo.

It would have to be changed as follows to add the two missing fields, that are not contained in 6.1 accounting file but are required by 6.2 reporting file format:

cat $ACCT_FILE | awk -F: '{ if( substr($0,0,1) == "#" ) print $0; else  printf ("%d:acct:%s:0:0\n",$9+1000,$0)}'


Jana

On 04/07/10 23:39, wagoodman wrote:

Thank you Daniel,



I guess I would have to start fresh, The users will still be able to use

qacct

(for the archived accounting file) and we will continue archive the

accounting

files. Thanks a lot for all your help.



Bill



-----Original Message-----

From: Daniel Templeton [mailto:daniel.templeton at oracle.com]

Sent: Wednesday, April 07, 2010 5:28 PM

To: Goodman, William

Cc: users

Subject: Re: [GE users] inserting large accounting file into ARCo

database



The accounting file is used by qacct.  The reporting file is used by

ARCo.  The information in the accounting file is a subset of what's in

the reporting file.  There is no tool to insert an accounting file into

the ARCo database.



Daniel



On 04/07/10 14:25, Goodman, William wrote:



O.k. I'm confused now, I thought that the reporting is generated from

the accounting file.

I will not have any reporting file until I install arco (which I



haven't



done yet), Does

SGE have a tool to dump the accounting data directly into the ARCo

database?



Bill



-----Original Message-----

From: Daniel Templeton [mailto:daniel.templeton at oracle.com]

Sent: Wednesday, April 07, 2010 5:16 PM

To: users

Cc: Goodman, William

Subject: Re: [GE users] inserting large accounting file into ARCo

database



Sadly, no.  The accounting file is only for qacct.  ARCo only reads



the



reporting file.  You'll need to either convert the accounting file

format into the reporting file format or write a utility to dump the

accounting data directly into the ARCo database.  I'd probably try to

former.  Check out the accounting and reporting man pages for format

details.



Daniel



On 04/07/10 14:11, wagoodman wrote:



Daniel,



The file is an archived accounting file. Under

$SGE_ROOT/default/common/accounting.

SGE is currently writing to that file. We archive the accounting



file(s)



when it reaches

100 MB (accounting.1.gz, accounting.2.gz and accounting.3.gz



etc..etc.



I



have

concatenated them to a file called accounting.orig that is 4 GB. So



what



I want to do

is "cat $SGE_ROOT/default/common/accounting accounting.orig>

accounting" then fire up

DB writer and I should have all the data collected in a calendar year



in



my ARCo database,

would that work?



Bill



-----Original Message-----

From: templedf [mailto:daniel.templeton at oracle.com]

Sent: Wednesday, April 07, 2010 4:22 PM

To: users at gridengine.sunsource.net<mailto:users at gridengine.sunsource.net>

Subject: Re: [GE users] inserting large accounting file into ARCo

database



When you say it's an accounting file, do you mean it's the file



called



"accounting", or is it the "reporting" file?



If the later, there's not much you need to do.  Just install ARCo and

fire up the DB writer.  It will slog through your data and dump it



into



the database.  If the former, you'll need to write a utility to



either



transform the accounting file into a reporting file or to add it to



the



DB directly.



Daniel



On 04/07/10 13:07, wgoodman wrote:



I have been testing the SGE ARCo database accounting and reporting



software for about 2 months, now I'm ready to bring it into my

production environment. However I want to keep all the data that I



have



collected for about a year. I have a 4 GB accounting flat file that I

want to incorporate into the ARCo database. So when I install ARCo, I

would have a calendar year. How do I import that 4 GB accounting file

into the ARCo  database?



Bill



------------------------------------------------------





http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessage



Id=252596



To unsubscribe from this discussion, e-mail:



[users-unsubscribe at gridengine.sunsource.net<mailto:users-unsubscribe at gridengine.sunsource.net>].



------------------------------------------------------





http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessage



Id=252597



To unsubscribe from this discussion, e-mail:

[users-unsubscribe at gridengine.sunsource.net<mailto:users-unsubscribe at gridengine.sunsource.net>].



------------------------------------------------------





http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessage



Id=252605



To unsubscribe from this discussion, e-mail:



[users-unsubscribe at gridengine.sunsource.net<mailto:users-unsubscribe at gridengine.sunsource.net>].







------------------------------------------------------

http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=252610



To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net<mailto:users-unsubscribe at gridengine.sunsource.net>].






More information about the gridengine-users mailing list