[GE users] web interface to "qstat -xml" -- any java folk able to lend a hand? (need a cocoon generator for qstat)

Chris Dagdigian dag at sonsorol.org
Fri Nov 11 12:40:20 GMT 2005

Thanks for the heads up on XML::Smart - I'll have to check that out!   
Looks like it needs XML::Xpath installed if you want to query via  
that method though.

I started with pure perl but if you want  full Xpath queries and XSLT  
transforms  there are a ton of dependencies to manage and code to build.

Ultimately I  got the best/easiest results on several systems by  
building libxml2 and libxslt2 libraries from source and then  
installing perl XML::LibXML and XML::LIBXSLT both of which work fast  
and easy *once* all the dependency issues were sorted and the  
underlying C libraries are installed.

You still ended up with perl code that is not easily adjustable for  
the "quickly grab a bit of info from the XML data ..."


   use XML::LibXML;
   use XML::LibXSLT;

   $parser = XML::LibXML->new();
   $xslt   = XML::LibXSLT->new();

   my $source    = $parser->parse_file("$filename");
   my $style_doc = $parser->parse_file("$xslStylesheet");
   $stylesheet   = $xslt->parse_stylesheet($style_doc);
   $results      = $stylesheet->transform($source);
   $result       = $stylesheet->output_string($results);


When it came time to rework the qstat web application, a main focus  
was portability and non-root installation. The pure-perl+libxml2/ 
libxslt method has too many dependencies and takes a decent amount of  
time to install.

It is sort of sad that I'm using a full Java servlet container  
engine  just to fiddle with SGE XML but the upside is worth it -- the  
application server infrastructure does many "smart" things like  
caching and not redoing XSLT transformations if the source XML data  
has remained unchanged etc.   Longer term I think I'll be better off  
working within Apache Cocoon or even their "Apache Forrest"  
publishing framework.


On Nov 10, 2005, at 9:02 PM, Joe Landman wrote:

> Hmmm:
> [use the right Perl modules Luke...  XML::Smart in this case ]
>   Our 6.0u6 perl based parser fits into a single line, after we  
> grab the data.
> 	$qstat=`/opt/gridengine/bin/lx24-amd64/qstat -xml`;
> 	$xml    = XML::Smart->new($qstat);
> (no schema/DTD needed)
> then for example, iterating over all the jobs  ...
> 	foreach ($xml->{job_info}->{queue_info}->{job_list}('@') )
> 	  {
> 	   ...
> 	  }
> We are doing this with HTML::Mason (http://www.masonhq.com) so we  
> have Perl interspersed with the HTML.  If you would like to see  
> this code, please fire me a note.   We are using this as a way to  
> generate a table of all jobs in flight, in queue, and calculate for  
> every job in flight how much time it has spent computing.  With a  
> little extra effort, we can get process memory and other  
> utilization, but our customers haven't asked us for this.
> So far it is working nicely on a system with 144 cores and well  
> over 800 jobs queued and 140 in flight, with a 5 second refresh.    
> This system showed the need for pagination which I haven't done  
> (yet), but ...
> If you need to use the Java bits due to your coding requirements,  
> you would need to use the other bits as indicated.  I am sure they  
> work well.
> Joe
> Chris Dagdigian wrote:
>> I've junked most of the perl CGIs and scripts responsible for   
>> collecting and transforming SGE XML and have started re- 
>> developing  the little "xml-qstat" interface under the Apache  
>> Cocoon XML  publishing framework.
>> A work in progress demo is here:
>> http://dcore-amd.sonsorol.net:8888/xmlqstat/qstat.html
>> The output format above is biased towards the "show me  
>> everything"  side which works for small clusters. XML is  
>> dynamically collected  from SGE at periodic intervals via an  
>> external caching daemon.
>> For very large and active clusters where the "show me everything"   
>> model breaks down ( or breaks browsers! ) I've started work on  
>> some  "terse" XSL stylesheets. The effect of the "terse"  
>> stylesheet as  applied to canned XML taken from a largish cluster  
>> can be seen here:
>> http://dcore-amd.sonsorol.net:8888/xmlqstat/demo/qstat-terse.html
>> The reason for this email is that I need to start work on the  
>> more  dynamic stuff -- for instance the ability to click on a SGE  
>> JobID and  have rendered output from a dynamic "qstat -j <jobID>"  
>> call appear  within a DHTML DIV element.
>> The "proper" way to do this within Apache Cocoon would be to write  
>> a  java "Cocoon Generator" capable of calling out to  "qstat -xml"  
>> and  bringing the resulting XML directly into a Cocoon XML object  
>> for  transformation and XHTML serialization.
>> But ...
>> I'm a perl geek, not a java software engineer. I have absolutely  
>> no  idea how to go about writing a qstat XML generator for Apache  
>> Cocoon.  I'm pretty sure its not all that hard to do but I'm just  
>> not capable  of getting my head around the problem.
>> If I can't figure out the pure-java-Cocoon methods I'm just going  
>> to  write some tiny perl CGI's and have Cocoon fetch the dynamic  
>> XML via  http operations which is not as efficient (or elegant!)  
>> The perl  stuff I can spin out easily but I'd rather this be a  
>> 100% Cocoon  package if I can manage it.
>> Anyone willing to help me out with this? The xml-qstat stuff is  
>> open  source under a creative commons license. The Cocoon  
>> Generator  approach will make the resulting package far easier to  
>> setup and  distribute since external CGIs for XML data fetching  
>> won't be needed.
>> -Chris
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
>> For additional commands, e-mail: users-help at gridengine.sunsource.net
> -- 
> Joseph Landman, Ph.D
> Founder and CEO
> Scalable Informatics LLC,
> email: landman at scalableinformatics.com
> web  : http://www.scalableinformatics.com
> phone: +1 734 786 8423
> fax  : +1 734 786 8452
> cell : +1 734 612 4615
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
> For additional commands, e-mail: users-help at gridengine.sunsource.net

To unsubscribe, e-mail: users-unsubscribe at gridengine.sunsource.net
For additional commands, e-mail: users-help at gridengine.sunsource.net

More information about the gridengine-users mailing list