[GE users] Starting beta testing of HTTPi-xmlqstat.

jjh jjh at 42quarks.com
Mon Nov 9 04:25:46 GMT 2009

    [ The following text is in the "utf-8" character set. ]
    [ Your display is set for the "ISO-8859-10" character set.  ]
    [ Some characters may be displayed incorrectly. ]

Hi Mark,

Thanks for your help. I'm still a bit confused. I've got the latest
version of xml-qstat but I still have the same problem, only httpi is
built and no copy of xmlqstat is made into htdocs directory.  As far
as I can tell it's not giving any errors or anything, but just seems
to build a standard httpi and install it. Apologies in advance if I'm
doing something dumb.

I've appended the build output below.


On Mon, Nov 9, 2009 at 12:48 AM, olesen
<Mark.Olesen at emcontechnologies.com> wrote:
> > Thanks heaps for all the effort you put into this. I am having problems
> > getting going. I ran the make-httpi.sh, answered the questions etc. but it
> > only ever build a copy of httpi (not httpi-xmlqstat as it seems it should in
> > the instructions). I've tried running httpi but all it returns is 404s.
> I'll just give a brief answer, since my previous answer was too long and my stupid
> email web interface timed out on me :(
> I isolated the problem to the fact that something like '~/olesenm-xml-qstat-f12d13f' was not expanded,
> and thus in your final server it will never find the correct installation.
> (The path "~/foo/bar" doesn't resolve to a path in Perl).
> The quick fix (if you don't want to grab the last commit):
> Define the xml-qstat resource root with an absolute path, or use $ENV{HOME} instead of
> "~/", but don't ignore the warning if it claims it can't find that path.
> OR look for these lines in the configured httpi
> ################################################################################
> #
> # the base location of the xmlqstat distribution
> my $resourceRoot = "/export/home/mark/xml-qstat";
> And adjust accordingly.
> Thanks for the bug report!
> /mark
> This e-mail message and any attachments may contain legally privileged, confidential or proprietary Information, or information otherwise protected by law of EMCON Technologies, its affiliates, or third parties. This notice serves as marking of its "Confidential" status as defined in any confidentiality agreements concerning the sender and recipient. If you are not the intended recipient(s), or the employee or agent responsible for delivery of this message to the intended recipient(s), you are hereby notified that any dissemination, distribution or copying of this e-mail message is strictly prohibited.
> If you have received this message in error, please immediately notify the sender and delete this e-mail message from your computer.
> ------------------------------------------------------
> http://gridengine.sunsource.net/ds/viewMessage.do?dsForumId=38&dsMessageId=225662
> To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

qbi-xgrid-01:olesenm-xml-qstat-96646a3 jhunt$ sudo ./make-httpi.sh
fetch/unpack HTTPi httpi-1.6.2 source
downloading httpi-1.6.2.tar.gz from
--2009-11-09 14:14:38--  http://www.floodgap.com/httpi//httpi-1.6.2.tar.gz
Resolving www.floodgap.com...
Connecting to www.floodgap.com||:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 63141 (62K) [application/x-gzip]
Saving to: `httpi-1.6.2.tar.gz'
100%[======================================>] 63,141      30.0K/s   in 2.1s
2009-11-09 14:14:41 (30.0 KB/s) - `httpi-1.6.2.tar.gz' saved [63141/63141]
unpack build-download/httpi-1.6.2.tar.gz -> build-httpi
synchronize HTTPi source
    -> build-httpi

synchronize xmlqstat httpi-modules/
httpi-modules/custom-config.pl httpi-modules/userfunc.in
httpi-modules/modules.in httpi-modules/uservar.in
    -> build-httpi

configure HTTPi interactively
Configure (demonic) for HTTPi/1.6.2 (C)1998-2009 Cameron Kaiser/Contributors
This is the configure script for Demonic HTTPi, the daemonized HTTPi version.
* your Perl doesn't support fork() (currently that's mostly the
weird non-Unix ports)
* you don't have a good idea of how your system implements sockets
* you would prefer to have HTTPi run in inetd, xinetd or launchd (run
one of the other scripts)
* you need SSL (currently only configure.stunnel supports this)
You don't need Socket.pm, though it helps. You *will* need socket constants. I
will first ask your C compiler what they are. If you don't have one, or it's
dain bramaged, I'll ask Socket.pm, and if *that* doesn't work, then you'll
need to bail me out by supplying them manually. (Hope you keep SOMAXCONN and
friends handy if this becomes needed!) Fortunately, I'm really smart, so I'll
probably get them on the first try.
Press ENTER to continue or BREAK/CTRL-C to bail out: []:
Checking system defaults ...
Finding uname ... /usr/bin/uname
Smells like Darwin.
Finding your perl ... /usr/bin/perl
If you want to use this Perl to execute HTTPi, just hit RETURN/ENTER. However,
if you have another Perl executable you want to use instead, then enter it
here; it will be probed and then put in HTTPi's #! line.
...  [/usr/bin/perl]:
/usr/bin/perl selected.
Checking out your Perl ...
5.8.8 ... has this wine been corked?
Your Perl does have a POSIX.pm.
Can we fork()? ... yes
Finding hostname ... /bin/hostname
Can we use alarm()? ... yes
Can we use setruid()? ... no (setruid() not implemented at (eval 5) line 1.)
Cool, we made it that far.
Now you'll need to answer a few questions about your installation options
and some functions that need to be hard-coded into HTTPi. If you just hit
RETURN/ENTER with nothing entered, the default (in [ ]) will be selected.
Answering questions incorrectly, or giving the configure script nonsense or
gibberish (like alpha where a number is expected), will undoubtedly give
you a defective executable. If it parses, it will probably not work quite
right. Common sense is a virtue :-)
Press RETURN or ENTER to continue. []:
Where do you want the resultant script placed? If you're using configure to
build multiple instances of HTTPi on different ports, make sure this changes
unless you're darn certain that they'll all be configured the same way.
If you are doing a full install to update the configuration files for
launchd/inetd/xinetd/stunnel/etc. simultaneously, THE PATH MUST BE ABSOLUTE!
You can use Perl variables for this option (example: $ENV{'HOME'}/bin/httpi).
As a shortcut, ~/ in first position will be turned into $ENV{'HOME'}/,
which is "/Network/Servers/qbi-xgrid-01.qbi.uq.edu.au/Volumes/InsData1/ClusterUsers/jhunt/".
Install path? [/usr/local/bin/httpi]:
/usr/local/bin/httpi selected.
As a reminder, you do have POSIX.pm, and the Perl you've decided to build
HTTPi with is version 5.8.8, which is capable of using sigaction().
Let's talk signals.
On Perls 5.005, 5.6 and prior to 5.8, POSIX sigaction() didn't work
properly (if at all). Those systems should continue to use the $SIG method
of signal handling, which is technically unsafe but mostly functional.
On Perls 5.8 and higher, sigaction() not only works, but actually works
better than the old $SIG method for HTTPi's purposes and may be required in
future Perls for HTTPi's signal handling to work at all. You should only use
$SIG in this case if you are building HTTPi for another system with an older
or impoverished Perl. If that Perl lacks POSIX.pm, consider setting the
environment variable PERL_SIGNALS to 'unsafe' for the previous behaviour.
If you answer 'n' then this will be added to HTTPi for you.
The recommended default for your version of Perl is given, but you can
override it here. If you don't know what to do, choose the default.
Use sigaction()/POSIX.pm for signal handling? [y]:
y selected.
Where do you want the server to serve documents from? All files that HTTPi
will make available, executables included, must be under this tree (except
for the user filesystem option if enabled, coming up shortly). This is the
webserver's mount directory.
You can use Perl variables for this option (example: $ENV{'HOME'}/bin/httpi).
As a shortcut, ~/ in first position will be turned into $ENV{'HOME'}/,
which is "/Network/Servers/qbi-xgrid-01.qbi.uq.edu.au/Volumes/InsData1/ClusterUsers/jhunt/".
Mount point? [/usr/local/htdocs]:
/usr/local/htdocs selected.

Where do you want the server to put the access log? If you don't want
logging, specify /dev/null. This is the webserver's log file path.
Note that the default is fetchable, i.e., your log can be requested by and
served to a client. Sometimes this is useful, and sometimes this is an
information hole. If this is not desirable to you, make sure the log is not
located under the webserver's mount point.
You can use Perl variables for this option (example: $ENV{'HOME'}/bin/httpi).
As a shortcut, ~/ in first position will be turned into $ENV{'HOME'}/,
which is "/Network/Servers/qbi-xgrid-01.qbi.uq.edu.au/Volumes/InsData1/ClusterUsers/jhunt/".
Log path? [/usr/local/htdocs/access.log]:
/usr/local/htdocs/access.log selected.
WARNING: Make sure the access log is writeable, or there won't be much in it.
Check the file's permissions, just to be safe.
What will the server's name be? This should be a Fully Qualified Domain Name,
like limburgher.cheese.com.
Server host name? [qbi-xgrid-01.qbi.uq.edu.au]:
qbi-xgrid-01.qbi.uq.edu.au selected.
HTTPi does CERN logging format making it compatible with most log analysers.
However, to make it as compatible as possible on as wide a range of Perls as
possible, it doesn't do locale() work to find out what your timezone is.
If you don't care, you can accept the default. If you do, enter a
five-character timezone here (e.g., if you're on Pacific time, like I am,
enter -0800 for 8 hours behind Greenwich mean). [+0000]: +1000
+1000 selected.
HTTPi's answer to .htaccess and access control is the restriction matrix,
allowing access control based on IP address, agent/browser type, and a user
list you can specify with HTTP Basic Auth. For example, the restriction
matrix can restrict access to a certain page only to user fred from the
local LAN, and *then* only if he's using Netscape. This code is slightly
complex, however, so it will add bulk and execution time to your build.
Note that if you plan to only do IP address-based restriction, solutions
like TCP wrappers are probably faster. xinetd also has IP address-based
restriction built in. In those cases, you would probably do better without
the restriction matrix involved.
The settings for the restriction matrix are hardcoded into uservar.in, and
to change your settings you must edit that file and rebuild HTTPi. For help,
please refer to the user's manual.
Enable restriction matrix? [y]: n
n selected.
Webserver logs are a pain, particularly when they get lengthy.
Logging format 1 (here a more CERN compliant variant) was what was supported
in the earliest versions of HTTPi:
host - - [CERNdate] "METHOD address HTTP/V.v" returncode contentlength\
"referer" ""
(example: stockholm.floodgap.com - - [31/Jan/1969:00:00:00] "GET / HTTP/1.0"
200 1000 "http://somewhere.com/" "")
This is a compatible and valid CERN-style log entry, but it doesn't keep or
know about user agents, and it could be smaller. So HTTPi also supports two
other formats:
Type 2 for more "complete" logging, in the Apache/NCSA style:
host - - [CERNdate] "METHOD address HTTP/V.v" returncode length "referer"\
... and type 3 for ultra-terse logging:
host - - [CERNdate] "METHOD address HTTP/V.v" returncode length "" ""
Which type of logging should be used, 1, 2 or 3? [2]:
2 selected.
Want faster CGIs? Meet HTTPi's answer to mod_perl, HTTPerl. mod_perl works its
magic by implementing a Perl interpreter in Apache; HTTPerl takes the obvious
step of reusing the interpreter already running HTTPi to run your executables.
The major advantages:
* Can be faster (see below for when it won't be), especially if
Perl keeps getting paged out.
* Your executables have access to all the HTTPi internal globals and
subroutines, including HTTP negotiation and logging subroutines.
* Works better with POST (lets you manipulate the socket directly).
The major disadvantages:
must run a precompiled binary, write a Perl wrapper, and have HTTPi run that.
Please read the docs, there's important information in there about this!
Enable HTTPerl? [n]:
n selected.
Normally HTTPi uses the execute bits to determine if an executable, well, is.
However, on certain filesystems, or certain filesystems trying to look like
something they're not (Cygwin on FAT32, for example), the execute bits may
not be controllable and HTTPi will end up trying to execute files it
shouldn't or can't. The usual symptom is spurious error messages trying to
serve content and a lot of status 100 in the server log.
To get around this problem, in addition to the execute bit HTTPi can be told
to only execute certain specific file extensions (i.e., they must both be
executable, and have an allowed extension). This is generally not preferred
but may be needed depending on your particular environment. For speed, this
includes only the following: [\-\._](exe|[ckpba]*sh|p[er]*l|cgi|cmd|com)$. Only
enable this if needed by your operating environment.
Enable CGI file extension validation? [n]:
n selected.
If you don't really care if a hostname or an IP address appears in your
access logs, you can save (in some cases substantial) time by instructing
HTTPi not to bother doing name lookups when logging. Most of you will
probably want the names resolved, but for a really sleek server you might not,
so it's now a configurable option.
Resolve IP addresses to hostnames? [y]:
y selected.
Since you're resolving hostnames, I'm sure you've seen the phenomenon of some
sites having bad or even fradulent PTRs when trying to reverse-resolve an
address. This minor anti-spoof feature makes all hostnames into the form
hostname/ip.address.of.hostname (example: localhost/ so that you
can independently see the IP address. If the IP cannot reverse resolve, then
you get a doublet (example:
I've found this handy for accounting, but this might break some loggers
or executables expecting a resolvable name, so this option defaults to no.
This will also affect the REMOTE_HOST CGI variable. REMOTE_ADDR is unchanged.
Always use name/address syntax for reverse resolved names? [n]:
n selected.
Since your Perl has the alarm() call -- and it might even work -- you can make
reverse lookups more reliable (even when the result you get back is fradulent)
instead of having HTTPi pause on bad or defective DNS servers. This defines a
new subroutine &absolver which kills lagging DNS queries after a few seconds.
Without it, you are at the mercy of the timeout specified by your operating
system's implementation (but this may be perfectly adequate, so this option
defaults to no). If you're concerned about this, test reliability both ways.
Use DNS "absolver" for reverse queries? [n]:
n selected.
Not absolved. Confess your sins later.
Modern HTTPi in any flavour can do IP-less virtual hosting by redirecting host
aliases to addresses. For example, you might define (as I do) the alias
httpi.floodgap.com to point to http://www.floodgap.com/httpi/.
This is useful for large hosting sites and aliases, but probably not for
HTTPi's prototypical usage of a little server on a little system, so it's
not enabled by default. If your HTTPi supports it (both xinetd and Demonic
do, for example), you may wish to consider IP-based virtual hosting where you
can do multihoming with individual HTTPi processes as an alternative.
The settings for IP-less virtual hosting are hardcoded into uservar.in, and
to change your settings you must edit that file and rebuild HTTPi. For help,
please refer to the user's manual.
Enable host name redirects? [n]:
n selected.
The New Security Model, introduced in 1.4, adds a additional level of control
over how files are served.
In the older model, HTTPi only changed uid for executables. In this model,
HTTPi changes uid for *all* files, meaning even preparsed documents cannot
take over the webserver. Furthermore, you can specify a uid for which it and
all UIDs lower, is illegal: the server will not change uid to them, and will
not, as a consequence, serve files owned by them (root uid is always illegal)
or run executables on behalf of them (again, root uid is always illegal too).
Other consequences exist -- PLEASE READ THE DOCUMENTATION FIRST.
The New Security Model is ONLY SALIENT IF YOU RUN HTTPi AS ROOT. Otherwise,
it simply adds bulk and overhead. It is also only relevant to Un*xy worlds.
As of 1.5, the New Security Model is now well-tested enough that it is the
strongly recommended default. It may break old installations, so the choice
is still offered, but if you use the user filesystem or preparsing and you
are running your server as root, it is strongly recommended.
Use the New Security Model? [y]:
y selected.
Specify the *lowest* UID that is ALLOWED to serve files. For example, consider
this hypothetical /etc/passwd file (crypts and uids changed to protect the
root:xyzPDQ12:0:0:The Root of All Evil:/doom:/usr/local/bin/mammonsh
ftp:xyzPDQ12:100:100:FTP User:/home/ftp:/bin/false
www:xyzPDQ12:500:500:Webmaster Goddess:/usr/local/htdocs:/usr/local/bin/tcsh
joeuser:xyzPDQ12:501:501:Joe User:/home/joeuser:/usr/local/bin/tcsh
Presumably, we only want www and joeuser to serve files, so we specify
uid 500 so that no UID of 499 or lower will be permitted, even users that
HTTPi as ROOT! Also note that root can NEVER serve files, so specifying zero
as the minimum UID is meaningless.
Lowest UID to serve files (FYI: your pre-sudo euid is 57052)? [57052]: 0
0 selected.
Can we use getpwnam()? ... 0 ... yes
Used to be that HTTPi was a tiny webserver for one person to run on his
machine, but the busy beavers in the HTTPi laboratories have been working
on ways of supporting a user filesystem while keeping HTTPi the slim beast
it is.
If you enable this option, users can now serve files from their own home
directories under ~/public_html. Note, however, that HTTPi draws no difference
between the root server documents and users', so users may also run executables
(and if HTTPi can't change its uid to the executable's owner, this could be
a rather large security hole). For this reason, this option defaults to no.
If you have the New Security Model on, *and* you're running as root, HTTPi
will also change its UID to match the document's, which is useful for
protecting things like /etc/passwd, and for preparsing.
Enable user filesystem? [n]:
n selected.
HTTPi now enables preparsing of selected content types. With the new preparse
module loaded, you can insert inline Perl with the <perl></perl> tags and
access server internals.
Preparsing is done only on files with extensions .sht, .shtm and .shtml,
unless you say otherwise.
preparsing runs as the UID of the webserver and this can be a *huge* security
hole if enabled with the user filesystem. Enable only if you really trust
your users, or if you will be the sole person creating content for HTTPi (or
if you're running HTTPi as some unprivileged user that can't do anything
antisocial) -- under severe, serious advisement!
For information on how to program with inline Perl, see the programming manual.
Enable preparsing? [n]:
n selected.
You said no preparsing, so I won't ask you any more questions about that.
Neener, neener; et la neener. That's French for phooey on you.
Now that I actually pay for my bandwidth, I've become a lot more jealous of
it, which is the rationale for building in a primitive throttling facility.
Primitive means exactly that; you are only guaranteed an approximate maximum
K/sec rating, and even then it's not smooth or well-conditioned. However, it
was easy to implement and it does work. Please note that the throttle settings
are PER PROCESS INSTANCE, not per server en toto.
You may specify how many bytes to suck in and spit out at a swallow, and
how long said swallows take (so, 32,768 bytes and 1 second wait time means,
roughly, 32K/sec maximum output rate per process instance).
Throttling does not yet apply to executable output, virtual file output,
or preparsed file output. These omissions, however, may be useful for
finer-grained throttle control. See the manual for more information.
With throttling off, HTTPi spews files as fast as it can, subject to network
and connection speed.
Use throttling? [n]:
n selected.
Hmm, bandwidth leeches ahoy, eh? ;-)
Skipping onward ...
Now the ugly kludge section. This is really only relevant to inetd users, but
this option may be occasionally useful to Demonic and xinetd installs.
Some inetds will time out, and then shut down, services that hold sockets
open for longer than a critical period of time (some Linux inetds are most
notorious). This usually happens when a very large file is being downloaded
over a very slow link. The upshot is, HTTPi will be turned off by inetd and
fail to respond to requests until inetd gets another -HUP signal. This might
be the basis of a nasty DoS attack, so here it is as a configure option.
HTTPi will simply kill the link if too much time passes, and save itself, if
this option is enabled. You can adjust the timeout with the next question.
You might also want to enable this if you get besieged with requests that
just hang sockets up on your system, but you don't need it if you're not
running inetd HTTPi.
Note that many inetds will not require this (AIX and HP/UX don't seem to).
Don't turn it on unless you think it's necessary for your OS or security.
This may have interesting interactions with the throttling option, by the way.
Auto-kill the link on slow data transfers? [n]:
n selected.
Murderous tendencies quelled.

xml-qstat customization:
Specify where the xml-qstat root is located.
The 'xmlqstat' directory under this root will be served by HTTPi
as the resource '/xmlqstat'.
You can use Perl variables for this option (example: $ENV{'HOME'}/xml-qstat).
As a shortcut, ~/ in first position will be turned into $ENV{'HOME'}/,
which is "/Network/Servers/qbi-xgrid-01.qbi.uq.edu.au/Volumes/InsData1/ClusterUsers/jhunt/".
xml-qstat resource root:? [~/xml-qstat]: ~/olesenm-xml-qstat-96646a3
~/olesenm-xml-qstat-96646a3 selected.
(expanded to /Network/Servers/qbi-xgrid-01.qbi.uq.edu.au/Volumes/InsData1/ClusterUsers/jhunt/olesenm-xml-qstat-96646a3)
Okay. The directory
Seems to have the correct xml-qstat directory structure.

xml-qstat customization:
Define the maximum timeout (seconds) when executing GridEngine commands [10]: 60
60 selected.
Virtual files, only available for Demonic, allows you to preload files or even
create files out of thin air, embed them in the server, and have them fully
served out of memory. Think of it as a stupid, fast, non-dynamic disk cache.
If you have trouble with slow disk access, this could speed your server
considerably if you have memory to burn. This doesn't add much code bloat to
HTTPi, but if you get happy with preloads, the data overhead could hurt. Don't
go overboard.
The settings for the virtual filesystem are hardcoded into uservar.in, and
to change your settings you must edit that file and rebuild HTTPi. For help,
please refer to the user's manual.
Enable virtual filesystem? [n]:
n selected.
HTTPi statistics, only available for Demonic, allows you to keep running
track of your server. You'll need to enable the restriction matrix in order
to keep the statistics URL secure, if you decide to do so.
Access statistics on your server with
for most recent server accesses, running tallies on bytes transferred and
requests handled, and uptime. This module is still in progress, but it works.
Enable HTTPi statistics? [n]:
n selected.
What numerical IP address do you want the webserver to bind to? You can run
multiple HTTPis bound to multiple addresses this way for virtual servers,
if you don't want to use the IP-less Host:-based redirect feature. YOU MUST
IP address information is *hardcoded* into your build, so different IP
addresses must be enabled separately into different executable builds., the default, will bind to whatever address(es) it can find. This is
usually fine for most systems, and should only be changed if you are in fact
trying to use HTTPi to support virtual servers.
Bind to which numerical IP address? []: selected.
What numerical TCP port do you want the webserver to run on? 80 is the default
but if you're using configure to build multiple HTTPis on multiple ports,
make sure you give a different answer this time.
Remember that port numbers below 1024 can only be bound by processes running
as superuser! (That's kind of a security hole, for any webserver even.)
Which TCP port number? [80]: 8080
8080 selected.
Now the tricky part: getting socket constants.
Let's see if you have a C compiler, because experience has told me it
is generally most accurate for these kind of things.
Finding gcc ... /usr/bin/gcc
Trying /usr/bin/gcc ./sockcons.c -o sockcons ...

Yay, got something back! This is what your system is reporting:
SOL_SOCKET = 65535
If this isn't right, hit CTRL-C now. Something's wrong. Otherwise, just
hit ENTER: []:
If you continue beyond this point, configure will build your HTTPi and
overwrite any file currently at /usr/local/bin/httpi.
If you want to continue, just press ENTER.
If you don't, press CTRL-C NOW! []:
Writing out the configured httpi to /usr/local/bin/httpi ... done.
chmod()ding /usr/local/bin/httpi to 0755 ... done.
Successfully configured!
Invoke your new HTTPi with a simple
No & required. Remember: if it's binding to a port < 1024, you'll need to
be root.
Remember to read the documentation file for last minute information. Bye!

Jonathan J Hunt <jjh at 42quarks.com>
Homepage: http://www.42quarks.com
(Further contact details there)
"Physics isn't the most important thing. Love is." Richard Feynman


To unsubscribe from this discussion, e-mail: [users-unsubscribe at gridengine.sunsource.net].

More information about the gridengine-users mailing list