Opened 6 years ago

#1522 new defect

Re: usage: io

Reported by: dlove Owned by:
Priority: normal Milestone:
Component: sge Version: 8.1.8
Severity: minor Keywords:


Dave Turner <drdaveturner@…> writes:


Below is one explanation I found, though it still doesn't explain

why rchar can be 40-50 times what read_bytes is. My understanding
from talking to some people is that rchar may over count when a seek
is done, for example, where no reading is actually being done.
read_bytes and write_bytes are intended to represent the data flow to
and from the physical disks. I've looked in depth at two separate
applications, one quantum chemistry and one bioinformatics, and the
large files that are read in both cases match up exactly with
read_bytes while rchar reports 45 times as much data.


Sorry for not replying earlier. I did check on this earlier and found
I'd even written a comment in the code based on those descriptions,
which I'd forgotten about:

/* rchar and wchar are the bytes going through read(2), write(2)

and friends (such as pread), which may just access the cache.
read_bytes and write_bytes reflect actual i/o on disk
devices, but apparently not on network filesystems. */

I figured it was more useful to have some indication of i/o on networked
filesystems on HPC systems, but I'm not sure how useful even that is on
such systems, especially if you're doing MPI-IO, for instance.

I guess it's easy enough to offer a choice. Would yet another switch in
exec_params to control that be useful?


I/O counter: chars read
The number of bytes which this task has caused to be read from storage. This
is simply the sum of bytes which this process passed to read() and pread().
It includes things like tty IO and it is unaffected by whether or not actual
physical disk IO was required (the read might have been satisfied from


I/O counter: chars written
The number of bytes which this task has caused, or shall cause to be written
to disk. Similar caveats apply here as with rchar.


I/O counter: bytes read
Attempt to count the number of bytes which this process really did cause to
be fetched from the storage layer. Done at the submit_bio() level, so it is
accurate for block-backed filesystems. <please add status regarding NFS and
CIFS at a later time>


I/O counter: bytes written
Attempt to count the number of bytes which this process caused to be sent to
the storage layer. This is done at page-dirtying time.

On Thu, Feb 5, 2015 at 11:44 AM, Dave Love <…> wrote:

Dave Turner <drdaveturner@…> writes:

qstat -j and qacct -j report the io in the usage: row. This value


apparently calculated from wchar and rchar values gotten from
/proc/[PID]/io. write_bytes and read_bytes are accurate representations
of the IO that applications are doing. wchar matches up with


but rchar is often 40-50 times higher than read_byters, and does not
accurately represent the actual input that applications are doing.
This inflates the IO that SGE reports to the user, making it essentially
useless. Simply changing this to use read_byters and write_bytes
rather than rchar and wchar would fix this and provide users with an
accurate representation of the IO their applications are doing.

Dave Turner

Thanks. I've looked at what's provided/used in the past and never
delved deeply enough to figure out what exactly is counted; I've had the
impression that i/o was under-accounted if anything. Is there doc
(other than the Linux source!) which explains? Presumably it's not
going to cover things like MPI-IO over IB anyway, and what's really
needed is integration with appropriate simple profiling data.

[I wonder why that didn't get into the tracker...]

Change History (0)

Note: See TracTickets for help on using tickets.