Custom Query (431 matches)
Results (97 - 99 of 431)
Ticket | Resolution | Summary | Owner | Reporter |
---|---|---|---|---|
#435 | fixed | IZ2298: array job accounting or scheduling problem | Mark Dixon <m.c.dixon@…> | pascalucsf |
Description |
[Imported from gridengine issuezilla http://gridengine.sunsource.net/issues/show_bug.cgi?id=2298] Issue #: 2298 Platform: All Reporter: pascalucsf (pascalucsf) Component: gridengine OS: All Subcomponent: scheduling Version: 6.1 CC: None defined Status: REOPENED Priority: P3 Resolution: Issue type: DEFECT Target milestone: --- Assigned to: andreas (andreas) QA Contact: andreas URL: * Summary: array job accounting or scheduling problem Status whiteboard: Attachments: Issue 2298 blocks: Votes for issue 2298: 35 Opened: Tue Jun 19 14:26:00 -0700 2007 ------------------------ Array job usage seems to be accounted for incorrectly. Example: 100 cpu's on cluster, OS fairshare policy, evenly balanced share tree. User a submits 1000 jobs. User b submits 1000 jobs. User c submits 1 array job, with 1000 members. Results look something like: 48 of user a's jobs running at any time 48 of user b's jobs running at any time 4 of user c's array job members run at any time. If the queue is empty, except for user c's jobs, they will all begin executing. Looking at fairshare usage (via qmon) shows that user c's "Actual Resource Share" (policy configuration -> share tree policy) is very high, (like 50-80%). I can provide detailed configuration on request. ------- Additional comments from pascalucsf Thu Jun 21 14:57:29 -0700 2007 ------- Notes from testing: 4 nodes totaling to 7 cpus on all.q, each node has 4 slots in queue config. scheduler conf: policy_hierarchy OS weight_tickets_share 100000 share tree: id=0 name=template type=0 shares=0 childnodes=1 id=1 name=default type=0 shares=100 childnodes=NONE queue is disabled, and empty. 1000 individual jobs are queued as user pascal 1 array job of 1000 subjobs is queued as user ben usage is cleared (qconf -clearusage) at the starting line: Queued per user: 1000 pascal qw 1000 ben qw bang: qmod -e all.q 1 minute in: Running per user: 8 pascal r 8 ben r Queued per user: 992 pascal qw 992 ben qw (jobs are cpuburners, 5 minutes each) A while later: Running per user: 10 pascal r 1 ben r Queued per user: 991 ben qw 973 pascal qw And it continues this way. ------- Additional comments from andreas Thu Jun 28 11:49:32 -0700 2007 ------- It is very, very likely that this is yet another symptom of #2222. The fix for #2222 will be part of 6.1u1 once it is available. In 6.0u11 this #2222 is already fixed. *** This issue has been marked as a duplicate of 2222 *** ------- Additional comments from andreas Fri Jun 29 02:04:12 -0700 2007 ------- Revert: Can't be duplicate since #2222 was already fixed with 6.1. ------- Additional comments from andreas Fri Jun 29 02:22:39 -0700 2007 ------- What are you using for weight_urgency weight_ticket weight_waiting_time in sched_conf(5). If your waiting time weight is non-zero this could cause the phenomenon you observe. Reason is that waiting time contributes to job urgency and urgency has higher weight than the ticket policy. ------- Additional comments from pascalucsf Fri Jun 29 08:25:58 -0700 2007 ------- These are all of the weight_* values for configurations where I have seen this problem. ---------------------------------------------------- weight_ticket 1.000000 weight_waiting_time 0.000000 weight_deadline 0.000000 weight_urgency 0.000000 weight_priority 0.000000 ---------------------------------------------------- weight_ticket 0.900000 weight_waiting_time 0.000000 weight_deadline 0.000000 weight_urgency 0.000000 weight_priority 0.100000 ---------------------------------------------------- weight_ticket 0.010000 weight_waiting_time 0.000000 weight_deadline 3600000.000000 weight_urgency 0.100000 weight_priority 1.000000 ---------------------------------------------------- ------- Additional comments from andreas Mon Jul 2 08:28:45 -0700 2007 ------- After running ---> 100 times: qsub -P A -b y /bin/sleep 5 ---> 100 times: qsub -P B -b y /bin/sleep 5 ---> 1 times: qsub -t 1-100 -P C -b y /bin/sleep 5 with SHARETREE_RESERVED_USAGE=true be set in global cluster configuration sge_conf(5) I get a combined resource usage which sometimes is really surprisingly unbalanced. I played around with different arrangements to get a clue on this phenomenon: Project| Comb. Usage | Sum. Acct. Usage ------------------------------------------- A | 1136.78 | 1085 B | 1161.77 | 1100 C | 1292.73 | 1189 (array) ------------------------------------------- A | 1294.78 | 1222 (array) B | 1159.82 | 1080 C | 1154.82 | 1097 ------------------------------------------- A | 1052.86 | 997 B | 1047.86 | 991 C | 1224.82 | 1137 (array) ------------------------------------------- A | 782.36 | 655 (array) B | 646.80 | 590 C | 645.80 | 586 ------------------------------------------- A | 635.88 | 568 B | 634.88 | 570 C | 647.88 | 569 (array) ------------------------------------------- A | 700.77 | 640 (array) B | 697.77 | 633 C | 670.77 | 605 ------------------------------------------- A | 656.83 | 585 (array) B | 629.84 | 570 C | 640.84 | 581 ------------------------------------------- this shows the accounted usage of array jobs is constantly higher than of of sequential jobs! I investigated it to the point that I can say: For some mysterious reason array tasks on average take longer time from fork() until /bin/sleep actually starts. Interestingly use of "-shell no" submit option finally gave me a very much balanced distribution of per project accounting, but I still can not explain why array jobs should be affected from the overhead than sequential jobs ... :-o With regards to the share tree behaviour I recommend use of a by far lower compensation factor than the default. With the compensation factor one can control how much projects with higher usage shall be penalized. When I used a 1 or 2 as compensation factor I got quite good results despite of the unbalanced accounting. ------- Additional comments from jlb Wed Mar 26 10:49:30 -0700 2008 ------- Testing on my production cluster (>300 nodes), I consistently see utilization numbers (as reported by sge_share_mon) ~22% higher for array jobs than for the equivalent number of individually submitted jobs. This is a rather significant difference, in my opinion. Using "-shell no" has absolutely no effect on this over-accounting in my testing. ------- Additional comments from pascalucsf Wed Mar 26 16:22:20 -0700 2008 ------- Another take on this: From a running, saturated queue on a cluster of 256 cpus over 128 machines, a sample of 3 users, 2 running 1 array job each and 1 running 21 single jobs is taken 60 seconds after a qconf -clearusage. In the following output, user1 is the user with the single jobs. Side notes: Each of these jobs runs only on a single processor. Each of these jobs is CPU bound. Other jobs by other users are running on the cluster. Other jobs by THESE users are NOT running on the cluster. Wed Mar 26 15:57:34 PDT 2008 user1 jobs running: 21 user2 jobs running: 54 user3 jobs running: 53 Wed Mar 26 15:58:34 PDT 2008 usage_time user_name actual_share usage cpu 1206572554 user1 1120.975817 1120.975817 1426.595958 1206572554 user3 46592.519562 46592.519562 12893.812376 1206572554 user2 45888.666024 45888.666024 12691.742827 user1 jobs running: 21 user2 jobs running: 54 user3 jobs running: 53 user1 jobs running: So what I think the times should be: user1: 60 (seconds) * 21 (jobs) = 1260 cpuseconds user2: 60 (seconds) * 54 (jobs) = 3240 cpuseconds user3: 60 (seconds) * 53 (jobs) = 3180 cpuseconds user1's output from sge_share_mon lines up reasonably well. user 2 and 3 are very over their estimated usage. Also it's unclear why usage and cpu differ so much, as I am only using cpu time for usage: usage_weight_list cpu=1.000000,mem=0.000000,io=0.000000 Are there any flaws in my testing method here? Does this shine any light on the situation? Thanks, -Pascal ------- Additional comments from jlb Mon Apr 7 16:49:17 -0700 2008 ------- Further observation -- the CPU usage reported by qacct is essentially equal for array jobs and equivalent numbers of individual jobs. In other words, 'ltcpu' as reported by sge_share_mon differs from 'CPU' as reported by qacct. Does that help narrow down where this bug may be at all? Also, if I switch to the functional share policy, then array jobs are scheduled with priority equal to that of individually submitted jobs. ------- Additional comments from andreas Mon Jul 7 09:08:19 -0700 2008 ------- Actually the result of my investigation was that array jobs on average cause higher utilization in SGE accounting than sequential jobs, but I could not find a reason for this. Are you using local spooling for your execution daemons? My suspicion was the deviations from the ideal total job run-time are an outcome of delays during job startup/shutdown due to a bottleneck situation at the file server. Though this would not explain higher utilization by array jobs, but I think understanding the net/gross deviation is prerequisite for getting an idea on how to level the sequential/array job variation. |
|||
#436 | fixed | IZ2304: NFS write error on SGE trace files | guenter_herbert | |
Description |
[Imported from gridengine issuezilla http://gridengine.sunsource.net/issues/show_bug.cgi?id=2304] Issue #: 2304 Platform: Sun Reporter: guenter_herbert (guenter_herbert) Component: gridengine OS: All Subcomponent: kernel Version: 6.0u10 CC: None defined Status: NEW Priority: P3 Resolution: Issue type: DEFECT Target milestone: --- Assigned to: guenter_herbert (guenter_herbert) QA Contact: andreas URL: * Summary: NFS write error on SGE trace files Status whiteboard: Attachments: Issue 2304 blocks: Votes for issue 2304: Opened: Wed Jun 20 08:13:00 -0700 2007 ------------------------ This is hook to CR6541085 |
|||
#438 | fixed | IZ2310: 3rdparty/qmon/Xmt310: FTBFS with lesstif 0.95.0 | anbe | |
Description |
[Imported from gridengine issuezilla http://gridengine.sunsource.net/issues/show_bug.cgi?id=2310] Issue #: 2310 Platform: All Reporter: anbe (anbe) Component: gridengine OS: All Subcomponent: build Version: 6.1 CC: [_] opoplawski [_] Remove selected CCs Status: NEW Priority: P3 Resolution: Issue type: DEFECT Target milestone: --- Assigned to: andre (andre) QA Contact: andreas URL: * Summary: 3rdparty/qmon/Xmt310: FTBFS with lesstif 0.95.0 Status whiteboard: Attachments: Date/filename: Description: Submitted by: Thu Mar 27 13:03:00 -0700 2008: gridengine-6.0u8-lesstif.patch Patch used by Fedora with lesstif (text/plain) opoplawski Issue 2310 blocks: Votes for issue 2310: Opened: Sun Jun 24 14:09:00 -0700 2007 ------------------------ gridengine (3rdparty/qmon/Xmt310) does not build if the most recent version of lesstif (0.95.0) is used since some symbols are no longer defined in /usr/include/Xm/XmStrDefs.h in this version: ../Xmt310/Xmt/PixmapCvt.c:77: error: 'XmRLargeIconPixmap' undeclared here (not in a function) ../Xmt310/Xmt/PixmapCvt.c:78: error: 'XmRSmallIconPixmap' undeclared here (not in a function) ../Xmt310/Xmt/PixmapCvt.c:79: error: 'XmRDynamicPixmap' undeclared here (not in a function) (tested with the Debian packages lesstif2-dev (1:0.94.4-2) from Debian stable and lesstif2-dev (1:0.95.0-2) from Debian testing/unstable) ------- Additional comments from opoplawski Thu Mar 27 13:03:20 -0700 2008 ------- Created an attachment (id=160) Patch used by Fedora with lesstif ------- Additional comments from andreas Mon Mar 31 00:56:17 -0700 2008 ------- Andre must review this. ------- Additional comments from opoplawski Tue May 27 14:01:22 -0700 2008 ------- Add myself to CC. |
Note: See TracQuery
for help on using queries.