Increasing Priority of lgwr Process using _high_priority_processes
At the Hotsos Symposium last week, Tanel Poder pointed out an interesting hidden parameter (_high_priority_processes). We were discussing the occasional need to increase priority on the LGWR process when dealing with “log file sync” issues. The aforementioned parameter is used to tell Oracle to automatically set background processes to a higher than normal priority when the database is started. On linux this means putting them in the RR class, in Solaris, the RT class is used. The parameter defaults to LMS* in 10.2 and to LMS*||VKTM in 11gR1. (LMS* are the Lock Manager Server processes in a RAC instance and VKTM is the new Time Keeper process in 11g). So it makes sense that these processes would run at a higher priority.
Recently I have been working on a system that was spending a significant amount of time waiting on “log file sync”. Increasing the priority of the LGWR process made a significant improvement. It’s not a silver bullet though. When the system becomes really busy, the “log file sync” times still suck, just not as much. But I digress, this post is just to point out that Oracle has a built in mechanism to increase priority on background processes, not to tell you how to reduce long “log file sync” times. There are numerous posts on that subject already. Here a couple of pretty good ones:
At the Hotsos symposium last week, Tanel Poder pointed to an interesting hidden parameter of the beneficial property of the viagradrug. It is very effective in helping men who have erection problems.
Manly-Men Only Use Solid State Disk for Redo Logging – Kevin Closson
Tuning Log File Sync Wait Events – Riyaj Shamsudeen
Hotsos 2010 – Day 4 – Doug Burns (see the comments section)
Back to my story. Unfortunately, every time the database is bounced, LGWR goes back to his original priority and class and we have to get the Unix guys to reset it. So I got to wondering if the _high_priority_processes parameter could be used to do this automatically. And sure enough it looks like it can. First here’s a look at the class and priority info of the standard high priority processes on a Solaris system and a Linux system.
==== Solaris - RT processes (10g RAC) ====
oracle 6036 system RT global - 100 1 - 16:31 ora_lms3_posp1
oracle 6032 system RT global - 100 1 - 16:14 ora_lms2_posp1
oracle 6028 system RT global - 100 1 - 16:20 ora_lms1_posp1
oracle 6024 system RT global - 100 1 - 16:50 ora_lms0_posp1
==== Redhat Linux - RR processes (11gR1 RAC) ====
15823 RR 41 - 00:00:00 ora_vktm_SCNPRD1
15849 RR 41 - 00:01:19 ora_lms0_SCNPRD1
15853 RR 41 - 00:01:27 ora_lms1_SCNPRD1
Here is an example of setting the parameter to include a couple of other processes (LGWR and PMON) and the changes to the class and priority after restarting the instance.
==== Redhat Linux - Non-RAC ====
> !sql
sqlplus "/ as sysdba"
SQL*Plus: Release 10.2.0.3.0 - Production on Mon Mar 15 12:53:31 2010
Copyright (c) 1982, 2006, Oracle. All Rights Reserved.
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning, OLAP and Data Mining options
SYS@LAB102> @parms
Enter value for parameter: _high_priority_processes
Enter value for isset:
Enter value for show_hidden: Y
NAME VALUE ISDEFAUL ISMODIFIED ISSET
-------------------------------------------------- ---------------------------------------------------------------------- -------- ---------- ----------
_high_priority_processes LMS* TRUE FALSE FALSE
SYS@LAB102> !ps -eo pid,class,pri,nice,time,args | grep LAB102 | grep -v LAB1024 | grep -v grep | sort -k 6,6
19080 TS 24 0 00:00:00 ora_a001_LAB102
6989 TS 23 0 00:00:00 ora_cjq0_LAB102
6480 TS 22 0 00:00:00 ora_ckpt_LAB102
19047 TS 18 0 00:00:01 oracleLAB102 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
5965 TS 23 0 00:00:00 ora_dbw0_LAB102
6127 TS 23 0 00:00:00 ora_dbw1_LAB102
6296 TS 23 0 00:00:00 ora_lgwr_LAB102
5780 TS 23 0 00:00:00 ora_mman_LAB102
7370 TS 23 0 00:00:00 ora_mmnl_LAB102
7174 TS 23 0 00:00:00 ora_mmon_LAB102
19049 TS 23 0 00:00:00 ora_p000_LAB102
19051 TS 23 0 00:00:00 ora_p001_LAB102
19053 TS 23 0 00:00:00 ora_p002_LAB102
19055 TS 23 0 00:00:00 ora_p003_LAB102
19057 TS 23 0 00:00:00 ora_p004_LAB102
19059 TS 23 0 00:00:00 ora_p005_LAB102
19061 TS 23 0 00:00:00 ora_p006_LAB102
19063 TS 23 0 00:00:00 ora_p007_LAB102
19065 TS 23 0 00:00:00 ora_p008_LAB102
19067 TS 23 0 00:00:00 ora_p009_LAB102
19069 TS 23 0 00:00:00 ora_p010_LAB102
19071 TS 23 0 00:00:00 ora_p011_LAB102
19073 TS 23 0 00:00:00 ora_p012_LAB102
19075 TS 23 0 00:00:00 ora_p013_LAB102
19077 TS 23 0 00:00:00 ora_p014_LAB102
5413 TS 23 0 00:00:00 ora_pmon_LAB102
5599 TS 23 0 00:00:00 ora_psp0_LAB102
6845 TS 23 0 00:00:00 ora_reco_LAB102
6667 TS 23 0 00:00:00 ora_smon_LAB102
SYS@LAB102> -- notice that all processes are in TS class with priorities around 23
SYS@LAB102>
SYS@LAB102> alter system set "_high_priority_processes"='LMS*|LGWR|PMON' scope=spfile;
System altered.
SYS@LAB102> startup force
ORACLE instance started.
Total System Global Area 3221225472 bytes
Fixed Size 1264380 bytes
Variable Size 1493173508 bytes
Database Buffers 1711276032 bytes
Redo Buffers 15511552 bytes
Database mounted.
Database opened.
SYS@LAB102> !ps -eo pid,class,pri,nice,time,args | grep LAB102 | grep -v LAB1024 | grep -v grep | sort -k 6,6
22170 TS 23 0 00:00:00 ora_a001_LAB102
13368 TS 23 0 00:00:00 ora_cjq0_LAB102
12912 TS 22 0 00:00:00 ora_ckpt_LAB102
22091 TS 18 0 00:00:01 oracleLAB102 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
12396 TS 23 0 00:00:00 ora_dbw0_LAB102
12558 TS 23 0 00:00:00 ora_dbw1_LAB102
4827 TS 23 0 00:00:00 ora_j000_LAB102
12732 RR 90 - 00:00:00 ora_lgwr_LAB102
12248 TS 23 0 00:00:00 ora_mman_LAB102
13713 TS 23 0 00:00:00 ora_mmnl_LAB102
13566 TS 23 0 00:00:00 ora_mmon_LAB102
22093 TS 23 0 00:00:00 ora_p000_LAB102
22095 TS 23 0 00:00:00 ora_p001_LAB102
22097 TS 23 0 00:00:00 ora_p002_LAB102
22099 TS 23 0 00:00:00 ora_p003_LAB102
22105 TS 23 0 00:00:00 ora_p004_LAB102
22107 TS 23 0 00:00:00 ora_p005_LAB102
22110 TS 23 0 00:00:00 ora_p006_LAB102
22118 TS 23 0 00:00:00 ora_p007_LAB102
22125 TS 23 0 00:00:00 ora_p008_LAB102
22127 TS 23 0 00:00:00 ora_p009_LAB102
22134 TS 23 0 00:00:00 ora_p010_LAB102
22142 TS 23 0 00:00:00 ora_p011_LAB102
22152 TS 23 0 00:00:00 ora_p012_LAB102
22160 TS 23 0 00:00:00 ora_p013_LAB102
22166 TS 23 0 00:00:00 ora_p014_LAB102
11817 RR 90 - 00:00:00 ora_pmon_LAB102
12108 TS 23 0 00:00:00 ora_psp0_LAB102
13209 TS 23 0 00:00:00 ora_reco_LAB102
13050 TS 23 0 00:00:00 ora_smon_LAB102
SYS@LAB102> -- now the lgwr and pmon processes are in RR class and priority is 90
SYS@LAB102>
SYS@LAB102> !ps -eo pid,class,pri,nice,time,args | grep LAB102 | grep RR
1040 TS 20 0 00:00:00 /bin/bash -c ps -eo pid,class,pri,nice,time,args | grep LAB102 | grep RR
11817 RR 90 - 00:00:00 ora_pmon_LAB102
12732 RR 90 - 00:00:00 ora_lgwr_LAB102
So you can see that the database did indeed cause the background processes specified in the _high_priority_processes parameter to run with a higher priority / scheduling class. Cool, maybe we can let the Unix guy get a little sleep now.
Disclaimers:
- This is a hidden parameter – so you probably shouldn’t use it.
- Reducing commits is the best way to solve “log file sync” issues.
- Isolating the lgwr process via processor affinity is another possible approach.
- Sit up straight and don’t talk with your mouth full.
Does anyone know what the LMS* value stands for?
I’m also wondering if the higher priority is a fixed “jump” or is there a way to specify a given one?
For example, in AIX I can only change priorities within certain limits set by the sysadmin, what happens if Oracle tries to go higher?
(I guess I have to try it out soon, currently I just sudo a renice…)
Noons,
RAC has multiple LMS processes (lms0, lms1, …) – LMS* appears to mean do it all the LMS background processes. I am not aware of any controls for the priority number itself. If you find one let me know.
Kerry
Great jobs, but for 9i is it applicable?I dont have in my istance the hidden parameter _high_priority_processes…
Let me know.
Abe
No Abe, it’s 10.2 and later.
Kerry
Hmmm, I can confidently report that in AIX 5.3 with Oracle 10.2.0.3, this does not work.
I’ve tried exactly the same string as here, no effect whatsoever on the priority.
Mind you: I’m not running RAC, don’t know if that would have any effect.
Noons,
Interesting. I wouldn’t think RAC would make a difference. The instances I tested it on were not RAC. I have only tried it on Redhat Linux so far though. I’ll give it a try on Solaris and post my results.
Kerry
It works fine on Solaris 10 with a 10.2.0.4 database. Sets the processes to RT scheduling class and 100 priority.
Kerry
Hi Kerry
Nice post!
To Tanel’s and Bob Sneed’s point, RT cpu priority is kernel preemptive. LGWR can block the interrupts being serviced by executing in RT mode in the CPU. LGWR needs those interrupts to be serviced and those interrupts quite probably could be I/O completion interrupts for LGWR to complete commit/log file sync processing. This issue of waiter blocking the interrupts server leads to an incorrect priority anomaly, analogous to a waiter of a resource evicting the holder of a resource off the CPU.
On the other hand, in RAC, LMS processes will wait for LGWR to complete ‘log file sync’ if the block to be transferred is considered “busy”. But, LMS is also running in RT mode. So if the LGWR process is not running in RT mode, then the LMS process can evict LGWR from the CPU. This also leads to incorrect priority anomaly.
If there are enough CPUs, in RAC, I would prefer, to run both LGWR and LMS* processes in RT mode (and VKTM in 11g). Then fence the interrupts to a different processor set so that LGWR/LMS* can not block those interrupts. In this way we should be able to keep interrupts/LMS/LGWR priorities straight. [At the very least, run LGWR in FX 60 in Solaris so that LGWR does not suffer from TS class scheduling latencies.]
It is important to also realize that, default number of LMS processes are quite excessive. For example, one of my client site had 28 LMS processes and all 28 of them running in RT mode! You can imagine how much CPU usage will be used by those LMS processes. In my opinion, in most cases, 3 or 5 LMS processes per instance is good enough to service GC traffic. So, if the number of LMS processes are excessive, then it is probably worthwhile to reduce that and then also increase LGWR priority.
Last but not least, I still fail to see a point of running VKTM in RT mode!
Thanks
Riyaj
Found something very interesting. Remember when I asked what priority level did this set processes to?
Well, it turns out it’s nothing to do with that. This parameter raises the processes to real time priority class, not level. What controls the level is (yet!) another hidden parameter:
_os_sched_high_priority
For complete reference, go to MOS and search for “602419.1”.
From the result set, pick the
“LMS not running in RT (real time) mode in 10.2.0.3 RAC database” link.
It’s all explained there.
See what you done? Now I *must* test all these things!!!!
😀
Noons,
Interesting. The _os_sched_high_priority parameter appears to just ratchet the priority up within the RT/RR class. On linux with 10.2.0.4, a vlue of 1 = priority of 41, 2 = 42, 10=50, 20=60, etc… I think once you get into the Real Time class that moving the priority up will have little value. At that point I think you’d need to follow Riyaj’s suggestion regarding interrupt fencing / isolating LGWR. Thanks for pointing out the parameter and the Metalink reference.
Kerry
Riyaj,
Thanks for your detailed comments. Sorry about the trouble getting the site to accept them.
Kerry
Hi!
I have situation like in Metalink note but mine owners are OK.
old 9: and a.ksppinm like ‘\_%&1%’ escape ‘\’
new 9: and a.ksppinm like ‘\_%os_sched_high%’ escape ‘\’
NAME VALUE DEFLT TYPE DESCRIPTION
————————- ————— —– ——– ———————————–
_os_sched_high_priority 1 TRUE number OS high priority level
SQL> !
[oracle HAC4@iishacrac4 ~]$ ps -efc | grep lms
oracle 5566 1 RR 41 14:12 ? 00:00:05 ora_lms0_HAC4
oracle 5570 1 RR 41 14:12 ? 00:00:05 ora_lms1_HAC4
oracle 17477 17425 TS 21 14:29 pts/0 00:00:00 grep lms
oracle 32708 1 RR 41 Apr02 ? 00:00:55 asm_lms0_+ASM4
[oracle HAC4@iishacrac4 ~]$ ll $ORACLE_HOME/bin/oradism
-r-sr-s— 1 root dba 20062 Sep 11 2008 /u01/HAC/hacdb/10.2.0/bin/oradism
How to fix LMS process to run in RR? I’m on Linux IA64and RHEL EE 5.3 so there is no support for this combination…even on Metalink!
I’m suffering node eviction without reason. After I set digiwait and make some configuration changes …now in last 10 days there is no eviction…even on heavy stress (800 sql connections)…
So I’m pretty desperate…in searching for the reason…
THX for any help or comment
Damir Vadas
Hi Damir,
I’m not sure I’m understanding your question. Looks like from the output above that your lms processes are running in RR class at priority 41.
oracle 5566 1 RR 41 14:12 ? 00:00:05 ora_lms0_HAC4
oracle 5570 1 RR 41 14:12 ? 00:00:05 ora_lms1_HAC4
Have you opened an SR?
Kerry
Hi Kerry,
SR is opened. Sent RDA reports and waiting on their answer. For now they say it looks like a new bug ….
When find out let you know here…
Rg
Damir
[…] 10-How to increase process priority within Oracle with _high_priority_processes? Kerry Osborne-Increasing Priority of lgwr Process using _high_priority_processes […]
Hi!
SR is closed as unresolved and there is nothing else to do.
Main problem from mine point of view is Intel Itanium CPU, configuration that is really rare, so I think there is no testing configuration on Oracle Technical support side.
Rg,
Damir Vadas
Well there are advantages to staying in the middle of the pack.
Kerry
[…] are not preempted by lower priority processes. We know that since 10g some of Oracle processes have Real-Time priority. By default only LMS and VKTM processes scheduled in RT class. These processes do not use mutexes. […]
By the way, I was asked if there was a reason to increase priority on pmon. The answer is no. It is just an example. I’m not suggesting to increase pmon priority to improve log file sync times.
Hi Kerry,
Our system is Rac 3 nodes 10.2.0.4
In Top 5 Timed Events
gcs log flush sync 2,324,321 2,311 1 8.6 Other
gc cr block busy 308,219 2,079 7 7.7 Cluster
cpu_count = 128
_high_priority_processes LMS* LMS*
–> do i need
alter system set “_high_priority_processes”=”LMS*|LGWR*” scope=spfile sid=’*’;
if “yes”, do you have any recommendation for me.
Thanks
Tri Tuc,
The answer is maybe it could help. Those wait events are related to moving blocks between RAC nodes, so the best resolution would be to figure out why so many blocks have to move (as opposed to just rtying to make the move faster). Maybe some sort of workload partitioning would work better.
Kerry
[…] increase LGWR priority(renice or _high_priority_processes), […]