Increasing Priority of lgwr Process using _high_priority_processes

At the Hotsos Symposium last week, Tanel Poder pointed out an interesting hidden parameter (_high_priority_processes). We were discussing the occasional need to increase priority on the LGWR process when dealing with “log file sync” issues. The aforementioned parameter is used to tell Oracle to automatically set background processes to a higher than normal priority when the database is started. On linux this means putting them in the RR class, in Solaris, the RT class is used. The parameter defaults to LMS* in 10.2 and to LMS*||VKTM in 11gR1. (LMS* are the Lock Manager Server processes in a RAC instance and VKTM is the new Time Keeper process in 11g). So it makes sense that these processes would run at a higher priority.

Recently I have been working on a system that was spending a significant amount of time waiting on “log file sync”. Increasing the priority of the LGWR process made a significant improvement. It’s not a silver bullet though. When the system becomes really busy, the “log file sync” times still suck, just not as much. But I digress, this post is just to point out that Oracle has a built in mechanism to increase priority on background processes, not to tell you how to reduce long “log file sync” times. There are numerous posts on that subject already. Here a couple of pretty good ones:


Manly-Men Only Use Solid State Disk for Redo Logging – Kevin Closson

Tuning Log File Sync Wait Events – Riyaj Shamsudeen
Hotsos 2010 – Day 4 – Doug Burns (see the comments section)

Back to my story. Unfortunately, every time the database is bounced, LGWR goes back to his original priority and class and we have to get the Unix guys to reset it. So I got to wondering if the _high_priority_processes parameter could be used to do this automatically. And sure enough it looks like it can. First here’s a look at the class and priority info of the standard high priority processes on a Solaris system and a Linux system.


==== Solaris - RT processes (10g RAC) ====

    oracle  6036   system   RT   global   - 100    1   -       16:31 ora_lms3_posp1
    oracle  6032   system   RT   global   - 100    1   -       16:14 ora_lms2_posp1
    oracle  6028   system   RT   global   - 100    1   -       16:20 ora_lms1_posp1
    oracle  6024   system   RT   global   - 100    1   -       16:50 ora_lms0_posp1


==== Redhat Linux - RR processes (11gR1 RAC) ====

    15823 RR   41   - 00:00:00 ora_vktm_SCNPRD1
    15849 RR   41   - 00:01:19 ora_lms0_SCNPRD1
    15853 RR   41   - 00:01:27 ora_lms1_SCNPRD1

Here is an example of setting the parameter to include a couple of other processes (LGWR and PMON) and the changes to the class and priority after restarting the instance.

==== Redhat Linux - Non-RAC ====
 
> !sql
sqlplus "/ as sysdba"
 
SQL*Plus: Release 10.2.0.3.0 - Production on Mon Mar 15 12:53:31 2010
 
Copyright (c) 1982, 2006, Oracle.  All Rights Reserved.
 
 
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.3.0 - Production
With the Partitioning, OLAP and Data Mining options
 
 
SYS@LAB102> @parms
Enter value for parameter: _high_priority_processes
Enter value for isset: 
Enter value for show_hidden: Y
 
NAME                                               VALUE                                                                  ISDEFAUL ISMODIFIED ISSET
-------------------------------------------------- ---------------------------------------------------------------------- -------- ---------- ----------
_high_priority_processes                           LMS*                                                                   TRUE     FALSE      FALSE
 
SYS@LAB102> !ps -eo pid,class,pri,nice,time,args | grep LAB102 | grep -v LAB1024 | grep -v grep | sort -k 6,6
19080 TS   24   0 00:00:00 ora_a001_LAB102
 6989 TS   23   0 00:00:00 ora_cjq0_LAB102
 6480 TS   22   0 00:00:00 ora_ckpt_LAB102
19047 TS   18   0 00:00:01 oracleLAB102 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
 5965 TS   23   0 00:00:00 ora_dbw0_LAB102
 6127 TS   23   0 00:00:00 ora_dbw1_LAB102
 6296 TS   23   0 00:00:00 ora_lgwr_LAB102
 5780 TS   23   0 00:00:00 ora_mman_LAB102
 7370 TS   23   0 00:00:00 ora_mmnl_LAB102
 7174 TS   23   0 00:00:00 ora_mmon_LAB102
19049 TS   23   0 00:00:00 ora_p000_LAB102
19051 TS   23   0 00:00:00 ora_p001_LAB102
19053 TS   23   0 00:00:00 ora_p002_LAB102
19055 TS   23   0 00:00:00 ora_p003_LAB102
19057 TS   23   0 00:00:00 ora_p004_LAB102
19059 TS   23   0 00:00:00 ora_p005_LAB102
19061 TS   23   0 00:00:00 ora_p006_LAB102
19063 TS   23   0 00:00:00 ora_p007_LAB102
19065 TS   23   0 00:00:00 ora_p008_LAB102
19067 TS   23   0 00:00:00 ora_p009_LAB102
19069 TS   23   0 00:00:00 ora_p010_LAB102
19071 TS   23   0 00:00:00 ora_p011_LAB102
19073 TS   23   0 00:00:00 ora_p012_LAB102
19075 TS   23   0 00:00:00 ora_p013_LAB102
19077 TS   23   0 00:00:00 ora_p014_LAB102
 5413 TS   23   0 00:00:00 ora_pmon_LAB102
 5599 TS   23   0 00:00:00 ora_psp0_LAB102
 6845 TS   23   0 00:00:00 ora_reco_LAB102
 6667 TS   23   0 00:00:00 ora_smon_LAB102
 
SYS@LAB102> -- notice that all processes are in TS class with priorities around 23
SYS@LAB102>
SYS@LAB102> alter system set "_high_priority_processes"='LMS*|LGWR|PMON' scope=spfile;                           
 
System altered.
 
SYS@LAB102> startup force
ORACLE instance started.
 
Total System Global Area 3221225472 bytes
Fixed Size                  1264380 bytes
Variable Size            1493173508 bytes
Database Buffers         1711276032 bytes
Redo Buffers               15511552 bytes
Database mounted.
Database opened.
SYS@LAB102> !ps -eo pid,class,pri,nice,time,args | grep LAB102 | grep -v LAB1024 | grep -v grep | sort -k 6,6
22170 TS   23   0 00:00:00 ora_a001_LAB102
13368 TS   23   0 00:00:00 ora_cjq0_LAB102
12912 TS   22   0 00:00:00 ora_ckpt_LAB102
22091 TS   18   0 00:00:01 oracleLAB102 (DESCRIPTION=(LOCAL=YES)(ADDRESS=(PROTOCOL=beq)))
12396 TS   23   0 00:00:00 ora_dbw0_LAB102
12558 TS   23   0 00:00:00 ora_dbw1_LAB102
 4827 TS   23   0 00:00:00 ora_j000_LAB102
12732 RR   90   - 00:00:00 ora_lgwr_LAB102
12248 TS   23   0 00:00:00 ora_mman_LAB102
13713 TS   23   0 00:00:00 ora_mmnl_LAB102
13566 TS   23   0 00:00:00 ora_mmon_LAB102
22093 TS   23   0 00:00:00 ora_p000_LAB102
22095 TS   23   0 00:00:00 ora_p001_LAB102
22097 TS   23   0 00:00:00 ora_p002_LAB102
22099 TS   23   0 00:00:00 ora_p003_LAB102
22105 TS   23   0 00:00:00 ora_p004_LAB102
22107 TS   23   0 00:00:00 ora_p005_LAB102
22110 TS   23   0 00:00:00 ora_p006_LAB102
22118 TS   23   0 00:00:00 ora_p007_LAB102
22125 TS   23   0 00:00:00 ora_p008_LAB102
22127 TS   23   0 00:00:00 ora_p009_LAB102
22134 TS   23   0 00:00:00 ora_p010_LAB102
22142 TS   23   0 00:00:00 ora_p011_LAB102
22152 TS   23   0 00:00:00 ora_p012_LAB102
22160 TS   23   0 00:00:00 ora_p013_LAB102
22166 TS   23   0 00:00:00 ora_p014_LAB102
11817 RR   90   - 00:00:00 ora_pmon_LAB102
12108 TS   23   0 00:00:00 ora_psp0_LAB102
13209 TS   23   0 00:00:00 ora_reco_LAB102
13050 TS   23   0 00:00:00 ora_smon_LAB102
 
SYS@LAB102> -- now the lgwr and pmon processes are in RR class and priority is 90
SYS@LAB102>
SYS@LAB102> !ps -eo pid,class,pri,nice,time,args | grep LAB102 | grep RR
 1040 TS   20   0 00:00:00 /bin/bash -c ps -eo pid,class,pri,nice,time,args | grep LAB102 | grep RR
11817 RR   90   - 00:00:00 ora_pmon_LAB102
12732 RR   90   - 00:00:00 ora_lgwr_LAB102

So you can see that the database did indeed cause the background processes specified in the _high_priority_processes parameter to run with a higher priority / scheduling class. Cool, maybe we can let the Unix guy get a little sleep now.

Disclaimers:

  1. This is a hidden parameter – so you probably shouldn’t use it.
  2. Reducing commits is the best way to solve “log file sync” issues.
  3. Isolating the lgwr process via processor affinity is another possible approach.
  4. Sit up straight and don’t talk with your mouth full.
  • 22 Comments

    1. Noons says:

      Does anyone know what the LMS* value stands for?
      I’m also wondering if the higher priority is a fixed “jump” or is there a way to specify a given one?
      For example, in AIX I can only change priorities within certain limits set by the sysadmin, what happens if Oracle tries to go higher?
      (I guess I have to try it out soon, currently I just sudo a renice…)

    2. osborne says:

      Noons,

      RAC has multiple LMS processes (lms0, lms1, …) – LMS* appears to mean do it all the LMS background processes. I am not aware of any controls for the priority number itself. If you find one let me know.

      Kerry

    3. Abe says:

      Great jobs, but for 9i is it applicable?I dont have in my istance the hidden parameter _high_priority_processes…
      Let me know.
      Abe

    4. osborne says:

      No Abe, it’s 10.2 and later.

      Kerry

    5. Noons says:

      Hmmm, I can confidently report that in AIX 5.3 with Oracle 10.2.0.3, this does not work.
      I’ve tried exactly the same string as here, no effect whatsoever on the priority.
      Mind you: I’m not running RAC, don’t know if that would have any effect.

    6. osborne says:

      Noons,

      Interesting. I wouldn’t think RAC would make a difference. The instances I tested it on were not RAC. I have only tried it on Redhat Linux so far though. I’ll give it a try on Solaris and post my results.

      Kerry

    7. osborne says:

      It works fine on Solaris 10 with a 10.2.0.4 database. Sets the processes to RT scheduling class and 100 priority.

      Kerry

    8. Riyaj says:

      Hi Kerry

      Nice post!

      To Tanel’s and Bob Sneed’s point, RT cpu priority is kernel preemptive. LGWR can block the interrupts being serviced by executing in RT mode in the CPU. LGWR needs those interrupts to be serviced and those interrupts quite probably could be I/O completion interrupts for LGWR to complete commit/log file sync processing. This issue of waiter blocking the interrupts server leads to an incorrect priority anomaly, analogous to a waiter of a resource evicting the holder of a resource off the CPU.

      On the other hand, in RAC, LMS processes will wait for LGWR to complete ‘log file sync’ if the block to be transferred is considered “busy”. But, LMS is also running in RT mode. So if the LGWR process is not running in RT mode, then the LMS process can evict LGWR from the CPU. This also leads to incorrect priority anomaly.

      If there are enough CPUs, in RAC, I would prefer, to run both LGWR and LMS* processes in RT mode (and VKTM in 11g). Then fence the interrupts to a different processor set so that LGWR/LMS* can not block those interrupts. In this way we should be able to keep interrupts/LMS/LGWR priorities straight. [At the very least, run LGWR in FX 60 in Solaris so that LGWR does not suffer from TS class scheduling latencies.]

      It is important to also realize that, default number of LMS processes are quite excessive. For example, one of my client site had 28 LMS processes and all 28 of them running in RT mode! You can imagine how much CPU usage will be used by those LMS processes. In my opinion, in most cases, 3 or 5 LMS processes per instance is good enough to service GC traffic. So, if the number of LMS processes are excessive, then it is probably worthwhile to reduce that and then also increase LGWR priority.

      Last but not least, I still fail to see a point of running VKTM in RT mode!

      Thanks

      Riyaj

    9. Noons says:

      Found something very interesting. Remember when I asked what priority level did this set processes to?
      Well, it turns out it’s nothing to do with that. This parameter raises the processes to real time priority class, not level. What controls the level is (yet!) another hidden parameter:
      _os_sched_high_priority
      For complete reference, go to MOS and search for “602419.1″.
      From the result set, pick the
      “LMS not running in RT (real time) mode in 10.2.0.3 RAC database” link.
      It’s all explained there.
      See what you done? Now I *must* test all these things!!!!
      :D

    10. osborne says:

      Noons,

      Interesting. The _os_sched_high_priority parameter appears to just ratchet the priority up within the RT/RR class. On linux with 10.2.0.4, a vlue of 1 = priority of 41, 2 = 42, 10=50, 20=60, etc… I think once you get into the Real Time class that moving the priority up will have little value. At that point I think you’d need to follow Riyaj’s suggestion regarding interrupt fencing / isolating LGWR. Thanks for pointing out the parameter and the Metalink reference.

      Kerry

    11. osborne says:

      Riyaj,

      Thanks for your detailed comments. Sorry about the trouble getting the site to accept them.

      Kerry

    12. Damir Vadas says:

      Hi!

      I have situation like in Metalink note but mine owners are OK.

      old 9: and a.ksppinm like ‘\_%&1%’ escape ‘\’
      new 9: and a.ksppinm like ‘\_%os_sched_high%’ escape ‘\’

      NAME VALUE DEFLT TYPE DESCRIPTION
      ————————- ————— —– ——– ———————————–
      _os_sched_high_priority 1 TRUE number OS high priority level

      SQL> !
      [oracle HAC4@iishacrac4 ~]$ ps -efc | grep lms
      oracle 5566 1 RR 41 14:12 ? 00:00:05 ora_lms0_HAC4
      oracle 5570 1 RR 41 14:12 ? 00:00:05 ora_lms1_HAC4
      oracle 17477 17425 TS 21 14:29 pts/0 00:00:00 grep lms
      oracle 32708 1 RR 41 Apr02 ? 00:00:55 asm_lms0_+ASM4

      [oracle HAC4@iishacrac4 ~]$ ll $ORACLE_HOME/bin/oradism
      -r-sr-s— 1 root dba 20062 Sep 11 2008 /u01/HAC/hacdb/10.2.0/bin/oradism

      How to fix LMS process to run in RR? I’m on Linux IA64and RHEL EE 5.3 so there is no support for this combination…even on Metalink!
      I’m suffering node eviction without reason. After I set digiwait and make some configuration changes …now in last 10 days there is no eviction…even on heavy stress (800 sql connections)…
      So I’m pretty desperate…in searching for the reason…

      THX for any help or comment
      Damir Vadas

    13. osborne says:

      Hi Damir,

      I’m not sure I’m understanding your question. Looks like from the output above that your lms processes are running in RR class at priority 41.

      oracle 5566 1 RR 41 14:12 ? 00:00:05 ora_lms0_HAC4
      oracle 5570 1 RR 41 14:12 ? 00:00:05 ora_lms1_HAC4

      Have you opened an SR?

      Kerry

    14. Damir Vadas says:

      Hi Kerry,

      SR is opened. Sent RDA reports and waiting on their answer. For now they say it looks like a new bug ….

      When find out let you know here…

      Rg
      Damir

    15. [...] 10-How to increase process priority within Oracle with _high_priority_processes? Kerry Osborne-Increasing Priority of lgwr Process using _high_priority_processes [...]

    16. Damir Vadas says:

      Hi!

      SR is closed as unresolved and there is nothing else to do.
      Main problem from mine point of view is Intel Itanium CPU, configuration that is really rare, so I think there is no testing configuration on Oracle Technical support side.
      Rg,
      Damir Vadas

    17. osborne says:

      Well there are advantages to staying in the middle of the pack.

      Kerry

    18. [...] are not preempted by lower priority processes. We know that since 10g some of Oracle processes have Real-Time priority. By default only LMS and VKTM processes scheduled in RT class. These processes do not use mutexes. [...]

    19. osborne says:

      By the way, I was asked if there was a reason to increase priority on pmon. The answer is no. It is just an example. I’m not suggesting to increase pmon priority to improve log file sync times.

    20. Tri Tuc says:

      Hi Kerry,

      Our system is Rac 3 nodes 10.2.0.4

      In Top 5 Timed Events

      gcs log flush sync 2,324,321 2,311 1 8.6 Other
      gc cr block busy 308,219 2,079 7 7.7 Cluster

      cpu_count = 128
      _high_priority_processes LMS* LMS*

      –> do i need
      alter system set “_high_priority_processes”=”LMS*|LGWR*” scope=spfile sid=’*’;

      if “yes”, do you have any recommendation for me.

      Thanks

    21. osborne says:

      Tri Tuc,

      The answer is maybe it could help. Those wait events are related to moving blocks between RAC nodes, so the best resolution would be to figure out why so many blocks have to move (as opposed to just rtying to make the move faster). Maybe some sort of workload partitioning would work better.

      Kerry

    22. [...] increase LGWR priority(renice or _high_priority_processes), [...]

    Leave a Reply