Kerry Osborne's Oracle Blog

This topic keeps coming up so I decided to write a little blurb on it. Mainly so I could stop repeating myself (I think I’ve written the same email about 10 times now). So here goes:

First the short version:

• If you are on 10gR2 or later you should probably not set DB_FILE_MULTIBLOCK_READ_COUNT.

Now the long winded discussion:

A fairly significant change to the Oracle optimizer was rolled out in 10gR2 with regards to the parameter DB_FILE_MULTIBLOCK_READ_COUNT (from here on out I’ll just refer to as DBFMBRC as it’s too long to type). Prior to 10gR2, this parameter was used for multiple purposes – namely:

• Representing the avg number of blocks that a multi-block read would return (used in the costing estimate when parsing)
• The number of blocks to attempt to read when a multi-block read is executed

Unfortunately, these two uses often worked against each other. Increasing the DBFMBRC in order to improve the throughput of multi-block reads generated by operations such as full table scans, would have the side effect of causing the optimizer to favor access paths that resulted in multi-block reads. I think that was probably the main driver for the introduction of the OPTIMIZER_INDEX_COST_ADJ and OPTIMIZER_INDEX_CACHING parameters – which allowed the optimizer’s preference for full table scans to be dialed down, but that’s a topic for another post.

The costing value and the execution value need not necessarily be related (and in 10gR2 they aren’t unless you make it so). For example, the estimate for costing purposes could be set to a relatively low value (so that the optimizer would not unduly favor multi-block read type operations) while the value used when actually requesting a multi-block i/o could be set relatively high (to maximize throughput in cases where a multi-block read was actually performed). 10gR2 accomplishes this by adding a couple of so called hidden parameters (you know, the one’s that start with an underscore). The parameters are:

• _db_file_optimizer_read_count – this is the one that’s used for costing
• _db_file_exec_read_count – this is the one that’s used when an i/o request is issued

If you leave DB_FILE_MULTIBLOCK_READ_COUNT set to its default value, these two parameters will be set to two different values. For example, on my 10.2.0.4 database using 8K blocks, the values are 8 and 128 respectively. So the optimizer should do it’s costing with a value of 8 but attempt to read 128 blocks, if it does indeed choose a full table scan. Let’s take a look:

> !sql
sqlplus "/ as sysdba"

SQL*Plus: Release 10.2.0.4.0 - Production on Wed Jan 13 17:21:55 2010

Copyright (c) 1982, 2007, Oracle.  All Rights Reserved.


Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options

SYS@LAB1024> select     pname, pval1
  2  from       sys.aux_stats$
  3  /

PNAME                               PVAL1
------------------------------ ----------
STATUS
DSTART
DSTOP
FLAGS                                   1
CPUSPEEDNW                     526.069569
IOSEEKTIM                              10
IOTFRSPEED                           4096
SREADTIM
MREADTIM
CPUSPEED
MBRC
MAXTHR
SLAVETHR

13 rows selected.

SYS@LAB1024> -- default values
SYS@LAB1024> 
SYS@LAB1024> select name, value, isdefault, ismodified, isset
  2  from
  3  (
  4  select flag,name,value,isdefault,ismodified,
  5  case when isdefault||ismodified = 'TRUEFALSE' then 'FALSE' else 'TRUE' end isset
  6  from
  7     (
  8         select
  9              decode(substr(i.ksppinm,1,1),'_',2,1) flag
 10              , i.ksppinm name
 11              , sv.ksppstvl value
 12              , sv.ksppstdf  isdefault
 13  --            , decode(bitand(sv.ksppstvf,7),1,'MODIFIED',4,'SYSTEM_MOD','FALSE') ismodified
 14              , decode(bitand(sv.ksppstvf,7),1,'TRUE',4,'TRUE','FALSE') ismodified
 15           from x$ksppi  i
 16              , x$ksppsv sv
 17          where i.indx = sv.indx
 18     )
 19  )
 20  where name like nvl('%¶meter%',name)
 21  and upper(isset) like upper(nvl('%&isset%',isset))
 22  and flag not in (decode('&show_hidden','Y',3,2))
 23  order by flag,replace(name,'_','')
 24  /
Enter value for parameter: read_count
Enter value for isset: 
Enter value for show_hidden: Y

NAME                                               VALUE                                                                  ISDEFAUL ISMODIFIED ISSET
-------------------------------------------------- ---------------------------------------------------------------------- -------- ---------- ----------
db_file_multiblock_read_count                      128                                                                    TRUE     FALSE      FALSE
_db_file_exec_read_count                           128                                                                    TRUE     FALSE      FALSE
_db_file_noncontig_mblock_read_count               11                                                                     TRUE     FALSE      FALSE
_db_file_optimizer_read_count                      8                                                                      TRUE     FALSE      FALSE
_sort_multiblock_read_count                        2                                                                      TRUE     FALSE      FALSE

SYS@LAB1024> -- default values again
SYS@LAB1024> 
SYS@LAB1024> -- let's turn on 10046 trace
SYS@LAB1024> 
SYS@LAB1024> set echo on 
SYS@LAB1024> @find_trace
SYS@LAB1024> col tracefile_name for a120
SYS@LAB1024> SELECT rtrim(k.value,'/')||'/'||LOWER(d.instance_name)||'_ora_'||p.spid
  2  ||DECODE(p.value,'','','_'||p.value)||'.trc' tracefile_name
  3  FROM v$parameter k, v$parameter p, v$instance d,
  4       sys.v_$session s, sys.v_$process p,
  5       (SELECT sid FROM v$mystat WHERE rownum=1) m
  6  WHERE p.name = 'tracefile_identifier'
  7    AND k.name = 'user_dump_dest'
  8    AND s.paddr = p.addr
  9    AND s.sid = m.sid
 10  /

TRACEFILE_NAME
------------------------------------------------------------------------------------------------------------------------
/u01/app/admin/LAB1024/udump/lab1024_ora_24004.trc


SYS@LAB1024> alter session set events '10046 trace name context forever, level 8';

Session altered.

SYS@LAB1024> set echo off
SYS@LAB1024> select avg(pk_col) from kso.skew a where col1 > 0;
select avg(pk_col) from kso.skew a where col1 > 0
                            *
ERROR at line 1:
ORA-01013: user requested cancel of current operation


SYS@LAB1024> -- bombed out with ctl-C 
SYS@LAB1024> -- now find the sql_id and verify the plan
SYS@LAB1024>
SYS@LAB1024> @find_sql
Enter value for sql_text: select avg(pk_col) from kso.skew a where col1 > 0
Enter value for sql_id: 

SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
05cq2hb1r37tr      0  568322376          1        183.30       35,082 select avg(pk_col) from kso.skew a where col1 > 0


SYS@LAB1024> @dplan
Enter value for sql_id: 05cq2hb1r37tr
Enter value for child_no: 0

PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  05cq2hb1r37tr, child number 1
-------------------------------------
select avg(pk_col) from kso.skew a where col1 > 0

Plan hash value: 568322376

---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |       |       | 45254 (100)|          |
|   1 |  SORT AGGREGATE    |      |     1 |    11 |            |          |
|*  2 |   TABLE ACCESS FULL| SKEW |    32M|   335M| 45254   (3)| 00:09:04 |
---------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

   2 - filter("COL1">0)

SYS@LAB1024> -- we can verify that it read 128 blocks at a time by looking at the trace file
SYS@LAB1024>
SYS@LAB1024> !cat /u01/app/admin/LAB1024/udump/lab1024_ora_24004.trc

...

PARSING IN CURSOR #6 len=49 dep=0 uid=0 oct=3 lid=0 tim=1233806813773567 hv=3279003447 ad='7e81b9b4'
select avg(pk_col) from kso.skew a where col1 > 0
END OF STMT
PARSE #6:c=213967,e=209976,p=27,cr=287,cu=0,mis=1,r=0,dep=0,og=2,tim=1233806813773550
EXEC #6:c=0,e=77,p=0,cr=0,cu=0,mis=0,r=0,dep=0,og=2,tim=1233806813773842
WAIT #6: nam='SQL*Net message to client' ela= 10 driver id=1650815232 #bytes=1 p3=0 obj#=472 tim=1233806813773941
WAIT #6: nam='db file sequential read' ela= 81 file#=4 block#=181290 blocks=1 obj#=54141 tim=1233806813791873
WAIT #6: nam='db file scattered read' ela= 5218 file#=4 block#=181291 blocks=128 obj#=54141 tim=1233806813800264
WAIT #6: nam='db file scattered read' ela= 3239 file#=4 block#=181419 blocks=128 obj#=54141 tim=1233806814404594

Note 1: Very nice! Maximum i/o without making the optimizer think that full table scans are better than index access paths.

Note 2: Setting the parameter to the default value is not the same as resetting it. If you were to change the value to 16 for example and then set it back to 8 (which was the value it defaulted to) you would see the costing and the execution done with 8. That is, the _db_file_optimizer_read_count and the _db_file_exec_read_count would both be set to 8. You have to unset the DBFMBRC parameter to get it to go back to automatic split of 8 and 128. See my previous post on that here: Resetting Oracle Parameters Here’s another quick demonstration:

SYS@LAB1024> @parms
Enter value for parameter: read_count
Enter value for isset: 
Enter value for show_hidden: Y

NAME                                               VALUE                                                                  ISDEFAUL ISMODIFIED ISSET
-------------------------------------------------- ---------------------------------------------------------------------- -------- ---------- ----------
db_file_multiblock_read_count                      128                                                                    TRUE     FALSE      FALSE
_db_file_exec_read_count                           128                                                                    TRUE     FALSE      FALSE
_db_file_noncontig_mblock_read_count               11                                                                     TRUE     FALSE      FALSE
_db_file_optimizer_read_count                      8                                                                      TRUE     FALSE      FALSE
_sort_multiblock_read_count                        2                                                                      TRUE     FALSE      FALSE

SYS@LAB1024> alter system set db_file_multiblock_read_count=16;

System altered.

SYS@LAB1024> @parms
Enter value for parameter: read_count
Enter value for isset: 
Enter value for show_hidden: Y

NAME                                               VALUE                                                                  ISDEFAUL ISMODIFIED ISSET
-------------------------------------------------- ---------------------------------------------------------------------- -------- ---------- ----------
db_file_multiblock_read_count                      16                                                                     TRUE     TRUE       TRUE
_db_file_exec_read_count                           16                                                                     TRUE     FALSE      FALSE
_db_file_noncontig_mblock_read_count               11                                                                     TRUE     FALSE      FALSE
_db_file_optimizer_read_count                      16                                                                     TRUE     FALSE      FALSE
_sort_multiblock_read_count                        2                                                                      TRUE     FALSE      FALSE

SYS@LAB1024> alter system set db_file_multiblock_read_count=128;

System altered.

SYS@LAB1024> @parms
Enter value for parameter: read_count
Enter value for isset: 
Enter value for show_hidden: Y

NAME                                               VALUE                                                                  ISDEFAUL ISMODIFIED ISSET
-------------------------------------------------- ---------------------------------------------------------------------- -------- ---------- ----------
db_file_multiblock_read_count                      128                                                                    TRUE     TRUE       TRUE
_db_file_exec_read_count                           128                                                                    TRUE     FALSE      FALSE
_db_file_noncontig_mblock_read_count               11                                                                     TRUE     FALSE      FALSE
_db_file_optimizer_read_count                      128                                                                    TRUE     FALSE      FALSE
_sort_multiblock_read_count                        2                                                                      TRUE     FALSE      FALSE

SYS@LAB1024> alter system set db_file_multiblock_read_count=0;

System altered.

SYS@LAB1024> @parms
Enter value for parameter: read_count
Enter value for isset: 
Enter value for show_hidden: Y

NAME                                               VALUE                                                                  ISDEFAUL ISMODIFIED ISSET
-------------------------------------------------- ---------------------------------------------------------------------- -------- ---------- ----------
db_file_multiblock_read_count                      128                                                                    TRUE     FALSE      FALSE
_db_file_exec_read_count                           128                                                                    TRUE     FALSE      FALSE
_db_file_noncontig_mblock_read_count               11                                                                     TRUE     FALSE      FALSE
_db_file_optimizer_read_count                      128                                                                    TRUE     FALSE      FALSE
_sort_multiblock_read_count                        2                                                                      TRUE     FALSE      FALSE

So once you set the value of DBFMBRC, the exec and read parms will be the same.

Note 3: The _db_file_noncontig_mblock_read_count has to do with index block pre-fetching. There is an excellent description on this oracle-l post by Tanel Poder

Note 4: Everything I’ve said so far assumes that System Statistics have not been set or gathered. If System Stats are set, then a completely different set of rules kick in – basically DBFMBRC is ignored.

And speaking of System Stats, I must say that I think the default no-workload System Stats work pretty well. In most cases, I prefer not to gather or set System Statistics, for a couple of reasons:

In the first place, I generally don’t believe in fixing things that aren’t broken. You could argue that System Statistics are not meant to “fix” anything, but are merely designed to give the optimizer more specific information about the particular system. And this is true, but the standard method of gathering Workload System Statistics often results in less than accurate results and at best they only represent an average over some period of time which may or may not be representative of what happens when the system is really under stress. And I guess we could have a few sets of values and change them regularly (like one set for batch processing and one for day time on-line processing), but that seems like a lot of complexity for little potential benefit. So unless there really is a specific problem that needs to be solved, I probably wouldn’t do that (in fact I have not ever done that).

Second, I would venture to guess that the majority of databases out there have not gathered or set System Statistics. I could be way off on that assumption, but most of the systems that I’ve looked at over the last couple of years have not had anything other than the default NOWORKLOAD values. Staying in the middle of the pack means we’re less likely to run into weird behavior that someone else hasn’t already run across.

Third, the interaction between the various flavors of System Statistics and the DBFMBRC are complicated. Randolf Geist has an excellent series of posts which discusses the interaction between System Stats and DFMBRC here: RG on System Stats
But it takes 3 fairly long and detailed posts to cover the topic. I am not going to discuss it here since he has already done a very thorough job.

Fourth, System Statistics is (are) a really big knob. What I mean by that is that a small change can have a profound effect. Since they control the costing of every single statement that runs through your system, changes to them could potentially affect every statement. So from a stability stand point, this is not something that you would want to change without careful thought and certainly not something that you would want to change on a regular basis.

Now for a couple of references:

Jonathan Lewis has several posts on his blog about System Stats. Here are links to a couple:

JL on System Stats
JL on System Stats 2

Be sure and read through the comments as there is some excellent information there including how the default values for the read and exec parameters are arrived at.

Finally, Tom Kyte has a good discussion on the subject here: TK on DBFMBRC