Kerry Osborne's Oracle Blog http://kerryosborne.oracle-guy.com Just another Oracle blog Mon, 16 Jan 2012 17:12:30 +0000 en hourly 1 http://wordpress.org/?v=3.2.1 C. J. Date Speaking in Dallas http://kerryosborne.oracle-guy.com/2012/01/c-j-date-speaking-in-dallas/ http://kerryosborne.oracle-guy.com/2012/01/c-j-date-speaking-in-dallas/#comments Tue, 10 Jan 2012 00:56:31 +0000 osborne http://kerryosborne.oracle-guy.com/?p=3806 Chris Date is one of the founding fathers of relational databases. Having worked with Ted Codd at IBM during the time when relational databases were being defined gives Chris a perspective that most of us just don’t have. I’ve had the good fortune to hear him speak in the past (at the Hotsos Symposium) and thought I would do a quick post to highlight the fact that he is scheduled to speak in Dallas the week of Jan 30. Method-R is hosting the event in the Enkitec training facilities in Dallas. So maybe I’ll get to hang around with Chris and Cary that week – that would be cool! Anyway, there are actually 2 classes:

SQL and Relational Theory: How to Write Accurate SQL Code
Normal Forms and All That Jazz: a Database Professional’s Guide to Database Design Theory

And here’s a link to the registration page:   C. J. Date Seminar Registration

By the way, I think every developer and every database architect should have a clear understanding of how the SQL language is designed to work and how relational databases were intended to be laid out. Chris obviously has a unique insight into those topics. One of the tenants of Chris’s teaching is that SQL is a complicated language and since comprehensive testing is almost never really feasible, it is important to write SQL using a disciplined approach based on the underlying relational theory. As a side note, I was talking to a few cohorts around the coffee pot today and was shocked to hear that one of the guys had a CS degree but was not required to take a relational theory class. Back when I got started that was the first class that people took, probably because there were almost no real implementations of the theory at that point. Oracle was just getting started and DB2 was still a distant gleam in Mr. Codd’s eye. But I digress.

It does seem to me that we have an awful lot of systems running on Oracle these days that were designed and written by people without a strong background in relational database fundamentals. I can’t even begin to count the number of times I’ve worked on systems that performed poorly due to poor SQL coding techniques and/or poor database design. Chris’s courses are designed to help you avoid these issues. So this is your chance to learn how to know for sure that your SQL is correct.

Hope to you see you there!

 

 

]]> http://kerryosborne.oracle-guy.com/2012/01/c-j-date-speaking-in-dallas/feed/ 2 Christmas Explain Plan Patterns http://kerryosborne.oracle-guy.com/2011/12/christmas-explain-plan-patterns/ http://kerryosborne.oracle-guy.com/2011/12/christmas-explain-plan-patterns/#comments Thu, 08 Dec 2011 20:44:37 +0000 osborne http://kerryosborne.oracle-guy.com/?p=3790 Here’s a lovely Candy Striped pattern in an Explain Plan. Looks like the traditional Christmas candy canes. Just in time for the Holiday Season!

?View Code NONE
| 156 |     FAST DUAL                                                |                                |     1 |       |     2   (0)| 00:00:01 |
| 157 |      FAST DUAL                                               |                                |     1 |       |     2   (0)| 00:00:01 |
| 158 |       FAST DUAL                                              |                                |     1 |       |     2   (0)| 00:00:01 |
| 159 |        FAST DUAL                                             |                                |     1 |       |     2   (0)| 00:00:01 |
| 160 |         FAST DUAL                                            |                                |     1 |       |     2   (0)| 00:00:01 |
| 161 |          FAST DUAL                                           |                                |     1 |       |     2   (0)| 00:00:01 |
| 162 |           FAST DUAL                                          |                                |     1 |       |     2   (0)| 00:00:01 |
| 163 |            FAST DUAL                                         |                                |     1 |       |     2   (0)| 00:00:01 |
| 164 |             FAST DUAL                                        |                                |     1 |       |     2   (0)| 00:00:01 |
| 165 |              FAST DUAL                                       |                                |     1 |       |     2   (0)| 00:00:01 |
| 166 |               FAST DUAL                                      |                                |     1 |       |     2   (0)| 00:00:01 |
| 167 |                FAST DUAL                                     |                                |     1 |       |     2   (0)| 00:00:01 |
| 168 |                 FAST DUAL                                    |                                |     1 |       |     2   (0)| 00:00:01 |
| 169 |                  FAST DUAL                                   |                                |     1 |       |     2   (0)| 00:00:01 |
| 170 |                   FAST DUAL                                  |                                |     1 |       |     2   (0)| 00:00:01 |
| 171 |                    FAST DUAL                                 |                                |     1 |       |     2   (0)| 00:00:01 |
| 172 |                     FAST DUAL                                |                                |     1 |       |     2   (0)| 00:00:01 |
| 173 |                      FAST DUAL                               |                                |     1 |       |     2   (0)| 00:00:01 |
| 174 |                       FAST DUAL                              |                                |     1 |       |     2   (0)| 00:00:01 |
| 175 |                        FAST DUAL                             |                                |     1 |       |     2   (0)| 00:00:01 |
| 176 |                         FAST DUAL                            |                                |     1 |       |     2   (0)| 00:00:01 |
| 177 |                          FAST DUAL                           |                                |     1 |       |     2   (0)| 00:00:01 |
| 178 |                           FAST DUAL                          |                                |     1 |       |     2   (0)| 00:00:01 |
| 179 |                            FAST DUAL                         |                                |     1 |       |     2   (0)| 00:00:01 |
| 180 |                             FAST DUAL                        |                                |     1 |       |     2   (0)| 00:00:01 |
| 181 |                              FAST DUAL                       |                                |     1 |       |     2   (0)| 00:00:01 |
| 182 |                               FAST DUAL                      |                                |     1 |       |     2   (0)| 00:00:01 |
| 183 |                                FAST DUAL                     |                                |     1 |       |     2   (0)| 00:00:01 |
| 184 |                                 FAST DUAL                    |                                |     1 |       |     2   (0)| 00:00:01 |
| 185 |                                  FAST DUAL                   |                                |     1 |       |     2   (0)| 00:00:01 |
| 186 |                                   FAST DUAL                  |                                |     1 |       |     2   (0)| 00:00:01 |
| 187 |                                    FAST DUAL                 |                                |     1 |       |     2   (0)| 00:00:01 |
| 188 |                                     FAST DUAL                |                                |     1 |       |     2   (0)| 00:00:01 |
| 189 |                                      FAST DUAL               |                                |     1 |       |     2   (0)| 00:00:01 |
| 190 |                                       FAST DUAL              |                                |     1 |       |     2   (0)| 00:00:01 |
| 191 |                                        FAST DUAL             |                                |     1 |       |     2   (0)| 00:00:01 |
| 192 |                                         FAST DUAL            |                                |     1 |       |     2   (0)| 00:00:01 |
| 193 |                                          FAST DUAL           |                                |     1 |       |     2   (0)| 00:00:01 |
| 194 |                                           FAST DUAL          |                                |     1 |       |     2   (0)| 00:00:01 |
| 195 |                                            FAST DUAL         |                                |     1 |       |     2   (0)| 00:00:01 |
| 196 |                                             FAST DUAL        |                                |     1 |       |     2   (0)| 00:00:01 |
| 197 |                                              FAST DUAL       |                                |     1 |       |     2   (0)| 00:00:01 |
| 198 |                                               FAST DUAL      |                                |     1 |       |     2   (0)| 00:00:01 |
| 199 |                                                FAST DUAL     |                                |     1 |       |     2   (0)| 00:00:01 |
| 200 |                                                 FAST DUAL    |                                |     1 |       |     2   (0)| 00:00:01 |
| 201 |                                                  FAST DUAL   |                                |     1 |       |     2   (0)| 00:00:01 |
| 202 |                                                   FAST DUAL  |                                |     1 |       |     2   (0)| 00:00:01 |
| 203 |                                                    FAST DUAL |                                |     1 |       |     2   (0)| 00:00:01 |
| 204 | L                                                   FAST DUA |                                |     1 |       |     2   (0)| 00:00:01 |
| 205 | AL                                                   FAST DU |                                |     1 |       |     2   (0)| 00:00:01 |
| 206 | UAL                                                   FAST D |                                |     1 |       |     2   (0)| 00:00:01 |
| 207 | DUAL                                                   FAST  |                                |     1 |       |     2   (0)| 00:00:01 |
| 208 |  DUAL                                                   FAST |                                |     1 |       |     2   (0)| 00:00:01 |
| 209 | T DUAL                                                   FAS |                                |     1 |       |     2   (0)| 00:00:01 |
| 210 | ST DUAL                                                   FA |                                |     1 |       |     2   (0)| 00:00:01 |
| 211 | AST DUAL                                                   F |                                |     1 |       |     2   (0)| 00:00:01 |
| 212 |.FAST DUAL                                                    |                                |     1 |       |     2   (0)| 00:00:01 |
| 213 |. FAST DUAL                                                   |                                |     1 |       |     2   (0)| 00:00:01 |
| 214 |.  FAST DUAL                                                  |                                |     1 |       |     2   (0)| 00:00:01 |
| 215 |.   FAST DUAL                                                 |                                |     1 |       |     2   (0)| 00:00:01 |
| 216 |.    FAST DUAL                                                |                                |     1 |       |     2   (0)| 00:00:01 |
| 217 |.     FAST DUAL                                               |                                |     1 |       |     2   (0)| 00:00:01 |
| 218 |.      FAST DUAL                                              |                                |     1 |       |     2   (0)| 00:00:01 |
| 219 |.       FAST DUAL                                             |                                |     1 |       |     2   (0)| 00:00:01 |
| 220 |.        FAST DUAL                                            |                                |     1 |       |     2   (0)| 00:00:01 |
| 221 |.         FAST DUAL                                           |                                |     1 |       |     2   (0)| 00:00:01 |
| 222 |.          FAST DUAL                                          |                                |     1 |       |     2   (0)| 00:00:01 |
| 223 |.           FAST DUAL                                         |                                |     1 |       |     2   (0)| 00:00:01 |
| 224 |.            FAST DUAL                                        |                                |     1 |       |     2   (0)| 00:00:01 |
| 225 |.             FAST DUAL                                       |                                |     1 |       |     2   (0)| 00:00:01 |
| 226 |.              FAST DUAL                                      |                                |     1 |       |     2   (0)| 00:00:01 |
| 227 |.               FAST DUAL                                     |                                |     1 |       |     2   (0)| 00:00:01 |
| 228 |.                FAST DUAL                                    |                                |     1 |       |     2   (0)| 00:00:01 |
| 229 |.                 FAST DUAL                                   |                                |     1 |       |     2   (0)| 00:00:01 |
| 230 |.                  FAST DUAL                                  |                                |     1 |       |     2   (0)| 00:00:01 |
| 231 |.                   FAST DUAL                                 |                                |     1 |       |     2   (0)| 00:00:01 |
| 232 |.                    FAST DUAL                                |                                |     1 |       |     2   (0)| 00:00:01 |
| 233 |.                     FAST DUAL                               |                                |     1 |       |     2   (0)| 00:00:01 |
| 234 |.                      FAST DUAL                              |                                |     1 |       |     2   (0)| 00:00:01 |
| 235 |.                       FAST DUAL                             |                                |     1 |       |     2   (0)| 00:00:01 |
]]> http://kerryosborne.oracle-guy.com/2011/12/christmas-explain-plan-patterns/feed/ 6 Tuning Oracle to Make a Query Slower http://kerryosborne.oracle-guy.com/2011/11/tuning-oracle-to-make-a-query-slower/ http://kerryosborne.oracle-guy.com/2011/11/tuning-oracle-to-make-a-query-slower/#comments Wed, 23 Nov 2011 19:15:20 +0000 osborne http://kerryosborne.oracle-guy.com/?p=3760 I had an interesting little project this morning. Of course it takes longer to write it down than to do actually do it, but it was kind of interesting and since I haven’t done a post in quite some time (and it’s the day before Thanksgiving, so it’s pretty quite at the office anyway) I decided to share. One of the Enkitec guys (Tim Fox) was doing a performance comparison between various platforms (Exadata using it’s IB Storage Network, Oracle Database Appliance (ODA) using it’s direct attached storage, and a standard database on a Dell box using EMC fiber channel attached storage). The general test idea was simple – see how the platforms stacked up for a query that required a full scan of a large table. More specifically, what Tim wanted to see was the relative speed at which the various storage platforms could return data. The expectation was that the direct attached storage would be fastest and the fibre channel storage would be slowest (especially since we only had a single 2G HBA). He tested ODA and Exadata and got basically what he expected, but when he went to test the database on the Dell he was surprised that it was actually faster than either of the other two tests. So here’s some output from the initial tests: First the Exadata. It’s an X2 quarter rack with one extra storage server. Note that we had to set cell_offload_processing to false to turn off the Exadata storage optimizations, thus giving us a measurement of the hardware capabilities without the Exadata offloading.

?View Code NONE
> !sqlp
sqlp
 
SQL*Plus: Release 11.2.0.2.0 Production on Wed Nov 23 11:08:28 2011
 
Copyright (c) 1982, 2010, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
 
SYS@DEMO1> @uptime
 
INSTANCE_NAME    STARTUP_TIME      CURRENT_TIME         DAYS    SECONDS
---------------- ----------------- ----------------- ------- ----------
DEMO1            07-NOV-2011 12:37 23-NOV-2011 11:08   15.94    1377058
 
SYS@DEMO1> set sqlprompt "_USER'@'EXADATA'>' "
SYS@EXADATA> 
SYS@EXADATA> ! cat /etc/redhat-release
Enterprise Linux Enterprise Linux Server release 5.5 (Carthage)
 
SYS@EXADATA> ! uname -a
Linux enkdb03.enkitec.com 2.6.18-194.3.1.0.3.el5 #1 SMP Tue Aug 31 22:41:13 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
 
SYS@EXADATA> alter session set "_serial_direct_read"=always;
 
Session altered.
 
SYS@EXADATA> alter session set cell_offload_processing=false;
 
Session altered.
 
SYS@EXADATA> set autotrace on
SYS@EXADATA> set timing on
SYS@EXADATA> select count(*) from instructor.class_sales;
 
  COUNT(*)
----------
  90000000
 
Elapsed: 00:00:43.01
 
Execution Plan
----------------------------------------------------------
Plan hash value: 3145879882
 
----------------------------------------------------------------------------------
| Id  | Operation                  | Name        | Rows  | Cost (%CPU)| Time     |
----------------------------------------------------------------------------------
|   0 | SELECT STATEMENT           |             |     1 |   314K  (1)| 00:00:02 |
|   1 |  SORT AGGREGATE            |             |     1 |            |          |
|   2 |   TABLE ACCESS STORAGE FULL| CLASS_SALES |    90M|   314K  (1)| 00:00:02 |
----------------------------------------------------------------------------------
 
 
Statistics
----------------------------------------------------------
          1  recursive calls
          0  db block gets
    1168567  consistent gets
    1168557  physical reads
          0  redo size
        526  bytes sent via SQL*Net to client
        524  bytes received via SQL*Net from client
          2  SQL*Net roundtrips to/from client
          0  sorts (memory)
          0  sorts (disk)
          1  rows processed
 
SYS@EXADATA> set autotrace off
SYS@EXADATA> @fss
Enter value for sql_text: select count(*) from instructor.class_sales
Enter value for sql_id: 
 
SQL_ID         CHILD      EXECS   AVG_ROWS     AVG_ETIME       AVG_CPU       AVG_PIO      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------- ------------- ------------ ----------------------------------------
b2br1x82p9862      0          1          1         43.00          3.16  1,168,557.00    1,168,567 select count(*) from instructor.class_sa
 
Elapsed: 00:00:00.08

So the test on the Exadata took 43 seconds to read and transport roughly 1 million 8K blocks. The same test on the ODA looked like this:

?View Code NONE
[oracle@patty scripts]$ !rl rlwrap sqlplus / as sysdba  
 
SQL*Plus: Release 11.2.0.2.0 Production on Wed Nov 23 10:16:51 2011  
 
Copyright (c) 1982, 2010, Oracle.  All rights reserved.   
 
 
Connected to: 
Oracle Database 11g Enterprise Edition Release 11.2.0.2.0 - 64bit Production 
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP, 
Data Mining and Real Application Testing options   
 
INSTANCE_NAME    STARTUP_TIME               CURRENT_TIME                  DAYS    SECONDS 
---------------- -------------------------- -------------------------- ------- ---------- 
ODA1             22-NOV-2011 12:23          23-NOV-2011 10:16              .91      78793  
 
SYS@ODA1> ! cat /etc/redhat-release 
 
Red Hat Enterprise Linux Server release 5.5 (Tikanga)  
 
SYS@ODA1> ! uname -a 
 
Linux patty 2.6.18-194.32.1.0.1.el5 #1 SMP Tue Jan 4 16:26:54 EST 2011 x86_64 x86_64 x86_64 GNU/Linux  
 
SYS@ODA1> set timing on
 
SYS@ODA1> alter session set '_serial_direct_read'=always;  
 
Session altered.  
 
Elapsed: 00:00:00.00 
 
SYS@ODA1> set autotrace on 
SYS@ODA1> select count(*) from instructor.class_sales;    
 
COUNT(*) 
----------   
90000000  
Elapsed: 00:00:30.60  
 
Execution Plan 
---------------------------------------------------------- 
Plan hash value: 3145879882  
 
-------------------------------------------------------------------------- 
| Id  | Operation          | Name        | Rows  | Cost (%CPU)| Time     | 
-------------------------------------------------------------------------- 
|   0 | SELECT STATEMENT   |             |     1 |   317K  (1)| 00:00:05 | 
|   1 |  SORT AGGREGATE    |             |     1 |            |          | 
|   2 |   TABLE ACCESS FULL| CLASS_SALES |    90M|   317K  (1)| 00:00:05 | 
--------------------------------------------------------------------------   
 
 
Statistics 
----------------------------------------------------------          
        99  recursive calls           
         0  db block gets     
   1154080  consistent gets     
   1153994  physical reads         
       516  redo size         
       526  bytes sent via SQL*Net to client         
       524  bytes received via SQL*Net from client           
         2  SQL*Net roundtrips to/from client           
         7  sorts (memory)           
         0  sorts (disk)           
         1  rows processed  
 
SYS@ODA1> set autotrace off 
SYS@ODA1> @fss 
Enter value for sql_text: select count(*) from instructor.class_sales 
Enter value for sql_id:   
 
SQL_ID         CHILD PLAN_HASH_VALUE      EXECS ROWS_PROCESSED  AVG_ETIME    AVG_CPU    AVG_PIO    AVG_LIO SQL_TEXT 
------------- ------ --------------- ---------- -------------- ---------- ---------- ---------- ---------- ---------------------------------------- 
b2br1x82p9862      0      3145879882          1              1      30.55      13.91  1,153,994  1,154,080 select count(*) from instructor.class_sa

As expected, the direct attached disk was faster than moving the blocks across the IB network. It took about 30 seconds to read roughly the same number of blocks. The same test on the Dell produced this output:

?View Code NONE
 [osborne@homer scripts]$ sqlp  
 
SQL*Plus: Release 11.2.0.3.0 Production on Wed Nov 23 11:20:20 2011  
 
Copyright (c) 1982, 2011, Oracle.  All rights reserved.   
 
 
Connected to: 
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - 64bit Production 
With the Partitioning, OLAP, Data Mining and Real Application Testing options  
 
SYS@EXADATA> set sqlprompt "'_USER'@'DELL'>'" 
SYS@DELL> !cat /etc/redhat-release 
 
Red Hat Enterprise Linux Server release 5.5 (Tikanga)  
 
SYS@DELL> ! uname -a 
 
Linux homer.enkitec.com 2.6.18-194.el5 #1 SMP Mon Mar 29 22:10:29 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux  
 
SYS@DELL> alter session set '_serial_direct_read'=always;  
 
Session altered.  
 
SYS@DELL> set autotrace on 
 
SYS@DELL> set timing on 
 
SYS@DELL> select count(*) from instructor.class_sales;    
 
COUNT(*) 
----------   
90000000  
 
Elapsed: 00:00:11.31  
 
Execution Plan 
---------------------------------------------------------- 
Plan hash value: 3145879882  
 
-------------------------------------------------------------------------- 
| Id  | Operation          | Name        | Rows  | Cost (%CPU)| Time     | 
-------------------------------------------------------------------------- 
|   0 | SELECT STATEMENT   |             |     1 |   315K  (1)| 01:03:08 | 
|   1 |  SORT AGGREGATE    |             |     1 |            |          | 
|   2 |   TABLE ACCESS FULL| CLASS_SALES |    90M|   315K  (1)| 01:03:08 | 
--------------------------------------------------------------------------   
 
 
Statistics ----------------------------------------------------------          
        77  recursive calls  
         0  db block gets  
   1168660  consistent gets  
   1168569  physical reads  
         0  redo size  
       526  bytes sent via SQL*Net to client  
       524  bytes received via SQL*Net from client  
         2  SQL*Net roundtrips to/from client  
        10  sorts (memory)  
         0  sorts (disk)  
         1  rows processed  
 
SYS@DELL> set autotrace off 
SYS@DELL> @fss 
Enter value for sql_text: select count(*) from instructor.class_sales 
Enter value for sql_id:   
 
SQL_ID         CHILD      EXECS   AVG_ROWS     AVG_ETIME       AVG_CPU       AVG_PIO      AVG_LIO SQL_TEXT 
------------- ------ ---------- ---------- ------------- ------------- ------------- ------------ ---------------------------------------- 
b2br1x82p9862      0          1          1         11.29         11.25  1,168,569.00    1,168,660 select count(*) from instructor.class_sa  
 
Elapsed: 00:00:00.03

This is where the results were surprising. Tim expected the single 2G HBA to be considerably slower than the other two set ups, but it only took 11 seconds on the Dell / EMC set up to read the same data. So this is where I got to help Tim try to make it slower! (there’s a first time for everything) So what gives? The first thought was that Oracle was caching the data: But no – you’ll notice that we set the _serial_direct_read parameter to always, which forces direct path reads and bypasses the buffer cache altogether. Also, the stats bear out that Oracle is doing physical reads. But 1 Million real i/o’s can’t happen in 11 seconds so it’s got to be memory access right? (I’m sure you’re way ahead of me by now) The answer of course is "Right – it is memory access!" – but it’s the Linux file system cache not the Oracle buffer cache. A quick check of the Linux memory usage showed us that the file cache was over 20G and the table was less than 10G. So Oracle must be set up without Direct I/O.

?View Code NONE
SYS@DELL> !cat /proc/meminfo 
 
MemTotal:     32948892 kB 
MemFree:       2769420 kB 
Buffers:        615664 kB 
Cached:       24727756 kB 
SwapCached:     101844 kB 
Active:       21322184 kB 
Inactive:      5261948 kB 
HighTotal:           0 kB 
HighFree:            0 kB 
LowTotal:     32948892 kB 
LowFree:       2769420 kB 
SwapTotal:    34996216 kB 
SwapFree:     34147776 kB 
Dirty:             448 kB 
Writeback:           0 kB 
AnonPages:     1138828 kB 
Mapped:        4758828 kB 
Slab:          1088356 kB 
PageTables:     167464 kB 
NFS_Unstable:        0 kB 
Bounce:              0 kB 
CommitLimit:  51470660 kB 
Committed_AS: 15637340 kB 
VmallocTotal: 34359738367 kB 
VmallocUsed:    371108 kB 
VmallocChunk: 34359367199 kB 
HugePages_Total:     0 
HugePages_Free:      0 
HugePages_Rsvd:      0 
Hugepagesize:     2048 kB  
 
SYS@DELL> !realfreemem.sh -a 
 
Free Memory: 2703M 
Cached Memory: 24148M 
Total Free Memory: 26851M 
Total Memory: 32176M 
Percent Memory Free (including cache): 83%  
 
SYS@DELL> @table_size 
Enter value for owner: INSTRUCTOR 
Enter value for table_name: CLASS_SALES 
Enter value for type:   
 
OWNER                SEGMENT_NAME                   TYPE               TOTALSIZE_MEGS TABLESPACE_NAME 
-------------------- ------------------------------ ------------------ -------------- ------------------------------ 
INSTRUCTOR           CLASS_SALES                    TABLE                     9,151.0 ODACOMP_DATA 
                                                                       -------------- 
sum                                                                           9,151.0  
 
Elapsed: 00:00:00.04
 
SYS@DELL> -- so how is Oracle configured for I/O? 
SYS@DELL> @parms 
Enter value for parameter: filesys 
Enter value for isset:  
Enter value for show_hidden:   
 
NAME                                               VALUE                                                                  ISDEFAUL ISMODIFIED ISSET 
-------------------------------------------------- ---------------------------------------------------------------------- -------- ---------- ---------- 
filesystemio_options                               NONE                                                                   TRUE     FALSE      FALSE  
 
Elapsed: 00:00:00.01

So the database was not configured to use direct i/o or async i/o (filesystemio_options=none). A quick check with strace verified that direct i/o was not being used. So we modified the filesystemio_options setting and tried again.

?View Code NONE
SYS@DELL> alter system set filesystemio_options=setall scope=spfile;  
 
System altered.  
 
SYS@DELL> startup force 
ORACLE instance started.  
 
Total System Global Area 4910620672 bytes 
Fixed Size                  2236648 bytes 
Variable Size            2298482456 bytes 
Database Buffers         2600468480 bytes 
Redo Buffers                9433088 bytes 
Database mounted. Database opened. 
 
SYS@ODACOMP>  
SYS@ODACOMP> @uptime  
 
INSTANCE_NAME    STARTUP_TIME               CURRENT_TIME                  DAYS    SECONDS 
---------------- -------------------------- -------------------------- ------- ---------- 
ODACOMP          23-NOV-2011 12:45          23-NOV-2011 12:47              .00        121  
 
SYS@ODACOMP> set sqlprompt "'_USER'@'DELL'>'" 
SYS@DELL>
SYS@DELL> ! cat /etc/redhat-release 
 
Red Hat Enterprise Linux Server release 5.5 (Tikanga)  
 
SYS@DELL> ! uname -a 
 
Linux homer.enkitec.com 2.6.18-194.el5 #1 SMP Mon Mar 29 22:10:29 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux  
 
SYS@DELL> set timing on 
SYS@DELL> alter session set '_serial_direct_read'=always;  
 
Session altered.  
 
Elapsed: 00:00:00.00
 
SYS@DELL> set autotrace on 
 
SYS@DELL> select count(*) from instructor.class_sales;    
 
COUNT(*) 
----------   
90000000  
 
Elapsed: 00:01:39.73  
 
Execution Plan 
---------------------------------------------------------- 
Plan hash value: 3145879882  
 
-------------------------------------------------------------------------- 
| Id  | Operation          | Name        | Rows  | Cost (%CPU)| Time     | 
-------------------------------------------------------------------------- 
|   0 | SELECT STATEMENT   |             |     1 |   315K  (1)| 01:03:08 | 
|   1 |  SORT AGGREGATE    |             |     1 |            |          | 
|   2 |   TABLE ACCESS FULL| CLASS_SALES |    90M|   315K  (1)| 01:03:08 | 
--------------------------------------------------------------------------   
 
 
Statistics ----------------------------------------------------------  
         1  recursive calls  
         0  db block gets  
   1168567  consistent gets  
   1168557  physical reads  
         0  redo size  
       526  bytes sent via SQL*Net to client  
       524  bytes received via SQL*Net from client  
         2  SQL*Net roundtrips to/from client  
         0  sorts (memory)  
         0  sorts (disk)  
         1  rows processed  
 
SYS@DELL> set autotrace off 
SYS@DELL> @fss 
Enter value for sql_text: select count(*) from instructor.class_sales 
Enter value for sql_id:   
 
SQL_ID         CHILD      EXECS   AVG_ROWS     AVG_ETIME       AVG_CPU       AVG_PIO      AVG_LIO SQL_TEXT 
------------- ------ ---------- ---------- ------------- ------------- ------------- ------------ ---------------------------------------- 
b2br1x82p9862      0          1          1         99.71          7.32  1,168,557.00    1,168,567 select count(*) from instructor.class_sa  
 
Elapsed: 00:00:00.05

Ah that’s more like it. Roughly 100 seconds now to complete the query. That’s 10X slower. Nice job. It feels so good when you can make something go slower. ;) I should note that in general, it is a very good idea to set filesystemio_options=SETALL. So please don’t jump to the conclusion that setting this parameter to NONE will make your database run faster. These days, the memory is generally better used by Oracle than by the file system cache. Glenn Fawcett has a good (if somewhat dated) post which compares the effects of caching blocks in the file system cache vs. in the Oracle buffer cache in Solaris. As you might image, allowing Oracle to use the memory is generally more effective as shown in his results. So that’s it for today. By the way, this post took 4 times as long to write as the actual testing took. Maybe I’ll get better at it if I start practicing a little more often.

]]> http://kerryosborne.oracle-guy.com/2011/11/tuning-oracle-to-make-a-query-slower/feed/ 4 Oracle Database Appliance – (Baby Exadata?) http://kerryosborne.oracle-guy.com/2011/09/oracle-database-appliance/ http://kerryosborne.oracle-guy.com/2011/09/oracle-database-appliance/#comments Wed, 21 Sep 2011 20:59:19 +0000 osborne http://kerryosborne.oracle-guy.com/?p=3722 Oracle today announced a new database appliance product. It’s called Oracle Database Appliance (ODA). I’m not crazy about the name, but I really like the product. Here’s a picture of the one in Enkitec’s lab:
|
|
The project was code named “Comet” – thus the yellow sticky note. ;)

I really like that name better than ODA, so I think I will just stick with Comet.

Enkitec participated in the beta test program for the product and we were very impressed, particularly with the speed at which the product could be deployed and configured. There is a new tool called the “OAK Configurator” that is sort like the Exadata OneCommand for configuring the system. Keep an eye out for Karl Arao‘s upcoming post with screen shots of the tool in action.

I’m sure there will be plenty of people talking about the specs so I won’t get carried away with that. But I will tell you that it’s basically 4 terrabytes of usable storage, 2 node RAC with up to 24 cores and a SSD component that is used for improving redo write speeds (more on that later), all in a 4U chassis. Andy Colvin has already got a really good post on the  hardware components that are included in the Oracle Database Appliance (along with pictures of the bits and bobs inside the chassis).

I should point out that while I have heard people refer to Comet as a “Baby Exadata”, I really don’t view it that way. That’s because it DOES NOT have separate storage and compute tiers. So there is no Exadata Smart Scan / Offloading secret sauce here. It also does not provide the ability to utilize Exadata’s Hybrid Columnar Compression. On the other hand, like Exadata, it is a pre-configured and tested system that can be dropped in a data center and be ready for use almost immediately (it took us only a couple of hours to set it up and create a database). Pretty unbelievable really.

So much like my favorite Bill Clinton quote, whether ODA is a “Baby Exadata” or not, really depends on your definition of the word “is”. It is a hardware platform that is built specifically to run Oracle databases, but it does not embed any of the unique Exadata software components. Nevertheless, it is an extremely capable platform that will appeal to wide variety of companies running Oracle databases.

And best of all, the list price for this puppy is only $50K. I predict this thing is going to sell like hot cakes!

]]> http://kerryosborne.oracle-guy.com/2011/09/oracle-database-appliance/feed/ 18 Funny Developer Tricks – Decode http://kerryosborne.oracle-guy.com/2011/09/funny-developer-tricks-decode/ http://kerryosborne.oracle-guy.com/2011/09/funny-developer-tricks-decode/#comments Mon, 12 Sep 2011 22:25:16 +0000 osborne http://kerryosborne.oracle-guy.com/?p=3704 I ran into a really ugly SQL statement last week. It was the most expensive statement running on the system for the last several weeks. On top of the fact that the statement ran for hours, it also had a number of “issues”. The statement had the following characteristics:

  • Several DISTINCT Operators – these tend to show up in statements where developers have left off a join condition
  • ANSI Join Syntax – generally Oracle converts this syntax back to Oracle join syntax under the covers. I realize this is standard SQL syntax, but it still makes me nervous due to previous experience with problems caused by this conversion process.
  • Functions Used in WHERE Clauses (DECODE and NVL in particular) – can disable indexes
  • UNION – forces a sort of the union’ed result sets so that it can eliminate duplicates (which is often not necessary). Depending on how the statement is coded, UNIONs can also sometimes result in multiple accesses of the same objects (i.e. if the developer has used the UNION in the place of a more robust where clause).
  • Numerous != and <> Comparison Operators – Not necessarily a problem but often can be coded in a more efficient manner
  • Several OR’s – OR’s are tricky. I get suspicious when there are numerous other problems in a statement as well.

I’ll refrain from publishing the actual statement to protect the guilty, but one of the most interesting bits of the statement looked something like this:

?View Code NONE
...
WHERE a.col1 =  DECODE ('XYZ', '*', a.col1 ,'XYZ')
...

The processing of the where clause can be roughly translated to the following psuedo code:

?View Code NONE
 
if 'XYZ' = '*' then 
  x = a.col1
else
  x = 'XYZ'
end if
 
...
WHERE a.col1 = x
...

Since the literal ‘XYZ’ will never be equal to the literal ‘*’, the value returned by this DECODE will always be ‘XYZ’. Therefore the statement could have been written much more simply as:

?View Code NONE
...
WHERE a.col1 = 'XYZ'
...

So this is an extremely strange way to write the statement and clouds the intention considerably, but does it hurt performance? Well actually no. The optimizer is smart enough to know that this DECODE will always result in the same literal value and so it appears that Oracle does not run the DECODE statement for each row. Here’s a quick example from a 10.2.0.4 database. (note that I used dplan.sql and fss.sql in this code snippet)

?View Code NONE
 
SYS@LAB1024> -- first a DECODE that actually does something
SYS@LAB1024> select count(pk_col) from kso.skew where col2 = DECODE(col1,1,col2,'asd');
 
COUNT(PK_COL)
-------------
      3199971
 
Elapsed: 00:00:06.62
SYS@LAB1024> @fss
Enter value for sql_text: select count(pk_col) from kso.skew where col2 = DECODE(col1,1,col2,'asd')
Enter value for sql_id: 
 
SQL_ID         CHILD      EXECS   AVG_ROWS     AVG_ETIME       AVG_CPU       AVG_PIO      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------- ------------- ------------ ----------------------------------------
45ynyjvg6hyyh      0          1          1          6.46          6.41    111,029.00      162,298 select count(pk_col) from kso.skew where
 
Elapsed: 00:00:00.07
SYS@LAB1024> @dplan
Enter value for sql_id: 45ynyjvg6hyyh
Enter value for child_no: 
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  45ynyjvg6hyyh, child number 0
-------------------------------------
select count(pk_col) from kso.skew where col2 =
DECODE(col1,1,col2,'asd')
 
Plan hash value: 568322376
 
---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |       |       |     4 (100)|          |
|   1 |  SORT AGGREGATE    |      |     1 |    16 |            |          |
|*  2 |   TABLE ACCESS FULL| SKEW |  1234 | 19744 |     4   (0)| 00:00:01 |
---------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   2 - filter("COL2"=DECODE("COL1",1,"COL2",'asd'))
 
 
20 rows selected.
 
Elapsed: 00:00:00.06
SYS@LAB1024> -- Now one that is equivalent to WHERE col2 = 'asddsadasd'
SYS@LAB1024> select count(pk_col) from kso.skew where col2 = DECODE('asddsadasd','*', col2, 'asddsadasd');
 
COUNT(PK_COL)
-------------
     32000000
 
Elapsed: 00:00:03.88
SYS@LAB1024> @fss
Enter value for sql_text: select count(pk_col) from kso.skew where col2 = DECODE('asddsadasd','*', col2, 'asddsadasd')
Enter value for sql_id: 
 
SQL_ID         CHILD      EXECS   AVG_ROWS     AVG_ETIME       AVG_CPU       AVG_PIO      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------- ------------- ------------ ----------------------------------------
4u3nbf55z1ztd      0          1          1          3.88          3.88    111,054.50      162,298 select count(pk_col) from kso.skew where
 
Elapsed: 00:00:00.07
SYS@LAB1024> @dplan 
Enter value for sql_id: 4u3nbf55z1ztd
Enter value for child_no: 
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  4u3nbf55z1ztd, child number 0
-------------------------------------
select count(pk_col) from kso.skew where col2 =
DECODE('asddsadasd','*', col2, 'asddsadasd')
 
Plan hash value: 568322376
 
---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |       |       |     4 (100)|          |
|   1 |  SORT AGGREGATE    |      |     1 |    11 |            |          |
|*  2 |   TABLE ACCESS FULL| SKEW |  1234 | 13574 |     4   (0)| 00:00:01 |
---------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   2 - filter("COL2"='asddsadasd')
 
 
20 rows selected.
 
Elapsed: 00:00:00.01

As you can see, the unnecessary DECODE ran much faster and if you look closely at the Predicate Information sections of the plans you’ll see that the unnecessary DECODE filter was converted to “COL2″=’asddsadasd’. That’s pretty cool.

Unfortunately, whether the DECODE slows down the processing is not the biggest issue. It’s highly probable that the developer thought this DECODE was doing something other than what it’s actually doing and thus that the result produced by this statement is incorrect. But that’s a story for another day.

]]> http://kerryosborne.oracle-guy.com/2011/09/funny-developer-tricks-decode/feed/ 6 Mastering Oracle Trace Data http://kerryosborne.oracle-guy.com/2011/08/mastering_oracle_trace_data/ http://kerryosborne.oracle-guy.com/2011/08/mastering_oracle_trace_data/#comments Fri, 19 Aug 2011 13:58:21 +0000 osborne http://kerryosborne.oracle-guy.com/?p=3687 Cary Millsap is teaching a new one day class next week in Dallas (well Southlake actually) on Oracle trace data. This is a class that he has personally been working on recently and is teaching. I am planning on attending. Here’s a link to the sign up page which has all the details:

Mastering Oracle Trace Data

Check it out.

]]> http://kerryosborne.oracle-guy.com/2011/08/mastering_oracle_trace_data/feed/ 1 New create_1_hint_sql_profile.sql http://kerryosborne.oracle-guy.com/2011/08/new-create_1_hint_sql_profile-sql/ http://kerryosborne.oracle-guy.com/2011/08/new-create_1_hint_sql_profile-sql/#comments Tue, 16 Aug 2011 14:45:39 +0000 osborne http://kerryosborne.oracle-guy.com/?p=3668 I modified my create_1_hint_sql_profile.sql script (which I blogged about here: Single Hint Profiles) to allow any arbitrary text sting including quote characters. This is a script that I use fairly often to apply a hint to a single SQL statement that is executing in a production system where we can’t touch the code for some reason. For example, it’s sometimes useful to add a MONITOR hint or a GATHER_PLAN_STATISTICS hint to a statement that’s behaving badly so we can get more information about what the optimizer is thinking. I recently updated the script to allow special characters in the hint syntax. This feature is useful when you want to add something like an OPT_PARAM hint that takes quoted arguments. The change makes use of the q-Quote feature which I blogged about here: q-Quote. (the original version just barfed on quotes being input as part of the hint)

Here’s an example of how to use it:

?View Code NONE
SYS@SANDBOX1> alter session set cell_offload_processing=false;
 
Session altered.
 
Elapsed: 00:00:00.00
SYS@SANDBOX1> select avg(pk_col) from kso.skew3 where col1 < 0;
 
AVG(PK_COL)
-----------
  1849142.5
 
1 row selected.
 
Elapsed: 00:00:28.08
SYS@SANDBOX1> @fsx
Enter value for sql_text: select avg(pk_col) from kso.skew3 where col1 < 0
Enter value for sql_id: 
 
SQL_ID         CHILD  PLAN_HASH  EXECS  AVG_ETIME AVG_PX OFFLOAD IO_SAVED_% SQL_TEXT
------------- ------ ---------- ------ ---------- ------ ------- ---------- ----------------------------------------------------------------------
a6j7wgqf84jvg      0 2684249835      1      28.07      0 No             .00 select avg(pk_col) from kso.skew3 where col1 < 0
 
1 row selected.
 
Elapsed: 00:00:00.02
SYS@SANDBOX1> @create_1_hint_sql_profile.sql
Enter value for sql_id: a6j7wgqf84jvg
Enter value for profile_name (PROFILE_sqlid_MANUAL): 
Enter value for category (DEFAULT): 
Enter value for force_matching (false): 
Enter value for hint_text: opt_param('cell_offload_processing' 'true')
 
Profile PROFILE_a6j7wgqf84jvg_MANUAL created.
 
Elapsed: 00:00:00.07
SYS@SANDBOX1> @sql_profile_hints
Enter value for profile_name: PROFILE_a6j7wgqf84jvg_MANUAL
 
HINT
------------------------------------------------------------------------------------------------------------------------------------------------------
opt_param('cell_offload_processing' 'true')
 
1 rows selected.
 
Elapsed: 00:00:00.04
SYS@SANDBOX1> select avg(pk_col) from kso.skew3 where col1 < 0;
 
AVG(PK_COL)
-----------
  1849142.5
 
1 row selected.
 
Elapsed: 00:00:05.11
SYS@SANDBOX1> @fsx
Enter value for sql_text: select avg(pk_col) from kso.skew3 where col1 < 0
Enter value for sql_id: 
 
SQL_ID         CHILD  PLAN_HASH  EXECS  AVG_ETIME AVG_PX OFFLOAD IO_SAVED_% SQL_TEXT
------------- ------ ---------- ------ ---------- ------ ------- ---------- ----------------------------------------------------------------------
a6j7wgqf84jvg      0 2684249835      1      28.07      0 No             .00 select avg(pk_col) from kso.skew3 where col1 < 0
a6j7wgqf84jvg      1 2684249835      1       5.10      0 Yes          99.99 select avg(pk_col) from kso.skew3 where col1 < 0

In the example I turned off cell offload processing with the ALTER SESSION and ran a SQL statement that took 28 seconds. Then I used my fsx.sql script to verify that the statement was not offloaded and to find the SQL_ID. Next I created a 1 hint Profile with an OPT_PARAM hint that set the cell_offload_processing parameter back to TRUE using the new version of my create_1_hint_sql_profile.sql script. Next I used my sql_profile_hints.sql script to verify the text of the hint that was added to the Profile. It looked good including the quotes. When I executed the statement a second time it ran in 5 seconds. I then used fsx.sql again to see that the statement was offloaded for the second execution (child 1).

]]> http://kerryosborne.oracle-guy.com/2011/08/new-create_1_hint_sql_profile-sql/feed/ 3 Baselines and SQL_ID http://kerryosborne.oracle-guy.com/2011/08/baselines-and-sql_id/ http://kerryosborne.oracle-guy.com/2011/08/baselines-and-sql_id/#comments Thu, 04 Aug 2011 07:57:41 +0000 osborne http://kerryosborne.oracle-guy.com/?p=3634 I ran across a great post by Marcin Przepiorowski (How to find SQL_ID and PLAN_HASH_VALUE in Oracle SQL Plan Management Baselines) thanks to Karl Arao‘s tweet. Marcin posted a bit of code for calculating a SQL_ID using the text of a SQL statement (based on some research by Tanel Poder). I was aware of Tanel’s research but had not taken the time to try to figure out how to code it. The context of Marcin’s post was based on wanting to relate a Baseline to a SQL_ID. This is something I’ve wanted to do in the past so I was quite interested in the post. As a bonus Marcin also demonstrated a technique to pull the PLAN_HASH_VALUE associated with a Baseline. Of course I can never leave well enough alone and so I had to re-arrange Marcin’s code a little bit to suite my own purposes. So I created a function that returns either the PLAN_HASH_VALUE or the SQL_ID of a Baseline. Here’s the code to create the function: create_baseline_info.sql and a script that uses it: baselines2.sql

Here’s an example of how to use them:

?View Code NONE
 
SYS@SANDBOX1> select count(*) from kso.skew;
 
  COUNT(*)
----------
  32000004
 
SYS@SANDBOX1> @fsx
Enter value for sql_text: select count(*) from kso.skew
Enter value for sql_id: 
 
SQL_ID         CHILD  PLAN_HASH  EXECS  AVG_ETIME AVG_PX OFFLOAD IO_SAVED_% SQL_TEXT
------------- ------ ---------- ------ ---------- ------ ------- ---------- ----------------------------------------------------------------------
gm0w4as5hhr4m      0  578366071      1       1.30      8 Yes          71.78 select count(*) from kso.skew
 
SYS@SANDBOX1> @dplan
Enter value for sql_id: gm0w4as5hhr4m
Enter value for child_no: 
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  gm0w4as5hhr4m, child number 0
-------------------------------------
select count(*) from kso.skew
 
Plan hash value: 578366071
 
----------------------------------------------------------------------------------------------------------------
| Id  | Operation                      | Name     | Rows  | Cost (%CPU)| Time     |    TQ  |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT               |          |       |  6183 (100)|          |        |      |            |
|   1 |  SORT AGGREGATE                |          |     1 |            |          |        |      |            |
|   2 |   PX COORDINATOR               |          |       |            |          |        |      |            |
|   3 |    PX SEND QC (RANDOM)         | :TQ10000 |     1 |            |          |  Q1,00 | P->S | QC (RAND)  |
|   4 |     SORT AGGREGATE             |          |     1 |            |          |  Q1,00 | PCWP |            |
|   5 |      PX BLOCK ITERATOR         |          |    32M|  6183   (1)| 00:01:15 |  Q1,00 | PCWC |            |
|*  6 |       TABLE ACCESS STORAGE FULL| SKEW     |    32M|  6183   (1)| 00:01:15 |  Q1,00 | PCWP |            |
----------------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   6 - storage(:Z>=:Z AND :Z<=:Z)
 
 
23 rows selected.
 
SYS@SANDBOX1> @create_baseline
Enter value for sql_id: gm0w4as5hhr4m
Enter value for plan_hash_value: 578366071
Enter value for fixed (NO): 
Enter value for enabled (YES): 
Enter value for plan_name (SQLID_sqlid_planhashvalue): 
sql_id: gm0w4as5hhr4m
plan_hash_value: 578366071
fixed: NO
enabled: YES
plan_name: SQLID_gm0w4as5hhr4m_578366071
sql_handle: SQL_e86d1a732b7a5c25
Baseline SQLID_gm0w4as5hhr4m_578366071 created.
 
PL/SQL procedure successfully completed.
 
SYS@SANDBOX1> select sql_handle, plan_name from dba_sql_plan_baselines where plan_name like upper('SQLID_gm0w4as5hhr4m_578366071');
 
SQL_HANDLE           PLAN_NAME
-------------------- ------------------------------
SQL_e86d1a732b7a5c25 SQLID_GM0W4AS5HHR4M_578366071
 
SYS@SANDBOX1> @baselines2
Enter value for sql_text: select count(*) from kso.skew%
Enter value for name: 
Enter value for plan_name: 
 
SQL_ID        SQL_TEXT                                           PLAN_HASH_VALUE SQL_HANDLE           PLAN_NAME                      ENABLED ACC FIX LAST_EXECUTED
------------- -------------------------------------------------- --------------- -------------------- ------------------------------ ------- --- --- ----------------
1jp7sjmt0wp1j select count(*) from kso.skew s , skew_gtt g where 2414665383      SQL_936b37ad684d18d4 SQL_PLAN_96utrppn4u66ncefcc71f YES     NO  NO
                                                                 2178955164      SQL_936b37ad684d18d4 SQL_PLAN_96utrppn4u66ne441e87d YES     NO  NO
gm0w4as5hhr4m select count(*) from kso.skew                      578366071       SQL_e86d1a732b7a5c25 SQLID_GM0W4AS5HHR4M_578366071  YES     YES NO
 
SQL> -- cool - it's a match

So in the previous listing I ran a select statement, found the SQL_ID and PLAN_HASH_VALUE for that statement (using fsx.sql), displayed the plan (using dplan.sql) and created a Baseline for the statement (using create_baseline.sql). I then verified that the Baseline existed in DBA_SQL_PLAN_BASELINES and ran the baselines2.sql script which returns the SQL_ID and PLAN_HASH_VALUE using the baseline_info() function, which as you can see matched the original query.

]]> http://kerryosborne.oracle-guy.com/2011/08/baselines-and-sql_id/feed/ 11 Tweeting http://kerryosborne.oracle-guy.com/2011/07/tweeting/ http://kerryosborne.oracle-guy.com/2011/07/tweeting/#comments Thu, 28 Jul 2011 09:43:34 +0000 osborne http://kerryosborne.oracle-guy.com/?p=3616 I think I made a mistake. I recently set up a twitter account @KerryOracleGuy (of course plain old @KerryOsborne was already taken). I only signed up because I wanted to see what @TanelPoder was saying. I wasn’t planning on tweeting at all when I signed up. I just wanted to see what Tanel was saying. Well the first day a bunch of people signed up to follow me. The whole thing seems a little strange. Although it is pretty easy to keep up with what people are doing. So I guess I’m going to give it a try and see if I like it. I have tweeted a few times and I think I am starting to get the hang of it (it’s hard to teach an old dog new tricks). I have figured out how to use tiny url’s now. I think I need to figure out how to tweet pictures as well. I’ve heard there is a Twit Pic for doing that. If I use it though, I guess that would make me a “Twit Pic-er”, which I don’t much like the sound of. So maybe I’ll stick to text based tweets for now.

]]> http://kerryosborne.oracle-guy.com/2011/07/tweeting/feed/ 5 Cardinality Feedback http://kerryosborne.oracle-guy.com/2011/07/cardinality-feedback/ http://kerryosborne.oracle-guy.com/2011/07/cardinality-feedback/#comments Thu, 21 Jul 2011 14:58:51 +0000 osborne http://kerryosborne.oracle-guy.com/?p=3521 I ran into an interesting issue last week having to do with plan stability. The problem description went something like this:

“I have a statement that runs relatively quickly the first time I run it (around 12 seconds). Subsequent executions always run much slower, usually around 20 or 30 minutes. If I flush the shared pool and run it again elapsed time is back to 12 seconds or so.”

The query looked a little like this:

?View Code NONE
SELECT d1.c1  AS c1, 
       d1.c2  AS c2, 
       d1.c3  AS c3, 
       d1.c4  AS c4, 
       d1.c5  AS c5, 
       d1.c6  AS c6, 
       d1.c7  AS c7, 
       d1.c8  AS c8, 
       d1.c9  AS c9, 
       d1.c10 AS c10, 
       d1.c11 AS c11 
FROM   ((SELECT DISTINCT d1.c1        AS c1, 
                         Lpad(' ', 1) AS c2, 
                         d1.c3        AS c3, 
                         d1.c4        AS c4, 
                         d1.c5        AS c5, 
                         d1.c6        AS c6, 
                         d1.c7        AS c7, 
                         d1.c8        AS c8, 
                         d1.c9        AS c9, 
                         d1.c10       AS c10, 
                         d1.c11       AS c11 
         FROM   (SELECT d1.c1  AS c1, 
                        d1.c2  AS c2, 
                        d1.c3  AS c3, 
                        d1.c4  AS c4, 
                        d1.c5  AS c5, 
                        d1.c6  AS c6, 
                        d1.c7  AS c7, 
                        d1.c8  AS c8, 
                        d1.c9  AS c9, 
                        d1.c10 AS c10, 
                        d1.c11 AS c11 
                 FROM   (SELECT d1.c1                            AS c1, 
                                d1.c2                            AS c2, 
                                d1.c3                            AS c3, 
                                d1.c4                            AS c4, 
                                d1.c5                            AS c5, 
                                d1.c6                            AS c6, 
                                d1.c7                            AS c7, 
                                d1.c8                            AS c8, 
                                d1.c9                            AS c9, 
                                d1.c10                           AS c10, 
                                d1.c11                           AS c11, 
                                Row_number() OVER (PARTITION BY d1.c2, d1.c3, 
                                d1.c5, 
                                d1.c6 ORDER 
                                BY 
                                        d1.c2 ASC, 
                                d1.c3 ASC, d1.c5 ASC, d1.c6 ASC) AS c12 
                         FROM   (SELECT d1.c1 
                                        AS c1 
                                        , 
                                        d1.c2 
                                        AS c2 
                                        , 
                                        d1.c3 
                                        AS c3, 
                                        d1.c4 
                                        AS c4 
                                        , 
                                        d1.c5 
                                        AS c5, 
                                        d1.c6 
                                        AS c6 
                                        , 
                                        d1.c7 
                                        AS c7, 
                                        d1.c8 
                                        AS c8 
                                        , 
                                        d1.c9 
                                        AS c9, 
                                        MIN(d1.c10) OVER (PARTITION BY d1.c5, 
                                        d1.c2, 
                                        d1.c6, 
                                        d1.c3) AS 
                                        c10, 
                                        MIN(d1.c11) OVER (PARTITION BY d1.c2, 
                                        d1.c5, 
                                        d1.c6, 
                                        d1.c3) AS 
                                        c11 
                                 FROM   (SELECT d1.c1 AS c1, 
                                                d1.c2 AS c2, 
                                                d1.c3 AS c3, 
                                                d1.c4 AS c4, 
                                                d1.c5 AS c5, 
                                                d1.c6 AS c6, 
                                                d1.c7 AS c7, 
                                                d1.c8 AS c8, 
                                                d1.c9 AS c9, 
                                                CASE 
                                                  WHEN CASE d1.c12 
                                                         WHEN 1 THEN d1.c9 
                                                         ELSE NULL 
                                                       END IS NOT NULL THEN 
                                                  Rank() OVER ( PARTITION BY 
                                                  d1.c5 
                                                  ORDER BY 
                                                  CASE 
                                                  d1.c12 WHEN 1 THEN d1.c9 ELSE 
                                                  NULL END DESC NULLS LAST ) 
                                                END   AS c10, 
                                                CASE 
                                                  WHEN CASE d1.c13 
                                                         WHEN 1 THEN d1.c9 
                                                         ELSE NULL 
                                                       END IS NOT NULL THEN 
                                                  Rank() OVER ( ORDER BY CASE 
                                                  d1.c13 
                                                  WHEN 1 
                                                  THEN d1.c9 
                                                  ELSE NULL END DESC NULLS 
                                                  LAST ) 
                                                END   AS c11 
                                         FROM   (SELECT 
                                        0 
                                        AS 
                                        c1 
                                        , 
                                        d1.c7 
                                        AS c2, 
                                        d1.c8 
                                        AS 
                                        c3 
                                        , 
                                        d1.c9 
                                        AS c4, 
                                        d1.c10 
                                        AS 
                                        c5 
                                        , 
                                        d1.c11 
                                        AS c6, 
                                        CASE 
                                          WHEN d1.c6 = 0.0 THEN 0 
                                          ELSE Nvl(Nvl(d1.c5 + d1.c4 + d1.c3 + 
                                                       d1.c2 + 
                                                       d1.c1, 0) 
                                                   / 
                                                   Nullif(Nvl(d1.c6, 0), 0), 0) 
                                               * 100.0 
                                        END 
                                        AS 
                                        c7 
                                        , 
                                        d1.c6 
                                        AS c8, 
                                        d1.c5 + d1.c4 + d1.c3 + d1.c2 + d1.c1 
                                        AS 
                                        c9 
                                        , 
                                        Row_number() OVER (PARTITION BY d1.c10, 
                                        d1.c7, 
                                        d1.c11, 
                                        d1.c8 
                                        ORDER BY 
                                                d1.c10 
                                        DESC, d1.c7 DESC, d1.c11 DESC, d1.c8 
                                        DESC) AS 
                                        c12 
                                        , 
                                        Row_number() OVER 
                                        (PARTITION BY d1.c7, d1.c10, d1.c11, 
                                        d1.c8 
                                        ORDER BY 
                                                d1.c7 DESC 
                                        , d1.c10 DESC, d1.c11 DESC, d1.c8 DESC) 
                                        AS 
                                        c13 
                                                 FROM   (SELECT d2.c6  AS c1, 
                                                                d2.c7  AS c2, 
                                                                d2.c8  AS c3, 
                                                                d2.c9  AS c4, 
                                                                d2.c10 AS c5, 
                                                                d2.c11 AS c6, 
                                                                d1.c5  AS c7, 
                                                                d1.c1  AS c8, 
                                                                d1.c2  AS c9, 
                                                                d1.c3  AS c10, 
                                                                d1.c4  AS c11 
                                                         FROM   (SELECT DISTINCT 
         CONCAT( 
         CONCAT(t702396.child_nod_dtl, 
         ' - '), 
          t702396.node_descr) AS 
         c1, 
                          CASE 
         WHEN t702396.child_nod_dtl IN 
         (SELECT 
         member_char AS c1 
         FROM 
         wc_abunitbyreg_tree_closure 
         t702410 
         /* Dim_WC_ABUNITBYREG_TREE_CLOSURE_Receivables_Org */ 
         WHERE 
         ( 
         is_leaf 
              = 1 ) 
                                 ) 
                          THEN 1 
                            ELSE 0 
                          END 
         AS c2, 
         t702396.child_nod_dtl 
         AS c3, 
         t702396.tree_node 
         AS c4, 
         t31796.org_name 
         AS c5 
         FROM   wc_abunitbyreg_tree t702396 
         /* Dim_WC_ABUNITBYREG_TREE_Receivables_Org */ 
         , 
         wc_abunitbyreg_tree_closure 
         t702410 
         /* Dim_WC_ABUNITBYREG_TREE_CLOSURE_Receivables_Org */ 
         , 
         wc_ar_aging_accruals_f t699904 
         /* Dim_WC_AR_AGING_ACCRUALS_F */ 
         , 
         w_int_org_d t435564 
         /* Dim_W_INT_ORG_D_Receivables_Org */ 
         , 
         w_party_org_d t31796, 
         w_day_d t31328 
         /* Dim_W_DAY_D_Common */ 
         , 
         wc_ar_aging_accruals_f t699868 
         /* Fact_WC_AR_AGING_ACCRUALS_F */ 
         WHERE  ( t699868.row_wid = 
         t699904.row_wid 
         AND t435564.row_wid = 
         t699868.bu_wid 
         AND t31796.row_wid = 
         t699868.cust_wid 
         AND t31328.row_wid = 
         t699868.snapshot_dt_wid 
         AND t699868.business_unit = 
         t702410.member_char 
         AND t31328.cal_month = 6 
         AND t31328.cal_year = 2011 
         AND t699904.class = 'Billed' 
         AND t702396.child_nod_dtl = 
         t702410.ancestor_char 
         AND CONCAT( 
         CONCAT(t435564.x_consol_cd, 
         '-'), 
         t435564.x_consol_cd_longnm) = 
         'A-Transocean Inc & Subs' 
         AND ( t31796.x_customer_type IN 
         ( 'STD', 'UNS' ) ) 
         AND t699904.due_terms <> 'Not Aged' )) 
         d1, 
         (SELECT SUM(CASE 
         WHEN t699868.due_terms = 
         '> 365 Days' THEN 
         t699868.total_usd_due 
         ELSE 0 
         END)                   AS c6, 
         SUM(CASE 
         WHEN t699868.due_terms = 
         '91-365 Days' 
         THEN 
         t699868.total_usd_due 
         ELSE 0 
         END)                   AS c7, 
         SUM(CASE 
         WHEN t699868.due_terms = '61-90' 
         THEN 
         t699868.total_usd_due 
         ELSE 0 
         END)                   AS c8, 
         SUM(CASE 
         WHEN t699868.due_terms = '31-60' 
         THEN 
         t699868.total_usd_due 
         ELSE 0 
         END)                   AS c9, 
         SUM(CASE 
         WHEN t699868.due_terms = '1-30' 
         THEN 
         t699868.total_usd_due 
         ELSE 0 
         END)                   AS c10, 
         SUM(t699868.total_usd_due) AS c11, 
         t31796.org_name            AS c12, 
         t702396.child_nod_dtl      AS c13 
         FROM   wc_abunitbyreg_tree t702396 
         /* Dim_WC_ABUNITBYREG_TREE_Receivables_Org */ 
         , 
         wc_abunitbyreg_tree_closure t702410 
         /* Dim_WC_ABUNITBYREG_TREE_CLOSURE_Receivables_Org */ 
         , 
         wc_ar_aging_accruals_f t699904 
         /* Dim_WC_AR_AGING_ACCRUALS_F */ 
         , 
         w_int_org_d t435564 
         /* Dim_W_INT_ORG_D_Receivables_Org */ 
         , 
         w_party_org_d t31796, 
         w_day_d t31328 
         /* Dim_W_DAY_D_Common */ 
         , 
         wc_ar_aging_accruals_f t699868 
         /* Fact_WC_AR_AGING_ACCRUALS_F */ 
         WHERE  ( t699868.row_wid = t699904.row_wid 
         AND t435564.row_wid = t699868.bu_wid 
         AND t31796.row_wid = t699868.cust_wid 
         AND t31328.row_wid = 
         t699868.snapshot_dt_wid 
         AND t699868.business_unit = 
         t702410.member_char 
         AND t31328.cal_month = 6 
         AND t31328.cal_year = 2011 
         AND t699904.class = 'Billed' 
         AND t702396.child_nod_dtl = 
         t702410.ancestor_char 
         AND CONCAT(CONCAT(t435564.x_consol_cd, 
         '-'), 
         t435564.x_consol_cd_longnm) = 
         'A-Transocean Inc & Subs' 
         AND ( t31796.x_customer_type IN 
         ( 'STD', 'UNS' ) ) 
         AND t699904.due_terms <> 'Not Aged' ) 
         GROUP  BY t31796.org_name, 
         t702396.child_nod_dtl 
         HAVING SUM(CASE 
         WHEN t699868.due_terms = '1-30' 
         THEN 
         t699868.total_usd_due 
         ELSE 0 
         END) + SUM(CASE 
         WHEN t699868.due_terms 
         = 
         '31-60' THEN 
         t699868.total_usd_due 
         ELSE 0 
         END) + SUM(CASE 
         WHEN t699868.due_terms = 
         '61-90' 
         THEN 
         t699868.total_usd_due 
         ELSE 0 
         END) + SUM(CASE 
         WHEN t699868.due_terms = 
         '91-365 Days' 
         THEN 
         t699868.total_usd_due 
         ELSE 0 
         END) + SUM(CASE 
         WHEN t699868.due_terms = 
         '> 365 Days' THEN t699868.total_usd_due 
         ELSE 0 
         END) <> 
         0) d2 
         WHERE  ( d1.c3 = d2.c13 
         AND Sys_op_map_nonnull(d1.c5) = Sys_op_map_nonnull(d2.c12) )) 
         d1) d1) d1) d1 
         WHERE  ( ( d1.c5 IN ( 'APPLIED DRILLING', 'CHALLENGER MINERAL', 'FEA', 
         'GGA', 
         'IME', 'MED', 'NAM', 'NRS', 
         'NRY', 'SAM', 'TI', 'WAS' ) ) 
         AND d1.c10 <= 10 )) d1 
         WHERE  ( d1.c12 = 1 )) d1 
         UNION ALL 
         SELECT d1.c1  AS c1, 
                d1.c2  AS c2, 
                d1.c3  AS c3, 
                d1.c4  AS c4, 
                d1.c5  AS c5, 
                d1.c6  AS c6, 
                d1.c7  AS c7, 
                d1.c8  AS c8, 
                d1.c9  AS c9, 
                d1.c10 AS c10, 
                d1.c11 AS c11 
         FROM   (SELECT d1.c1 
                        AS c1 
                        , 
                        d1.c2 
                        AS c2, 
                        d1.c3 
                        AS c3 
                        , 
                        d1.c4 
                        AS c4, 
                        d1.c5 
                        AS c5 
                        , 
                        d1.c6 
                        AS c6, 
                        d1.c7 
                        AS c7 
                        , 
                        d1.c8 
                        AS c8, 
                        d1.c9 
                        AS c9 
                        , 
                        d1.c10 
                        AS c10, 
                        d1.c11 
                        AS 
                        c11, 
                        Row_number() OVER (PARTITION BY d1.c2 ORDER BY d1.c2 ASC 
                        ) AS 
                        c12 
                 FROM   (SELECT d1.c1                                 AS c1, 
                                d1.c2                                 AS c2, 
                                d1.c3                                 AS c3, 
                                d1.c4                                 AS c4, 
                                d1.c5                                 AS c5, 
                                d1.c6                                 AS c6, 
                                d1.c7                                 AS c7, 
                                d1.c8                                 AS c8, 
                                d1.c9                                 AS c9, 
                                d1.c10                                AS c10, 
                                MIN(d1.c11) OVER (PARTITION BY d1.c2) AS c11 
                         FROM   (SELECT d1.c1 AS c1, 
                                        d1.c2 AS c2, 
                                        d1.c3 AS c3, 
                                        d1.c4 AS c4, 
                                        d1.c5 AS c5, 
                                        d1.c6 AS c6, 
                                        d1.c7 AS c7, 
                                        d1.c8 AS c8, 
                                        d1.c9 AS c9, 
                                        CASE 
                                          WHEN d1.c9 IS NOT NULL THEN 
                                          Rank() OVER ( PARTITION BY d1.c12 
                                          ORDER BY 
                                          d1.c9 
                                          DESC NULLS LAST ) 
                                        END   AS c10, 
                                        CASE 
                                          WHEN CASE d1.c13 
                                                 WHEN 1 THEN d1.c9 
                                                 ELSE NULL 
                                               END IS NOT NULL THEN 
                                          Rank() OVER ( ORDER BY CASE 
                                          d1.c13 
                                          WHEN 1 
                                          THEN d1.c9 
                                          ELSE NULL END DESC NULLS 
                                          LAST ) 
                                        END   AS c11 
                                 FROM   (SELECT 1 
                                                AS c1 
                                                , 
                                                d1.c7 
                                                AS c2 
                                                , 
                                                Lpad(' ', 1) 
                                                AS c3, 
                                                CAST(NULL AS INTEGER) 
                                                AS c4 
                                                , 
                                                Lpad(' ', 1) 
                                                AS c5, 
                                                Lpad(' ', 1) 
                                                AS c6 
                                                , 
                                                CASE 
                                                  WHEN d1.c6 = 0.0 THEN 0 
                                                  ELSE Nvl(Nvl(d1.c5 + d1.c4 + 
                                                               d1.c3 + 
                                                               d1.c2 + 
                                                               d1.c1, 0) 
                                                           / 
                                                           Nullif(Nvl(d1.c6, 0), 
                                                           0), 0) 
                                                       * 100.0 
                                                END 
                                                AS c7 
                                                , 
                                                d1.c6 
                                                AS c8, 
                                                d1.c5 + d1.c4 + d1.c3 + d1.c2 + 
                                                d1.c1 
                                                AS c9 
                                                , 
                                                d1.c9 
                                                AS c12, 
                                                Row_number() OVER (PARTITION BY 
                                                d1.c7 
                                                ORDER BY 
                                                d1.c7 
                                                DESC) AS 
                                                c13 
                                         FROM   (SELECT SUM(d1.c1) OVER ( 
                                                        PARTITION BY 
                                                        d1.c7) AS 
                                                        c1, 
                                                        SUM(d1.c2) OVER ( 
                                                        PARTITION BY 
                                                        d1.c7) AS 
                                                        c2, 
                                                        SUM(d1.c3) OVER ( 
                                                        PARTITION BY 
                                                        d1.c7) AS 
                                                        c3, 
                                                        SUM(d1.c4) OVER ( 
                                                        PARTITION BY 
                                                        d1.c7) AS 
                                                        c4, 
                                                        SUM(d1.c8) OVER ( 
                                                        PARTITION BY 
                                                        d1.c7) AS 
                                                        c5, 
                                                        SUM(d1.c6) OVER ( 
                                                        PARTITION BY 
                                                        d1.c7) AS 
                                                        c6, 
                                                        d1.c7 
                                                        AS 
                                                        c7, 
                                                        d1.c8 
                                                        AS 
                                                        c8, 
                                                        d1.c9 
                                                        AS 
                                                        c9 
                                                 FROM   (SELECT SUM(CASE 
                                                        WHEN t699868.due_terms = 
                                                             '> 365 Days' THEN 
                                                        t699868.total_usd_due 
                                                        ELSE 0 
                                                                    END) 
                                                                AS c1 
                                                                , 
                SUM(CASE 
                      WHEN t699868.due_terms = 
                           '91-365 Days' 
                    THEN 
                      t699868.total_usd_due 
                      ELSE 0 
                    END)                   AS c2, 
                SUM(CASE 
                      WHEN t699868.due_terms = '61-90' 
                    THEN 
                      t699868.total_usd_due 
                      ELSE 0 
                    END)                   AS c3, 
                SUM(CASE 
                      WHEN t699868.due_terms = '31-60' 
                    THEN 
                      t699868.total_usd_due 
                      ELSE 0 
                    END)                   AS c4, 
                SUM(t699868.total_usd_due) AS c6, 
                t31796.org_name            AS c7, 
                SUM(CASE 
                      WHEN t699868.due_terms = '1-30' 
                    THEN 
                      t699868.total_usd_due 
                      ELSE 0 
                    END)                   AS c8, 
                t702396.child_nod_dtl      AS c9 
                FROM   wc_abunitbyreg_tree t702396 
                /* Dim_WC_ABUNITBYREG_TREE_Receivables_Org */ 
                , 
                wc_abunitbyreg_tree_closure t702410 
                /* Dim_WC_ABUNITBYREG_TREE_CLOSURE_Receivables_Org */ 
                , 
                wc_ar_aging_accruals_f t699904 
                /* Dim_WC_AR_AGING_ACCRUALS_F */ 
                , 
                w_int_org_d t435564 
                /* Dim_W_INT_ORG_D_Receivables_Org */ 
                , 
                w_party_org_d t31796, 
                w_day_d t31328 
                /* Dim_W_DAY_D_Common */ 
                , 
                wc_ar_aging_accruals_f t699868 
                /* Fact_WC_AR_AGING_ACCRUALS_F */ 
                WHERE  ( t699868.row_wid = t699904.row_wid 
                  AND t435564.row_wid = t699868.bu_wid 
                  AND t31796.row_wid = t699868.cust_wid 
                  AND t31328.row_wid = 
                      t699868.snapshot_dt_wid 
                  AND t699868.business_unit = 
                      t702410.member_char 
                  AND t31328.cal_month = 6 
                  AND t31328.cal_year = 2011 
                  AND t699904.class = 'Billed' 
                  AND t702396.child_nod_dtl = 
                      t702410.ancestor_char 
                  AND t702396.child_nod_dtl = 'TI' 
                  AND CONCAT(CONCAT(t435564.x_consol_cd, 
                             '-'), 
                      t435564.x_consol_cd_longnm) = 
                      'A-Transocean Inc & Subs' 
                  AND ( t31796.x_customer_type IN 
                        ( 'STD', 'UNS' ) ) 
                  AND t699904.due_terms <> 'Not Aged' ) 
                GROUP  BY t31796.org_name, 
                   t702396.child_nod_dtl) d1) d1 
                WHERE  ( d1.c5 + d1.c4 + d1.c3 + d1.c2 + d1.c1 <> 0 )) d1) d1) 
                d1 
                 WHERE  ( d1.c10 <= 10 )) d1 
         WHERE  ( d1.c12 = 1 ) 
         UNION ALL 
         SELECT d1.c1  AS c1, 
                d1.c2  AS c2, 
                d1.c3  AS c3, 
                d1.c4  AS c4, 
                d1.c5  AS c5, 
                d1.c6  AS c6, 
                d1.c7  AS c7, 
                d1.c8  AS c8, 
                d1.c9  AS c9, 
                d1.c10 AS c10, 
                d1.c11 AS c11 
         FROM   (SELECT 2                                     AS c1, 
                        d1.c7                                 AS c2, 
                        Lpad(' ', 1)                          AS c3, 
                        CAST(NULL AS INTEGER)                 AS c4, 
                        d1.c8                                 AS c5, 
                        Lpad(' ', 1)                          AS c6, 
                        CASE 
                          WHEN d1.c6 = 0.0 THEN 0 
                          ELSE Nvl(Nvl(d1.c5 + d1.c4 + d1.c3 + d1.c2 + d1.c1, 0) 
                                   / 
                                   Nullif(Nvl(d1.c6, 0), 0), 0) 
                               * 100.0 
                        END                                   AS c7, 
                        d1.c6                                 AS c8, 
                        d1.c5 + d1.c4 + d1.c3 + d1.c2 + d1.c1 AS c9, 
                        CAST(NULL AS INTEGER)                 AS c10, 
                        CASE 
                          WHEN d1.c5 + d1.c4 + d1.c3 + d1.c2 + d1.c1 IS NOT NULL 
                        THEN 
                          Rank() OVER ( ORDER BY d1.c5 + d1.c4 + d1.c3 + d1.c2 + 
                          d1.c1 
                          DESC 
                          NULLS LAST ) 
                        END                                   AS c11, 
                        CASE 
                          WHEN d1.c5 + d1.c4 + d1.c3 + d1.c2 + d1.c1 IS NOT NULL 
                        THEN 
                          Rank() OVER ( PARTITION BY d1.c8 ORDER BY d1.c5 + 
                          d1.c4 + 
                          d1.c3 
                          + 
                          d1.c2 + 
                          d1.c1 
                          DESC NULLS LAST ) 
                        END                                   AS c12 
                 FROM   (SELECT SUM(CASE 
                                      WHEN t699868.due_terms = '> 365 Days' THEN 
                                      t699868.total_usd_due 
                                      ELSE 0 
                                    END)                   AS c1, 
                                SUM(CASE 
                                      WHEN t699868.due_terms = '91-365 Days' 
                                    THEN 
                                      t699868.total_usd_due 
                                      ELSE 0 
                                    END)                   AS c2, 
                                SUM(CASE 
                                      WHEN t699868.due_terms = '61-90' THEN 
                                      t699868.total_usd_due 
                                      ELSE 0 
                                    END)                   AS c3, 
                                SUM(CASE 
                                      WHEN t699868.due_terms = '31-60' THEN 
                                      t699868.total_usd_due 
                                      ELSE 0 
                                    END)                   AS c4, 
                                SUM(CASE 
                                      WHEN t699868.due_terms = '1-30' THEN 
                                      t699868.total_usd_due 
                                      ELSE 0 
                                    END)                   AS c5, 
                                SUM(t699868.total_usd_due) AS c6, 
                                t31796.org_name            AS c7, 
                                t702396.child_nod_dtl      AS c8 
                         FROM   wc_abunitbyreg_tree t702396 
                                /* Dim_WC_ABUNITBYREG_TREE_Receivables_Org */ 
                                , 
                                wc_abunitbyreg_tree_closure t702410 
                                /* Dim_WC_ABUNITBYREG_TREE_CLOSURE_Receivables_Org */ 
                                , 
                                wc_ar_aging_accruals_f t699904 
                                /* Dim_WC_AR_AGING_ACCRUALS_F */ 
                                , 
                                w_int_org_d t435564 
                                /* Dim_W_INT_ORG_D_Receivables_Org */ 
                                , 
                                w_party_org_d t31796, 
                                w_day_d t31328 
                                /* Dim_W_DAY_D_Common */ 
                                , 
                                wc_ar_aging_accruals_f t699868 
                         /* Fact_WC_AR_AGING_ACCRUALS_F */ 
                         WHERE  ( t699868.row_wid = t699904.row_wid 
                                  AND t435564.row_wid = t699868.bu_wid 
                                  AND t31796.row_wid = t699868.cust_wid 
                                  AND t31328.row_wid = t699868.snapshot_dt_wid 
                                  AND t699868.business_unit = 
                                      t702410.member_char 
                                  AND t31328.cal_month = 6 
                                  AND t31328.cal_year = 2011 
                                  AND t699904.class = 'Billed' 
                                  AND t702396.child_nod_dtl = 
                                      t702410.ancestor_char 
                                  AND t702396.child_nod_dtl = 'TI' 
                                  AND CONCAT(CONCAT(t435564.x_consol_cd, '-'), 
                                      t435564.x_consol_cd_longnm) = 
                                      'A-Transocean Inc & Subs' 
                                  AND ( t31796.x_customer_type IN 
                                        ( 'STD', 'UNS' ) ) 
                                  AND t699904.due_terms <> 'Not Aged' ) 
                         GROUP  BY t31796.org_name, 
                                   t702396.child_nod_dtl 
                         HAVING SUM(CASE 
                                      WHEN t699868.due_terms = '1-30' THEN 
                                      t699868.total_usd_due 
                                      ELSE 0 
                                    END) + SUM(CASE 
                                                 WHEN t699868.due_terms = 
                                                      '31-60' THEN 
                                                 t699868.total_usd_due 
                                                 ELSE 0 
                                               END) + SUM(CASE 
                                WHEN t699868.due_terms = 
                                     '61-90' 
                                                          THEN 
t699868.total_usd_due 
ELSE 0 
END) + SUM(CASE 
WHEN t699868.due_terms = 
'91-365 Days' 
         THEN 
t699868.total_usd_due 
           ELSE 0 
         END) + SUM(CASE 
WHEN t699868.due_terms = 
'> 365 Days' THEN t699868.total_usd_due 
ELSE 0 
                    END) <> 
0 
) d1) 
d1 
WHERE  ( d1.c12 <= 10 ))) d1 
ORDER  BY c1

Simple right?

As expected based on the problem description, there were many cursors with different plans and very different statistics in the shared pool. So I had them run a script showing some statistical info about the various plans:

?View Code NONE
SQL_ID        PLAN_HASH_VALUE          EXECS    AVG_ETIME AVG_CPU_TIME          AVG_LIO         AVG_PIO
------------- --------------- -------------- ------------ ------------ ---------------- - ---------------
9tx5b0cctd8j5      1512746808              5         .618         .398          5,608.4             8.0
9tx5b0cctd8j5       310704641              1        1.496         .900          4,672.0             6.0
9tx5b0cctd8j5      2273374088              1        1.531         .950          5,810.0             8.0
9tx5b0cctd8j5      2878409905              1        1.432        1.080          5,810.0             8.0
9tx5b0cctd8j5      2225548639              1       14.193       10.310      1,057,712.0             8.0
9tx5b0cctd8j5      2174863591              1       13.623       10.660      1,049,157.0             8.0
9tx5b0cctd8j5      2214096856              1       13.573       10.720      1,057,955.0             8.0
9tx5b0cctd8j5      4111066445              1       13.187       11.090      1,049,157.0            10.0
9tx5b0cctd8j5      2307125907              1       13.411       11.200      1,057,779.0             8.0
9tx5b0cctd8j5       983902177              2       12.356       11.245      1,057,614.5            70.5
9tx5b0cctd8j5      2070474756              1       12.940       11.310      1,049,157.0             8.0
9tx5b0cctd8j5       986683435              1       12.158       11.590      1,026,638.0             8.0
9tx5b0cctd8j5      2004450975              1       12.463       11.700      1,049,167.0             8.0
9tx5b0cctd8j5      3593472063              1       12.336       11.710      1,026,638.0            30.0
9tx5b0cctd8j5       453801037              1       13.494       11.820      1,057,931.0             8.0
9tx5b0cctd8j5      3874155392              1       12.252       11.850      1,049,157.0             8.0
9tx5b0cctd8j5      1048579546              1       12.974       11.930      1,057,712.0             8.0
9tx5b0cctd8j5       936861260              1       72.135       66.720     11,711,783.0             2.0
9tx5b0cctd8j5      2279755221              2      128.250      121.020     21,775,758.0              .0
9tx5b0cctd8j5      1351237956              1      554.598      495.800     82,680,484.0             1.0
9tx5b0cctd8j5       116866680              2      539.060      521.715     83,920,850.5              .0
9tx5b0cctd8j5      3869018014              1    1,230.521      585.880    117,828,348.0              .0
9tx5b0cctd8j5      3732072456              2      662.755      644.470    104,823,026.5             2.0
9tx5b0cctd8j5      3282940025              1      825.540      811.690    132,362,095.0              .0
9tx5b0cctd8j5      2292909572              1      957.555      923.070    153,126,414.0             3.0

As you can imagine, the plans are pretty long as well. I’ll spare you the detail of them but show you the notes section of the plans:

?View Code NONE
 
-- one of the 12 second ones
 
Note
-----
   - star transformation used for this statement
 
 
-- multiple slow ones
 
Note
-----
   - star transformation used for this statement
   - cardinality feedback used for this statement

One of the statements showed about a 12 second average execution time without a note saying cardinality feedback had been used. The rest of the plans showed that cardinality feedback had been used. I suggested that maybe cardinality feedback was introducing the instability and disabling it might be a reasonable thing to try. They tried it using an alter session and the statement consistently ran in 11-12 seconds. I should note that you can also disable this feature for an individual statement by using an OPT_PARAM hint like so:

select /*+ opt_param(‘_optimizer_use_feedback’ ‘false’) */ …

By the way, as you may have already noticed, it’s clear from the stats that the problem description did not exactly match the data. If you’ll look back at the stats by plan hash value output you’ll see that there are also several plans that produced results in the 1 second range (which they failed to mention). It appears that cardinality feedback actually enabled some pretty good plans that reduced the time by an order of magnitude from the original 12 second plan . But the client wasn’t really looking for the absolute best performance, what they were looking for was stability. They were actually quite happy with 12 seconds, as long as it did that all the time. Had I had access to the system we might have a attempted to lock in the 1 second plan. But this was all done via email without actually logging into the system (which is a bit like performing surgery with a blind fold on – with someone else describing what you’re cutting in to).

I’m used to seeing plan stability issues (often caused by bind variable peeking), but this is the first time I’ve seen this type of behavior due to the new 11g Cardinality Feedback feature. Generally speaking, this feature is a really good idea because it compares cardinality estimates that the optimizer calculates with actual row counts when a query runs. This is actually a very good technique for SQL tuning and Wolfgang Breitling has written an excellent paper on the subject. The basic idea is that if the estimated cardinality and the actual row counts are way off on any of the steps, the optimizer can automatically apply a fudge factor to correct the optimizer’s calculation. The correction is exposed in a 10053 trace file by the application of an OPT_ESTIMATE hint (you know, the same one that is used by SQL Profiles created by the SQL Tuning Advisor). Unfortunately, these hints are not exposed in the OTHER_XML field of the V$SQL_PLAN table or any place else that I’ve uncovered thus far (although it must be stored in memory somewhere). So the only way I know to determine what modifications are being made is to Wolfgang the statement and look at the trace file. Here’s a little excerpt from a 10053 trace file that has used this feature:

?View Code NONE
 
...
 
******************************************
----- Current SQL Statement for this session (sql_id=2ajw1uv0gpzwq) -----
select product_name
from oe.order_items o, oe.product_information p
where o.unit_price = 15 and quantity > 1
and p.product_id = o.product_id
*******************************************
...
 
Final query after transformations:******* UNPARSED QUERY IS *******
SELECT /*+ OPT_ESTIMATE (TABLE "P" MIN=1.000000 ) OPT_ESTIMATE (INDEX_SCAN "P" "PRODUCT_INFORMATION_PK" MIN=1.000000 ) OPT_ESTIMATE (INDEX_FILTER "P" "PRODUCT_INFORMATION_PK" MIN=1.000000 ) OPT_ESTIMATE (TABLE "O" ROWS=13.000000 ) */ "P"."PRODUCT_NAME" "PRODUCT_NAME" FROM "OE"."ORDER_ITEMS" "O","OE"."PRODUCT_INFORMATION" "P" WHERE "O"."UNIT_PRICE"=15 AND "O"."QUANTITY">1 AND "P"."PRODUCT_ID"="O"."PRODUCT_ID"
kkoqbc: optimizing query block SEL$1 (#0)
 
...
 
Content of other_xml column
===========================
  cardinality_feedback: yes
  db_version     : 11.2.0.2
  parse_schema   : SYS
  plan_hash      : 1553478007
  plan_hash_2    : 2615131494
  Outline Data:
  /*+
    BEGIN_OUTLINE_DATA
      IGNORE_OPTIM_EMBEDDED_HINTS
      OPTIMIZER_FEATURES_ENABLE('11.2.0.2')
      DB_VERSION('11.2.0.2')
      OPT_PARAM('optimizer_dynamic_sampling' 0)
      ALL_ROWS
      OUTLINE_LEAF(@"SEL$1")
      FULL(@"SEL$1" "O"@"SEL$1")
      FULL(@"SEL$1" "P"@"SEL$1")
      LEADING(@"SEL$1" "O"@"SEL$1" "P"@"SEL$1")
      USE_HASH(@"SEL$1" "P"@"SEL$1")
    END_OUTLINE_DATA
  */
 
...

You can see all the OPT_ESTIMATE hints that have been added to the statement. V$SQL_SHARED_CURSOR does have a flag to indicate whether a cursor has been marked to use this feature. So you can do a little investigation on your own system to see how often this feature is kicking in.

?View Code NONE
SYS@SANDBOX1> @find_card_feedback_cursors.sql
SYS@SANDBOX1> col sql_text for a80 word_wrap
SYS@SANDBOX1> select a.sql_id, child_number child_no, b.sql_text from v$sql_shared_cursor a, v$sqlstats b
  2  where a.sql_id = b.sql_id
  3  and USE_FEEDBACK_STATS = 'Y'
  4  and a.sql_id like nvl('&sql_id',a.sql_id)
  5  and upper(sql_text) like upper(nvl('&sql_text',sql_text))
  6  and child_number > 0
  7  order by 1, 2
  8  /
Enter value for sql_id: 
Enter value for sql_text: %oe.%
 
SQL_ID          CHILD_NO SQL_TEXT
------------- ---------- --------------------------------------------------------------------------------
0cm4r08vj075r          2 select /*+ monitor */ product_name from oe.order_items o, oe.product_information
                         p where o.unit_price = 15 and quantity > 1 and p.product_id = o.product_id
 
0cm4r08vj075r          3 select /*+ monitor */ product_name from oe.order_items o, oe.product_information
                         p where o.unit_price = 15 and quantity > 1 and p.product_id = o.product_id
 
2ajw1uv0gpzwq          1 select product_name from oe.order_items o, oe.product_information p where
                         o.unit_price = 15 and quantity > 1 and p.product_id = o.product_id

This feature is similar to another 11g feature call Adaptive Cursor Sharing which also attempts to automatically recognize performance problems and create new cursors to overcome the issues. Both features share a couple of traits.

  1. The statement must be run at least once without benefit of the feature in order to discover that there is a potential problem.
  2. The data collected about the statement is not persisted. So any event that causes a statement to be flushed from the shared pool will cause the “learning process” to be repeated.

There is not much available on the web about this feature yet. Here are a few posts that provide some insight.

By Tom Kite
By Dion Cho
By The Optimizer Development Group

One of the comments that Tom makes in his discussion is that SQL Plan Management (Baselines) could be used to curb potential instability issues caused by this new feature. I thought this was an interesting idea so I thought it might be worthwhile to look into the interaction between the Cardinality Feedback feature and Baselines. Note, the client I worked with on the aforementioned issue had actually tried using baselines to keep the plans from changing. But without having access to the system it was just too difficult to try to understand what had or hadn’t been done with the baselines. So I did a little investigation on our Exadata lab environment. After finding a statement that causes Cardinality Feedback to kick in, I flushed the shared pool and ran it a few times. I then create a baseline to force it to go back to the original plan.

?View Code NONE
SYS@SANDBOX1> !cat b.sql
select /*+ monitor */ product_name
from oe.order_items o, oe.product_information p
where o.unit_price = 15 and quantity > 1
and p.product_id = o.product_id; 
 
SYS@SANDBOX1> set termout off
SYS@SANDBOX1> @b
. . .
SYS@SANDBOX1> @b
SYS@SANDBOX1> set termout on
SYS@SANDBOX1> 
SYS@SANDBOX1> -- let's check the shared pool now
SYS@SANDBOX1> @fsx
Enter value for sql_text: 
Enter value for sql_id: 0cm4r08vj075r
 
SQL_ID         CHILD  PLAN_HASH  EXECS  AVG_ETIME AVG_PX OFFLOAD IO_SAVED_% SQL_TEXT
------------- ------ ---------- ------ ---------- ------ ------- ---------- ----------------------------------------------------------------------
0cm4r08vj075r      0 1255158658      1        .04      0 No             .00 select /*+ monitor */ product_name from oe.order_items o, oe.product_i
0cm4r08vj075r      1 1553478007      4        .00      0 No             .00 select /*+ monitor */ product_name from oe.order_items o, oe.product_i
 
2 rows selected.
SYS@SANDBOX1>  
SYS@SANDBOX1> -- yes we have two cursors with different plans
SYS@SANDBOX1> -- let's see what the plans look like
SYS@SANDBOX1> @dplan
Enter value for sql_id: 0cm4r08vj075r
Enter value for child_no: 
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  0cm4r08vj075r, child number 0
-------------------------------------
select /*+ monitor */ product_name from oe.order_items o,
oe.product_information p where o.unit_price = 15 and quantity > 1 and
p.product_id = o.product_id
 
Plan hash value: 1255158658
 
-------------------------------------------------------------------------------------------------------
| Id  | Operation                    | Name                   | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                        |       |       |     7 (100)|          |
|   1 |  NESTED LOOPS                |                        |       |       |            |          |
|   2 |   NESTED LOOPS               |                        |     4 |   128 |     7   (0)| 00:00:01 |
|*  3 |    TABLE ACCESS STORAGE FULL | ORDER_ITEMS            |     4 |    48 |     3   (0)| 00:00:01 |
|*  4 |    INDEX UNIQUE SCAN         | PRODUCT_INFORMATION_PK |     1 |       |     0   (0)|          |
|   5 |   TABLE ACCESS BY INDEX ROWID| PRODUCT_INFORMATION    |     1 |    20 |     1   (0)| 00:00:01 |
-------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   3 - storage(("O"."UNIT_PRICE"=15 AND "QUANTITY">1))
       filter(("O"."UNIT_PRICE"=15 AND "QUANTITY">1))
   4 - access("P"."PRODUCT_ID"="O"."PRODUCT_ID")
 
SQL_ID  0cm4r08vj075r, child number 1
-------------------------------------
select /*+ monitor */ product_name from oe.order_items o,
oe.product_information p where o.unit_price = 15 and quantity > 1 and
p.product_id = o.product_id
 
Plan hash value: 1553478007
 
--------------------------------------------------------------------------------------------------
| Id  | Operation                  | Name                | Rows  | Bytes | Cost (%CPU)| Time     |
--------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT           |                     |       |       |     9 (100)|          |
|*  1 |  HASH JOIN                 |                     |    13 |   416 |     9  (12)| 00:00:01 |
|*  2 |   TABLE ACCESS STORAGE FULL| ORDER_ITEMS         |    13 |   156 |     3   (0)| 00:00:01 |
|   3 |   TABLE ACCESS STORAGE FULL| PRODUCT_INFORMATION |   288 |  5760 |     5   (0)| 00:00:01 |
--------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   1 - access("P"."PRODUCT_ID"="O"."PRODUCT_ID")
   2 - storage(("O"."UNIT_PRICE"=15 AND "QUANTITY">1))
       filter(("O"."UNIT_PRICE"=15 AND "QUANTITY">1))
 
Note
-----
   - cardinality feedback used for this statement
 
 
54 rows selected.
SYS@SANDBOX1> --
SYS@SANDBOX1> -- so the second cursor (1) has a new plan (1553478007) because of cardinality feedback
SYS@SANDBOX1> -- let's create a baseline to force it to stick with the original plan (1255158658)
SYS@SANDBOX1> @create_baseline
Enter value for sql_id: 0cm4r08vj075r
Enter value for plan_hash_value: 1255158658
Enter value for fixed (NO): 
Enter value for enabled (YES): 
Enter value for plan_name (SQLID_sqlid_planhashvalue): 
sql_id: 0cm4r08vj075r
plan_hash_value: 1255158658
fixed: NO
enabled: YES
plan_name: SQLID_0cm4r08vj075r_1255158658
sql_handle: SQL_7912d4b3adf9c3b4
Baseline SQLID_0cm4r08vj075r_1255158658 created.
 
PL/SQL procedure successfully completed.
 
SYS@SANDBOX1> -- now let's run the statement a few more times
SYS@SANDBOX1> set termout off
SYS@SANDBOX1> @b
SYS@SANDBOX1> @b
. . .
SYS@SANDBOX1> @b
SYS@SANDBOX1> @b
SYS@SANDBOX1> set termout on
 
SYS@SANDBOX1> -- and check the children and plans
SYS@SANDBOX1> @fsx
Enter value for sql_text: 
Enter value for sql_id: 0cm4r08vj075r
 
SQL_ID         CHILD  PLAN_HASH  EXECS  AVG_ETIME AVG_PX OFFLOAD IO_SAVED_% SQL_TEXT
------------- ------ ---------- ------ ---------- ------ ------- ---------- ----------------------------------------------------------------------
0cm4r08vj075r      0 1255158658      1        .00      0 No             .00 select /*+ monitor */ product_name from oe.order_items o, oe.product_i
0cm4r08vj075r      1 1553478007      1        .00      0 No             .00 select /*+ monitor */ product_name from oe.order_items o, oe.product_i
0cm4r08vj075r      2 1255158658      1        .00      0 No             .00 select /*+ monitor */ product_name from oe.order_items o, oe.product_i
0cm4r08vj075r      3 1255158658      1        .00      0 No             .00 select /*+ monitor */ product_name from oe.order_items o, oe.product_i
0cm4r08vj075r      5 1255158658      4        .00      0 No             .00 select /*+ monitor */ product_name from oe.order_items o, oe.product_i

Interesting that it created a bunch of new cursors – 2,3 (presumably 4 that got flushed for some reason) and 5. Looks like it finally settled down with 5, since child 5 has been executed 4 times. And look what the plan looks like for the one it settled down on:

?View Code NONE
 
SYS@SANDBOX1> @dplan
Enter value for sql_id: 0cm4r08vj075r
Enter value for child_no: 5
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  0cm4r08vj075r, child number 5
-------------------------------------
select /*+ monitor */ product_name from oe.order_items o,
oe.product_information p where o.unit_price = 15 and quantity > 1 and
p.product_id = o.product_id
 
Plan hash value: 1255158658
 
-------------------------------------------------------------------------------------------------------
| Id  | Operation                    | Name                   | Rows  | Bytes | Cost (%CPU)| Time     |
-------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |                        |       |       |    16 (100)|          |
|   1 |  NESTED LOOPS                |                        |       |       |            |          |
|   2 |   NESTED LOOPS               |                        |    13 |   416 |    16   (0)| 00:00:01 |
|*  3 |    TABLE ACCESS STORAGE FULL | ORDER_ITEMS            |    13 |   156 |     3   (0)| 00:00:01 |
|*  4 |    INDEX UNIQUE SCAN         | PRODUCT_INFORMATION_PK |     1 |       |     0   (0)|          |
|   5 |   TABLE ACCESS BY INDEX ROWID| PRODUCT_INFORMATION    |     1 |    20 |     1   (0)| 00:00:01 |
-------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   3 - storage(("O"."UNIT_PRICE"=15 AND "QUANTITY">1))
       filter(("O"."UNIT_PRICE"=15 AND "QUANTITY">1))
   4 - access("P"."PRODUCT_ID"="O"."PRODUCT_ID")
 
Note
-----
   - SQL plan baseline SQLID_0CM4R08VJ075R_1255158658 used for this statement
   - cardinality feedback used for this statement
 
 
31 rows selected.

Hmmm … Oracle says it used both the baseline and cardinality feedback, but the plan stayed the same as without the cardinality feedback. It seems a little weird that cardinality feedback would be applied to statement that already has a baseline on it. But it makes sense that the plan would stay consistent because the hints would generally limit the CBO’s choices down to the point where adjustments to cardinality calculations wouldn’t matter. At any rate, it’s an interesting bit of trivia.

Note that I have used several scripts in this post:

  1. create_baseline.sql – create a Baseline on a statement in the shared pool
  2. dplan.sql – produces XPLAN output for a statement in the shared pool
  3. find_card_feedback_cursors.sql – looks for cursors using cardinality feedback
  4. fsx.sql – shows SQL statements in the shared pool along with indication if Exadata Smart Scans were used

Cardinality Feedback Wrap Up

In general I think Cardinality Feedback is a great new feature, but in this particular case it did introduce some instability. This may be the result of some unintended behavior but we did not open an SR to track that down since disabling the feature provided an adequate solution in this case. I would still like to find where it stores those fudge factors used by the OPT_ESTIMATE hints. I was hoping I could find it in one of the SQL Profile objects like sqlobj$data, but haven’t had any luck with that line of investigation so far. So if you have any ideas please let me know.

]]> http://kerryosborne.oracle-guy.com/2011/07/cardinality-feedback/feed/ 14