Kerry Osborne's Oracle Blog

Exadata Book

osborne — Wed, 01 Sep 2010 12:38:51 +0000

Here’s some white Exaddata text

Well, I guess it’s official. A couple of weeks ago I committed to write an Exadata book for Apress, along with my intrepid co-author Randy Johnson. For those of you who don’t know Randy, he’s a very experienced Oracle Guy with a wealth of knowledge, particularly around RAC. I think the two of us make a pretty good team - making up for each others weaknesses (oh wait, I should say we have “Complimentary Skill Sets” - yeah that sounds better).

Anyway, it turns out that writing a book is a lot of work! The way Tom Kyte turns them out I thought it must be pretty easy, but I’ve always been a little overconfident. So I’m starting to realize that I may not have time to do as many blog posts as I might like. But I must say that I am really excited about the subject matter! So I think it will be worth the effort. By the way, that’s not the official cover art (or even the official title as far as I know). I just hacked that together with a Adobe Illustrator. ;-)

As I have said many times, I think the Exadata storage software will usher in a whole new era in relational databases. Not just for Oracle, because you know the other guys will be trying to follow in their footsteps. But I think Oracle is miles ahead at this point, and they own their own hardware platform. Oracle claims that it has been their most successful product launch ever and I believe it. They are starting to pop up like weeds. It will be very interesting to see what the future holds for this platform. To be honest, I think we’re just seeing the tip of the iceberg at this point. Anyway, I appreciate the guys at Apress having the faith in us to take on this project. I hope we don’t disappoint them (I don’t think we will).

Guess Who Got Inducted into the OakTable Network

osborne — Sat, 28 Aug 2010 17:10:32 +0000

I am very proud to be one of the newest OakTable Network members!

(That My Happy Face)

Notes on Applying Exadata Bundle Patch (BP5)

osborne — Thu, 26 Aug 2010 22:44:10 +0000

Randy Johnson has done a brief post after applying BP5 on our Exadata Lab machine. Looks like it went pretty smoothly with the exception of a problem with DBFS and some misleading comments in the README file regarding using the RDS protocol (both of which we had in play). Here’s a link to his post:

Exadata Bundle Patch 5 Gotcha’s

Running Oracle Exadata V2 on Dell Hardware

osborne — Wed, 25 Aug 2010 02:52:16 +0000

Well we had to give it a shot.

So we created an Oracle Exadata Storage Server Software CELLBOOT USB flash drive. I’m not kidding, that’s what the Oracle/Sun guys decided to call it. They didn’t even use an acronym in the manual (I guess “ESSSCB USB FD” doesn’t roll off the tongue much better than the whole thing anyway). We used the make_cellboot_usb utility to create the thing off one of our storage servers, which by the way was not that easy to do, since the USB ports are in the back of the 4275’s and they are not easy to get to with all the cabling that’s back there. Anyway, once we had the little bugger created we pulled it out of the back of the rack and booted a Dell Latitude D630 off of it. Here’s a picture:

Notice the thumb drive is all lit up like a Christmas tree.

Here is a close up of the screen (in case your eyes are going bad like mine):

So we tried a couple of different options but eventually got to this screen:

Notice the ERROR line in the middle of the screen. Somebody wisely put a check in the boot procedure to verify the machine type, presumably if it’s not a Sun 4170 it will throw an error. We thought about hacking the system but decided not to at this point as we had real work to do. (maybe later when we’ve got nothing else to do)

Oracle Exadata - Storage Indexes

osborne — Tue, 10 Aug 2010 18:09:29 +0000

Wow! - I was stunned a few days ago by Exadata’s Storage Indexes. I was doing a little testing to see what could be offloaded and what couldn’t (more on that later). I have a 384 million row table I was using on our Exadata Quarter Rack test system. A single threaded full scan with no where clause on the table takes about 24 seconds (ho hum - it’s amazing how quickly we become numbed to the outstanding performance ). So imagine my surprise when I decided to check and see how many nulls I had in a column and the result came back in .07 seconds. Wow! I thought it was a bug! Turns out it was the Storage Indexes. Alright already, I’ll show you some output from the system (by the way, as usual I used a couple of scripts: fsx.sql and mystats.sql):

^?View Code NONE

SYS@LABRAT1> select count(*) from kso.skew3;
 
  COUNT(*)
----------
 384000048
 
1 row selected.
 
Elapsed: 00:00:24.06
SYS@LABRAT1> /
 
  COUNT(*)
----------
 384000048
 
1 row selected.
 
Elapsed: 00:00:23.94
 
SYS@LABRAT1> set timing off
SYS@LABRAT1> @mystats
Enter value for name: %storage%
 
NAME                                                                             VALUE
---------------------------------------------------------------------- ---------------
cell physical IO bytes saved by storage index                                        0
 
1 row selected.
 
SYS@LABRAT1> set timing on
SYS@LABRAT1> select count(*) from kso.skew3 where col1 is null;
 
  COUNT(*)
----------
        12
 
1 row selected.
 
Elapsed: 00:00:00.07
SYS@LABRAT1> set timing off
SYS@LABRAT1> @fsx
Enter value for sql_text: select count(*) from kso.skew3 where col1 is null
Enter value for sql_id: 
Enter value for inst_id: 
 
 
 INST SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO    AVG_PIO AVG_PX OFFLOADABLE IO_SAVED_% SQL_TEXT
----- ------------- ------ ---------- ---------- ------------- ------------ ---------- ------ ----------- ---------- ----------------------------------------
    1 0u1q4b7puqz6g      0 2684249835          5           .09    1,956,226  1,956,219      0 Yes             100.00 select count(*) from kso.skew3 where col
 
1 row selected.
 
SYS@LABRAT1> @mystats
Enter value for name: %storage%
 
NAME                                                                             VALUE
---------------------------------------------------------------------- ---------------
cell physical IO bytes saved by storage index                              16012763136
 
1 row selected.

So apparently Storage Indexes are NULL aware. Very cool! This may have repercussions regarding design and implementation decisions. There are systems that don’t use NULLs in order to insure that they can access records via B-Tree indexes (which as you’re aware do not store NULLs). SAP for example uses a single space character instead of NULLs.

The Sun / Oracle Exadata V2 has a number of features that get a lot of attention because of the dramatic performance gains they can provide. Smart Scan / Offloading, Hybrid Columnar Compression (HCC), Flash Cache, Infiniband Fabric, etc… Exadata Storage Indexes are not really high on the list of things that the sales guys tout, but they can provide a very significant performance benefit. One of the reasons they aren’t talked about much is that there is nothing really that you can do to affect how they work. There is no tuning them in other words. They just work. However, understanding how they work can certainly help you make better decisions about how to build systems and how to write SQL statements. I had an online discussion with someone a few weeks ago about Storage Indexes and the comment was made that we didn’t really need to understand them, because there was nothing we could do about them anyway. My thought was that we couldn’t do anything about the fact that normal B-Tree indexes don’t store NULLs either, but it is pretty important information to know when you are designing a system or trying to access data using a B-Tree index.

I found a very good description of Storage Indexes in an Oracle White Paper:

Oracle Sun Database Machine Application Best Practices for Data Warehousing [ID 1094934.1]

Exdata Storage Index

In Oracle Database 11gR2 a new feature called the Exadata Storage Index has been introduced. The storage index contains a summary of the data distribution on the disk and provides an additional method to eliminate unnecessary disk IO.

Each disk in the Exadata storage cell is divided into equal sized pieces called storage regions (default 1MB). There is an index entry for every storage regions (1MB of data stored on disk). Each entry contains the minimum and maximum value for columns seen in ‘where’ clause predicates. Information for up to 8 columns can be stored. The index is then used to eliminate disk IO by identifying which storage regions don’t match the ‘where’ clause of a query. For example consider the following query

The storage index is stored in the memory on each of the Exadata storage cells and is created and maintained transparently. However, if a storage cell is shutdown or rebooted the storage index will be lost from memory and will be recreated on subsequent accesses to the data after the cell has been brought back online.

The storage index can be used as an additional partitioning strategy if the data to be loaded is sorted on the additional partitioning column. For example, in the TPC-H benchmark, the Lineitem table is range partitioned on L_PARTKEY but several of the queries have where clause predicates on the SHIPDATE column. By loading the data in the Lineitem table sorted by SHIPDATE, the storage index can be used to eliminate data based on the SHIPDATE.

In order for the data to be fully sorted during the load you MUST load using an INSERT /*+ APPEND */ statement with an order by clause on the column you want indexed. Before doing the load you should examine the execution plan to check that the sort and the load are being done by the same set of parallel server processes and that no data redistribution takes place between the sort and the load steps in the plan.

The same document also contained this nifty graphic. You get the idea though right? Keep a min/max value for each 1M storage region.

The idea of using the Storage Index as an additional partitioning strategy is a very interesting. But as you can see from my example, Storage Indexes can also provide a huge benefit for highly skewed data. This is an area that Oracle has traditionally struggled with. The typical scenario involves using an index for highly selective values and using a table scan for non-selective values. This choice is complicated by Oracle’s default bind variable peeking behavior. With Exadata, we may find that we are better off just ignoring the issue altogether. The full table scan will pretty much always provide the best performance possible, without the possibility of bind variable peeking or the overhead of a normal B-Tree index (that is to say, we probably won’t create the index at all). The storage index will provide very quick access to the highly selective values while the normal Exadata Offloading will handle the non-selective values better than we ever thought possible.

By the way, Netezza has a very similar “feature” called zone maps, although they apparently use a 3M storage unit. The larger unit means that it would be much more sensitive to how the data is sorted on disk, at least for the “virtual” partition elimination behavior.

One last thing. There is a way to turn off Storage Indexes on Exadata via a hidden parameter (_kcfis_storageidx_disabled), if you should ever want to do that (just for testing the feature perhaps).

Here’s a quick example:

^?View Code NONE

SYS@LABRAT1> select name, value from v$mystat s, v$statname n where n.statistic# = s.statistic# and name like '%storage%';
 
NAME                                                                             VALUE
---------------------------------------------------------------------- ---------------
cell physical IO bytes saved by storage index                                        0
 
Elapsed: 00:00:00.01
SYS@LABRAT1> alter session set "_kcfis_storageidx_disabled"=true -- turn them off
  2  /
 
Session altered.
 
Elapsed: 00:00:00.00
SYS@LABRAT1> select count(*) from kso.skew3 where col1 is null;
 
  COUNT(*)
----------
        12
 
Elapsed: 00:00:13.91
SYS@LABRAT1> select name, value from v$mystat s, v$statname n where n.statistic# = s.statistic# and name like '%storage%';
 
NAME                                                                             VALUE
---------------------------------------------------------------------- ---------------
cell physical IO bytes saved by storage index                                        0
 
Elapsed: 00:00:00.00
SYS@LABRAT1> alter session set "_kcfis_storageidx_disabled"=false -- back to default which turns it on
  2  /
 
Session altered.
 
Elapsed: 00:00:00.00
SYS@LABRAT1> select count(*) from kso.skew3 where col1 is null;
 
  COUNT(*)
----------
        12
 
Elapsed: 00:00:00.07
SYS@LABRAT1> select name, value from v$mystat s, v$statname n where n.statistic# = s.statistic# and name like '%storage%';
 
NAME                                                                             VALUE
---------------------------------------------------------------------- ---------------
cell physical IO bytes saved by storage index                              16012763136
 
Elapsed: 00:00:00.01

As always, your comments are welcomed.

Oracle Support Sanctions Manually Created SQL Profiles!

osborne — Sat, 24 Jul 2010 20:00:45 +0000

I originally titled this post: “SQLT - coe_xfr_sql_profile.sql”

Catchy title huh? - (that’s why I changed it)

I’ve been promoting the use of SQL Profiles as a plan control mechanism for some time. The basic idea is to use the undocumented procedure dbms_sqltune.import_sql_profile to build a set of hints to be applied behind the scenes via a SQL Profile. The hints can be created anyway can think of, but one of my favorite ways to generate them is to pull the hints from the other_xml filed of v$sql. This is a technique suggested to me originally by Randolf Geist. I have used this approach several times in the past but occasionally I’ve had a few doubts as to whether this is a good idea or even if SQL Profiles can apply all valid hints (see Jonathan Lewis’s comments on this post, Why Oracle Isn’t Using My Profile, where he expresses some doubts as well - he’s also written a bit about SQL Profiles on his site as you might imagine).

So anyway, I just found out this week that there is a script published on Oracle’s Support site that does exactly the same thing. It’s part of the SQLT zip file published in note 215187.1. By the way, SQLT has quite a bit of interesting information in it and the source (PL/SQL) is not wrapped, so it’s worth having a look at. There’s not much in the way of information about it out there, although I did see a reference to it in a comment on one of Jonathan’s recent posts. Maybe I’ll get around to doing another post on that topic some other time. Anyway, the name of the SQL Profile building script is coe_xfr_sql_profile.sql. It basically pulls the hints from the other_xml field of v$sql and turns them into a SQL Profile. So I’m feeling better about myself now that I know that this approach is at least in some way sanctioned by Oracle support.

Here’s an example:

^?View Code NONE

 
SYS@LAB112> @fs
Enter value for sql_text: %skew%
Enter value for sql_id: 
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
688rj6tv1bav0      0  568322376          1          6.78      163,077 select avg(pk_col) from kso.skew where col1 = 1
abwg9nwg8prsj      0 3723858078          1           .01           39 select avg(pk_col) from kso.skew where col1 = 136135
 
2 rows selected.
 
SYS@LAB112> @sql_hints
Enter value for sql_id: abwg9nwg8prsj
Enter value for child_no: 0
 
OUTLINE_HINTS
-----------------------------------------------------------------------------------------------------------------------------------------------------------
IGNORE_OPTIM_EMBEDDED_HINTS
OPTIMIZER_FEATURES_ENABLE('11.2.0.1')
DB_VERSION('11.2.0.1')
ALL_ROWS
OUTLINE_LEAF(@"SEL$1")
INDEX_RS_ASC(@"SEL$1" "SKEW"@"SEL$1" ("SKEW"."COL1"))
 
6 rows selected.
 
SYS@LAB112> @coe_xfr_sql_profile    
 
Parameter 1:
SQL_ID (required)
 
Enter value for 1: abwg9nwg8prsj          
 
 
PLAN_HASH_VALUE AVG_ET_SECS
--------------- -----------
     3723858078        .006
 
Parameter 2:
PLAN_HASH_VALUE (required)
 
Enter value for 2: 3723858078
 
Values passed:
~~~~~~~~~~~~~
SQL_ID         : "abwg9nwg8prsj"
PLAN_HASH_VALUE: "3723858078"
 
 
Execute coe_xfr_sql_profile_abwg9nwg8prsj_3723858078.sql
on TARGET system in order to create a custom SQL Profile
with plan 3723858078 linked to adjusted sql_text.
 
 
COE_XFR_SQL_PROFILE completed.
SQL>@coe_xfr_sql_profile_abwg9nwg8prsj_3723858078.sql
SQL>REM
SQL>REM $Header: 215187.1 coe_xfr_sql_profile_abwg9nwg8prsj_3723858078.sql 11.4.1.4 2010/07/23 csierra $
SQL>REM
SQL>REM Copyright (c) 2000-2010, Oracle Corporation. All rights reserved.
SQL>REM
SQL>REM AUTHOR
SQL>REM   carlos.sierra@oracle.com
SQL>REM
SQL>REM SCRIPT
SQL>REM   coe_xfr_sql_profile_abwg9nwg8prsj_3723858078.sql
SQL>REM
SQL>REM DESCRIPTION
SQL>REM   This script is generated by coe_xfr_sql_profile.sql
SQL>REM   It contains the SQL*Plus commands to create a custom
SQL>REM   SQL Profile for SQL_ID abwg9nwg8prsj based on plan hash
SQL>REM   value 3723858078.
SQL>REM   The custom SQL Profile to be created by this script
SQL>REM   will affect plans for SQL commands with signature
SQL>REM   matching the one for SQL Text below.
SQL>REM   Review SQL Text and adjust accordingly.
SQL>REM
SQL>REM PARAMETERS
SQL>REM   None.
SQL>REM
SQL>REM EXAMPLE
SQL>REM   SQL> START coe_xfr_sql_profile_abwg9nwg8prsj_3723858078.sql;
SQL>REM
SQL>REM NOTES
SQL>REM   1. Should be run as SYSTEM or SYSDBA.
SQL>REM   2. User must have CREATE ANY SQL PROFILE privilege.
SQL>REM   3. SOURCE and TARGET systems can be the same or similar.
SQL>REM   4. To drop this custom SQL Profile after it has been created:
SQL>REM  EXEC DBMS_SQLTUNE.DROP_SQL_PROFILE('coe_abwg9nwg8prsj_3723858078');
SQL>REM   5. Be aware that using DBMS_SQLTUNE requires a license
SQL>REM  for the Oracle Tuning Pack.
SQL>REM
SQL>WHENEVER SQLERROR EXIT SQL.SQLCODE;
SQL>REM
SQL>VAR signature NUMBER;
SQL>REM
SQL>DECLARE
  2  sql_txt CLOB;
  3  h       SYS.SQLPROF_ATTR;
  4  BEGIN
  5  sql_txt := q'[
  6  select avg(pk_col) from kso.skew where col1 = 136135
  7  ]';
  8  h := SYS.SQLPROF_ATTR(
  9  q'[BEGIN_OUTLINE_DATA]',
 10  q'[IGNORE_OPTIM_EMBEDDED_HINTS]',
 11  q'[OPTIMIZER_FEATURES_ENABLE('11.2.0.1')]',
 12  q'[DB_VERSION('11.2.0.1')]',
 13  q'[ALL_ROWS]',
 14  q'[OUTLINE_LEAF(@"SEL$1")]',
 15  q'[INDEX_RS_ASC(@"SEL$1" "SKEW"@"SEL$1" ("SKEW"."COL1"))]',
 16  q'[END_OUTLINE_DATA]');
 17  :signature := DBMS_SQLTUNE.SQLTEXT_TO_SIGNATURE(sql_txt);
 18  DBMS_SQLTUNE.IMPORT_SQL_PROFILE (
 19  sql_text    => sql_txt,
 20  profile     => h,
 21  name        => 'coe_abwg9nwg8prsj_3723858078',
 22  description => 'coe abwg9nwg8prsj 3723858078 '||:signature||'',
 23  category    => 'DEFAULT',
 24  validate    => TRUE,
 25  replace     => TRUE,
 26  force_match => FALSE /* TRUE:FORCE (match even when different literals in SQL). FALSE:EXACT (similar to CURSOR_SHARING) */ );
 27  END;
 28  /
 
PL/SQL procedure successfully completed.
 
SQL>WHENEVER SQLERROR CONTINUE
SQL>SET ECHO OFF;
 
            SIGNATURE
---------------------
 15022055147995020558
 
 
... manual custom SQL Profile has been created
 
 
COE_XFR_SQL_PROFILE_abwg9nwg8prsj_3723858078 completed
 
SYS@LAB112> @sql_profiles
Enter value for sql_text: 
Enter value for name: 
 
NAME                           CATEGORY        STATUS   SQL_TEXT                                                               FORCE
------------------------------ --------------- -------- ---------------------------------------------------------------------- -----
PROFILE_fgn6qzrvrjgnz          DEFAULT         DISABLED select /*+ index(a SKEW_COL1) */ avg(pk_col) from kso.skew a           NO
PROFILE_8hjn3vxrykmpf          DEFAULT         DISABLED select /*+ invalid_hint (doda) */ avg(pk_col) from kso.skew where col1 NO
PROFILE_69k5bhm12sz98          DEFAULT         DISABLED SELECT dbin.instance_number,        dbin.db_name, dbin.instance_name,  NO
PROFILE_8js5bhfc668rp          DEFAULT         DISABLED select /*+ index(a SKEW_COL2_COL1) */ avg(pk_col) from kso.skew a wher NO
PROFILE_bxd77v75nynd8          DEFAULT         DISABLED select /*+ parallel (a 4) */ avg(pk_col) from kso.skew a where col1 >  NO
PROFILE_7ng34ruy5awxq          DEFAULT         DISABLED select i.obj#,i.ts#,i.file#,i.block#,i.intcols,i.type#,i.flags,i.prope NO
SYS_SQLPROF_0126f1743c7d0005   SAVED           ENABLED  select avg(pk_col) from kso.skew                                       NO
PROF_6kymwy3guu5uq_1388734953  DEFAULT         ENABLED  select 1                                                               YES
PROFILE_cnpx9s9na938m_MANUAL   DEFAULT         ENABLED  select /*+ opt_param('statistics_level','all') */ * from kso.skew wher NO
PROF_79m8gs9wz3ndj_3723858078  DEFAULT         ENABLED  /* SQL Analyze(252,1) */ select avg(pk_col) from kso.skew              NO
PROFILE_9ywuaagwscbj7_GPS      DEFAULT         ENABLED  select avg(pk_col) from kso.skew                                       NO
PROF_arcvrg5na75sw_3723858078  DEFAULT         ENABLED  select /*+ index(skew@sel$1 skew_col1) */ avg(pk_col) from kso.skew wh NO
SYS_SQLPROF_01274114fc2b0006   DEFAULT         ENABLED  select i.table_owner, i.table_name, i.index_name, FUNCIDX_STATUS, colu NO
SYS_SQLPROF_0127d10ffaa60000   DEFAULT         ENABLED  select table_owner||'.'||table_name tname , index_name, index_type, st NO
SYS_SQLPROF_01281e513ace0000   DEFAULT         ENABLED  SELECT TASK_LIST.TASK_ID FROM (SELECT /*+ NO_MERGE(T) ORDERED */ T.TAS NO
PROFILE_5bgcrdwfhbc83_EXACT    DEFAULT         ENABLED  select avg(pk_col) from kso.skew where col1 = :"SYS_B_0"               YES
coe_abwg9nwg8prsj_3723858078   DEFAULT         ENABLED                                                                         NO
 
17 rows selected.
 
SYS@LAB112> -- that's interesting - looks like the sql_text has gotten wiped out
SYS@LAB112> -- let's see if it works anyway
SYS@LAB112> 
SYS@LAB112> select avg(pk_col) from kso.skew where col1 = 136135;
 
AVG(PK_COL)
-----------
   15636135
 
SYS@LAB112> @fs
Enter value for sql_text: select avg(pk_col) from kso.skew where col1 = 136135
Enter value for sql_id: 
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
abwg9nwg8prsj      0 3723858078          1           .02           47 select avg(pk_col) from kso.skew where col1 = 136135
 
1 row selected.
 
SYS@LAB112> @dplan
Enter value for sql_id: abwg9nwg8prsj
Enter value for child_no: 
 
PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  abwg9nwg8prsj, child number 0
-------------------------------------
select avg(pk_col) from kso.skew where col1 = 136135
 
Plan hash value: 3723858078
 
------------------------------------------------------------------------------------------
| Id  | Operation                    | Name      | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT             |           |       |       |    32 (100)|          |
|   1 |  SORT AGGREGATE              |           |     1 |    24 |            |          |
|   2 |   TABLE ACCESS BY INDEX ROWID| SKEW      |    32 |   768 |    32   (0)| 00:00:01 |
|*  3 |    INDEX RANGE SCAN          | SKEW_COL1 |    32 |       |     3   (0)| 00:00:01 |
------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   3 - access("COL1"=136135)
 
Note
-----
   - SQL profile coe_abwg9nwg8prsj_3723858078 used for this statement
 
 
24 rows selected.

So it is very similar to my create_sql_profile.sql script. The Oracle COE script does have the advantage of creating an output script that can be run to create the SQL Profile. That means you have a chance to edit the hints before creating the SQL Profile. It also means you can easily move a SQL Profile from one environment (TEST for example) to another (PROD for example).

But the best thing about it is that I no longer have to be concerned about using an undocumented procedure to do something that it may not have been intended to do in the first place!

Writing Some Chapters in a Book

osborne — Mon, 12 Jul 2010 01:49:27 +0000

My friend Karen Morton asked if I would be willing to contribute to a book that she is working on (i.e. write a few chapters). Of course I said yes. The book’s title is Pro Oracle SQL and it is to be published by Apress sometime before the end of the year (my first deadline is fast approaching).

Karen is the lead author, but there are also several co-authors involved in this project, all of whom I have a lot of respect for. Here’s the List (in alphabetical order by last name):

Robyn Sands
Riyaj Shamsudeen
Jared Still

If you’re reading this you are probably already familiar with Apress. They have published a number of Oracle books by notable authors including Tom Kyte, Jonathan Lewis and Chris Antognini. They have also published a few collaborations by members of the Oak Table Network. So I am happy to be joining that illustrious group. Anyway, I am particularly excited about getting to write the chapter on Plan Stability and Control which is a subject near and dear to my heart. So I’ll get to drone on about Outlines, Profiles, and Baselines - among other things. You can pre-order the book from Amazon. ;)

Exadata and Parallel Queuing

osborne — Thu, 08 Jul 2010 03:31:08 +0000

Over the years Oracle has added many enhancements in order to allow individual SQL statements to take full advantage of multiprocessor computers. A few months ago Cary Millsap did a talk where he recalled the presentation Larry Ellison did when Oracle first announced the Parallel Query feature. During Larry’s demo he had a multiprocessor computer all to himself. I don’t remember how many processors it had, but I remember he had some kind of graphic showing individual CPU utilization on one screen while he fired up a parallel query on another screen. The monitoring screen lit up like a Christmas tree. Every one of the CPU’s was pegged during his demo. When Cary was telling the story he said that he had wondered at the time what would have happened if there had been other users on the system during the demo. Their experience would probably not have been a good one. I remember having the exact same thought.

Oracle’s parallel capabilities have been a great gift but they have also been a curse because controlling the beast in an environment where there are multiple users trying to share the resources is pretty difficult. There have been many attempts at coming up with a reasonable way of throttling big parallel statements along the way. But to date, I think this technology has only been used effectively in batch processing environments and large data warehouses where consuming the whole machine’s resources is acceptable due to the relatively low degree of concurrency required by those environments.

So why did I mention Exadata in the title of this post? Well I believe that one of the most promising aspects of Exadata is it’s potential with regard to running mixed work loads (OLTP and DW) on the same database, without crippling one or the other. In order to do that, Oracle needs some mechanism to separate the workloads. Resource Manager is an option in this area, but it doesn’t go far enough in controlling throughput on parallel queries. This new queuing mechanism should be a great help in that regard. So let’s review the options:

Parallel Adaptive Multi User (the old way)
This ability to automatically downgrade the degree of parallelism based on what’s happening on the system when a query kicks off is actually a powerful mechanism and is the best approach we’ve had prior to 11g Release 2. The downside of this approach is that parallelized statements can have wildly varying execution times. As you can imagine, a statement that gets 32 slaves one time and then gets downgraded to serial execution the next time will probably not make the user very happy. The argument for this type of approach is that stuff is going to run slower if the system is busy regardless of what you do. And that users expect it to run slower when the system is busy. The first part of that statement may be true but I don’t believe the second part is (at least in most cases). The bigger problem with the downgrade mechanism though is that the the decision about how many slaves to use is based on a single point in time (the point when the parallel statement starts). And once the degree of parallelism (DOP) is set for a statement it can not be changed. That execution of the statement will run with the number of slaves it got to start with, even if additional resources become available. So consider the statement that takes a minute with 32 slaves that gets downgraded to serial due to a momentarily high load. And say that 10 seconds after it starts the system load drops back to more normal levels. Unfortunately, the serialized statement will continue to run for nearly 30 minutes with it’s single process, even though on average the system is not busier than usual.

Parallel Queuing (the new way)
Now let’s compare that with the new mechanism introduced in 11gR2 that allows parallel statements to be queued in a First In - First Out fashion. This mechanism separates (presumably) long running parallel queries from the rest of the workload. The mechanics are pretty simple. Turn the feature on. Set a target number of parallel slaves (parallel_servers_target). Run stuff. If a statement tries to start that requires exceeding the target, it will be queued until the required number of slaves become available.

The Parallel Queuing feature is controlled by a hidden parameter called “_parallel_statement_queuing”. A value of TRUE turns it on and FALSE turns it off. FALSE is the default by the way. This parameter is not documented but is set automatically when the PARALLEL_DEGREE_POLICY parameter is set to AUTO. Unfortunately, PARALLEL_DEGREE_POLICY is one of those parameters that controls more than one thing. When set to AUTO it also turns on Automatic DOP calculation. This feature calculates a DOP for each statement regardless of whether any objects have been decorated with a parallel setting. The result is that all kinds of statements are run in parallel, even if no objects have been specifically defined with a parallel degree setting. This is truly automatic parallel processing because the database decides what to run in parallel and with how many slaves. On top of that, by default, the slaves may be spread across multiple nodes in a RAC database (this can be disabled by setting PARALLEL_FORCE_LOCAL to TRUE). Finally, AUTO is supposed to enable “In Memory Parallel Query”. This poorly named feature refers to 11gR2’s ability to make use of the SGA for parallel query, as opposed to using direct reads exclusively. Note: I haven’t actually seen this kick in yet, which is probably good, since Exadata Offloading depends on direct reads. I haven’t seen it kick in on non-Exadata databases either though.

Unfortunately this combination of features is a little like the wild west with things running in parallel all over the place. But the ability to queue parallel statements does provide some semblance of order. And to be fair, there are a number of parameters that can be set to control how the calculations are performed. Anyway, here’s a brief synopsis of parameter changes caused by the various settings PARALLEL_DEGREE_POLICY.

Parameters Affected by Parallel_Degree_Policy
Parallel_Degree_Policy	Parameter	Value
MANUAL	_parallel_statement_queuing	FALSE
	_parallel_cluster_cache_policy	ADAPTIVE
LIMITED	_parallel_statement_queuing	FALSE
	_parallel_cluster_cache_policy	ADAPTIVE
AUTO	_parallel_statement_queuing	TRUE
	_parallel_cluster_cache_policy	CACHED

So let’s look at how it behaves:

^?View Code NONE

-bash-3.2$ !sql
sqlplus / as sysdba
 
SQL*Plus: Release 11.2.0.1.0 Production on Mon Jul 5 13:10:26 2010
 
Copyright (c) 1982, 2009, Oracle.  All rights reserved.
 
 
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
 
EXADATA> @parms 
Enter value for parameter: parallel
Enter value for isset: 
Enter value for show_hidden: 
 
NAME                                               VALUE                                                                  ISDEFAUL ISMODIFIED ISSET
-------------------------------------------------- ---------------------------------------------------------------------- -------- ---------- ----------
fast_start_parallel_rollback                       LOW                                                                    TRUE     FALSE      FALSE
parallel_adaptive_multi_user                       TRUE                                                                   TRUE     FALSE      FALSE
parallel_automatic_tuning                          FALSE                                                                  TRUE     FALSE      FALSE
parallel_degree_limit                              CPU                                                                    TRUE     FALSE      FALSE
parallel_degree_policy                             AUTO                                                                   TRUE     TRUE       TRUE
parallel_execution_message_size                    16384                                                                  TRUE     FALSE      FALSE
parallel_force_local                               FALSE                                                                  TRUE     FALSE      FALSE
parallel_instance_group                                                                                                   TRUE     FALSE      FALSE
parallel_io_cap_enabled                            FALSE                                                                  TRUE     FALSE      FALSE
parallel_max_servers                               160                                                                    TRUE     FALSE      FALSE
parallel_min_percent                               0                                                                      TRUE     FALSE      FALSE
parallel_min_servers                               0                                                                      TRUE     FALSE      FALSE
parallel_min_time_threshold                        AUTO                                                                   TRUE     FALSE      FALSE
parallel_server                                    TRUE                                                                   TRUE     FALSE      FALSE
parallel_server_instances                          2                                                                      TRUE     FALSE      FALSE
parallel_servers_target                            64                                                                     TRUE     FALSE      FALSE
parallel_threads_per_cpu                           2                                                                      TRUE     FALSE      FALSE
recovery_parallelism                               0                                                                      TRUE     FALSE      FALSE
 
18 rows selected.
 
EXADATA> select owner, table_name, degree from dba_tables where table_name = 'SKEW';
 
OWNER      TABLE_NAME                     DEGREE
---------- ------------------------------ ------
KSO        SKEW                                1
 
EXADATA> set echo on
EXADATA> @avgskew
EXADATA> select avg(pk_col) from kso.skew a where col1 > 0
  2  /
 
AVG(PK_COL)
-----------
 16093749.3
 
EXADATA> @fs
Enter value for sql_text: select avg(pk_col) from kso.skew a where col1 > 0
Enter value for sql_id: 
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
05cq2hb1r37tr      0  568322376          1         28.77      162,427 select avg(pk_col) from kso.skew a where col1 > 0
 
1 row selected.
 
EXADATA> @dplan
Enter value for sql_id: 05cq2hb1r37tr
Enter value for child_no: 
 
PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  05cq2hb1r37tr, child number 0
-------------------------------------
select avg(pk_col) from kso.skew a where col1 > 0
 
Plan hash value: 568322376
 
---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |       |       | 28420 (100)|          |
|   1 |  SORT AGGREGATE    |      |     1 |    24 |            |          |
|*  2 |   TABLE ACCESS FULL| SKEW |    32M|   732M| 28420   (2)| 00:05:42 |
---------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   2 - filter("COL1">0)
 
 
19 rows selected.
 
EXADATA> alter system set parallel_degree_policy=auto;
 
System altered.
 
EXADATA> @avgskew
 
AVG(PK_COL)
-----------
 16093749.3
 
1 row selected.
 
EXADATA> @fs
Enter value for sql_text: 
Enter value for sql_id: 05cq2hb1r37tr
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
05cq2hb1r37tr      0  568322376          1         28.77      162,427 select avg(pk_col) from kso.skew a where col1 > 0
05cq2hb1r37tr      2  578366071          1         40.33      162,437 select avg(pk_col) from kso.skew a where col1 > 0
 
2 rows selected.
 
EXADATA> @dplan
Enter value for sql_id: 05cq2hb1r37tr
Enter value for child_no: 2
 
PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  05cq2hb1r37tr, child number 2
-------------------------------------
select avg(pk_col) from kso.skew a where col1 > 0
 
Plan hash value: 578366071
 
----------------------------------------------------------------------------------------------------------------
| Id  | Operation              | Name     | Rows  | Bytes | Cost (%CPU)| Time     |    TQ  |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT       |          |       |       |  6308 (100)|          |        |      |            |
|   1 |  SORT AGGREGATE        |          |     1 |    24 |            |          |        |      |            |
|   2 |   PX COORDINATOR       |          |       |       |            |          |        |      |            |
|   3 |    PX SEND QC (RANDOM) | :TQ10000 |     1 |    24 |            |          |  Q1,00 | P->S | QC (RAND)  |
|   4 |     SORT AGGREGATE     |          |     1 |    24 |            |          |  Q1,00 | PCWP |            |
|   5 |      PX BLOCK ITERATOR |          |    32M|   732M|  6308   (1)| 00:01:16 |  Q1,00 | PCWC |            |
|*  6 |       TABLE ACCESS FULL| SKEW     |    32M|   732M|  6308   (1)| 00:01:16 |  Q1,00 | PCWP |            |
----------------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   6 - access(:Z>=:Z AND :Z<=:Z)
       filter("COL1">0)
 
Note
-----
   - automatic DOP: Computed Degree of Parallelism is 5
 
 
28 rows selected.
 
EXADATA> alter system set parallel_degree_policy=auto;
 
System altered.
 
EXADATA> !ss.sh avgskew.sql 20 kso/kso
 
starting 20 copies of avgskew.sql
 
EXADATA> set echo on
EXADATA> @queued_sql
EXADATA> col sql_text for a60 trunc
EXADATA> SELECT sid, sql_id, sql_exec_id, sql_text
  2  from v$sql_monitor
  3  WHERE status='QUEUED'
  4  order by 3
  5  /
 
 SID SQL_ID        SQL_EXEC_ID SQL_TEXT
---- ------------- ----------- ------------------------------------------------------------
 293 05cq2hb1r37tr    16777265 select avg(pk_col) from kso.skew a where col1 > 0
 270 05cq2hb1r37tr    16777266 select avg(pk_col) from kso.skew a where col1 > 0
   7 05cq2hb1r37tr    16777267 select avg(pk_col) from kso.skew a where col1 > 0
  22 05cq2hb1r37tr    16777268 select avg(pk_col) from kso.skew a where col1 > 0
  42 05cq2hb1r37tr    16777269 select avg(pk_col) from kso.skew a where col1 > 0
 101 05cq2hb1r37tr    16777270 select avg(pk_col) from kso.skew a where col1 > 0
  78 05cq2hb1r37tr    16777271 select avg(pk_col) from kso.skew a where col1 > 0
  60 05cq2hb1r37tr    16777272 select avg(pk_col) from kso.skew a where col1 > 0
 118 05cq2hb1r37tr    16777273 select avg(pk_col) from kso.skew a where col1 > 0
 160 05cq2hb1r37tr    16777274 select avg(pk_col) from kso.skew a where col1 > 0
 137 05cq2hb1r37tr    16777275 select avg(pk_col) from kso.skew a where col1 > 0
 181 05cq2hb1r37tr    16777276 select avg(pk_col) from kso.skew a where col1 > 0
 199 05cq2hb1r37tr    16777277 select avg(pk_col) from kso.skew a where col1 > 0
 216 05cq2hb1r37tr    16777278 select avg(pk_col) from kso.skew a where col1 > 0
 
14 rows selected.
 
EXADATA> -- using Tanel Poder's snapper - 
EXADATA> @snapper ash=sid+event+wait_class,ash1=plsql_object_id+plsql_subprogram_id+sql_id 5 1 all
Sampling with interval 5 seconds, 1 times...
 
-- Session Snapper v3.11 by Tanel Poder @ E2SN ( http://tech.e2sn.com )
 
 
--------------------------------------------------------------
Active% |    SID | EVENT                     | WAIT_CLASS
--------------------------------------------------------------
   100% |    118 | enq: JX - SQL statement q | Scheduler
   100% |    216 | enq: JX - SQL statement q | Scheduler
   100% |     78 | enq: JX - SQL statement q | Scheduler
   100% |     63 | enq: JX - SQL statement q | Scheduler
   100% |    233 | enq: JX - SQL statement q | Scheduler
   100% |    199 | enq: JX - SQL statement q | Scheduler
   100% |    137 | enq: JX - SQL statement q | Scheduler
   100% |     96 | enq: JX - SQL statement q | Scheduler
   100% |    101 | enq: JX - SQL statement q | Scheduler
   100% |     27 | enq: JX - SQL statement q | Scheduler
 
---------------------------------------------------
Active% | PLSQL_OBJE | PLSQL_SUBP | SQL_ID
---------------------------------------------------
  2095% |            |            | 05cq2hb1r37tr
     3% |            |            |
 
--  End of ASH snap 1, end=2010-07-04 20:18:05, seconds=5, samples_taken=39
 
 
PL/SQL procedure successfully completed.

There are several things worth mentioning in this example. First we see that with PARALLEL_DEGREE_POLICY set to it’s default value of MANUAL, my avgskew.sql script runs a serial plan. Second we see that when we set PARALLEL_DEGREE_POLICY to AUTO, the statement is automatically parallelized. Notice the note at the bottom of the plan output and you see that Automatic DOP was set at 5. Magical! I then used a shell script (ss.sh) to fire off 20 copies of the avgskew.sql script in rapid succession. Querying V$SQL_MONITOR showed that the statements were indeed queuing. Finally, I ran Tanel Poder’s snapper to see what event the queued statements were waiting on. Turns out it was “enq: JX - SQL statement queue”. Note that there is also an event called “PX Queuing: statement queue” that we didn’t see in the snapper output. I’m not exactly sure why Oracle split these two events apart, but apparently “PX Queuing: statement queue” is the event that clocks time when a statement is next in line, while “enq: JX - SQL statement queue” is used when a statement is in the queue but not the next one up. Guy Harrison has a really good post on the parallel_degree_policy that covers these two events here.

Oracle has provided some control over the Automatic DOP calculations as well with the PARALLEL_DEGREE_LIMIT parameter. The default value for this parameter is CPU which comes up with an “Ideal DOP” based on the amount of data, but then caps it with a formula based on CPU_COUNT, THREADS_PER_CPU and ACTIVE_INSTANCE_COUNT. There is a good description in this an Oracle white paper Parallel Execution Fundamentals that covers this calculation.

Of course, automatic DOP calculations are still a little scary. So it’s nice that there is a way to turn on the Parallel Queuing feature without enabling the Automatic DOP. Here’s another quick example showing that Queuing can be turned on without the other features:

^?View Code NONE

EXADATA> alter system set parallel_degree_policy=manual;
 
System altered.
 
EXADATA> @flush_pool
 
System altered.
 
EXADATA> @avgskew
 
AVG(PK_COL)
-----------
 16093749.3
 
EXADATA> @fs
Enter value for sql_text: %avg%kso.skew%
Enter value for sql_id: 
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
05cq2hb1r37tr      0  568322376          1         10.92      162,496 select avg(pk_col) from kso.skew a where col1 > 0
 
1 row selected.
 
EXADATA> @dplan
Enter value for sql_id: 05cq2hb1r37tr
Enter value for child_no: 
 
PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  05cq2hb1r37tr, child number 0
-------------------------------------
select avg(pk_col) from kso.skew a where col1 > 0
 
Plan hash value: 568322376
 
---------------------------------------------------------------------------
| Id  | Operation          | Name | Rows  | Bytes | Cost (%CPU)| Time     |
---------------------------------------------------------------------------
|   0 | SELECT STATEMENT   |      |       |       | 28420 (100)|          |
|   1 |  SORT AGGREGATE    |      |     1 |    24 |            |          |
|*  2 |   TABLE ACCESS FULL| SKEW |    32M|   732M| 28420   (2)| 00:05:42 |
---------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   2 - filter("COL1">0)
 
 
19 rows selected.
 
EXADATA> -- so with MANUAL - the DOP is not set automatically
EXADATA> -- let's set the table to have a parallel degree
EXADATA>
EXADATA> alter table kso.skew parallel (degree 6);
 
Table altered.
 
EXADATA> @avgskew
 
AVG(PK_COL)
-----------
 16093749.3
 
1 row selected.
 
EXADATA> @fs
Enter value for sql_text: 
Enter value for sql_id: 05cq2hb1r37tr
 
SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO SQL_TEXT
------------- ------ ---------- ---------- ------------- ------------ ------------------------------------------------------------
05cq2hb1r37tr      0  578366071          1         13.93      142,437 select avg(pk_col) from kso.skew a where col1 > 0
 
1 row selected.
 
EXADATA> @dplan
Enter value for sql_id: 05cq2hb1r37tr
Enter value for child_no: 
 
PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  05cq2hb1r37tr, child number 0
-------------------------------------
select avg(pk_col) from kso.skew a where col1 > 0
 
Plan hash value: 578366071
 
----------------------------------------------------------------------------------------------------------------
| Id  | Operation              | Name     | Rows  | Bytes | Cost (%CPU)| Time     |    TQ  |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT       |          |       |       |  5256 (100)|          |        |      |            |
|   1 |  SORT AGGREGATE        |          |     1 |    24 |            |          |        |      |            |
|   2 |   PX COORDINATOR       |          |       |       |            |          |        |      |            |
|   3 |    PX SEND QC (RANDOM) | :TQ10000 |     1 |    24 |            |          |  Q1,00 | P->S | QC (RAND)  |
|   4 |     SORT AGGREGATE     |          |     1 |    24 |            |          |  Q1,00 | PCWP |            |
|   5 |      PX BLOCK ITERATOR |          |    32M|   732M|  5256   (1)| 00:01:04 |  Q1,00 | PCWC |            |
|*  6 |       TABLE ACCESS FULL| SKEW     |    32M|   732M|  5256   (1)| 00:01:04 |  Q1,00 | PCWP |            |
----------------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   6 - access(:Z>=:Z AND :Z<=:Z)
       filter("COL1">0)
 
 
24 rows selected.
 
EXADATA>  !ss.sh avgskew.sql 20 kso/kso
 
starting 20 copies of avgskew.sql
 
EXADATA> @queued_sql
 
no rows selected
 
EXADATA> -- so now it's parallel but only objects or statements that we define as parallel
EXADATA> -- but no queuing, let's turn on queuing
EXADATA>
EXADATA> alter system set "_parallel_statement_queuing"=true;
 
System altered.
 
EXADATA> !ss.sh avgskew.sql 20 kso/kso
 
starting 20 copies of avgskew.sql 
 
EXADATA> @queued_sql
 
 SID SQL_ID        SQL_EXEC_ID SQL_TEXT
---- ------------- ----------- ------------------------------------------------------------
 181 05cq2hb1r37tr    16777363 select avg(pk_col) from kso.skew a where col1 > 0
 216                  16777364
  44                  16777365
 273                  16777366
 234                  16777367
 160                  16777368
 100                  16777369
 233                  16777370
  30                  16777371
 138                  16777372
   7                  16777373
 293                  16777374
 137                  16777375
  83                  16777376
 251                  16777377
  66                  16777378
 123                  16777379
 195                  16777380
 
18 rows selected.
 
EXADATA> -- now we have control of which statements are parallelized and we have queuing

This example shows that you can have Parallel Queuing without turning on Automatic DOP. Of course it’s using a hidden parameter, so don’t do this without checking with your mother first (I mean Oracle support). Maybe they will make this a real parameter in a future release (one can only hope). Note that there are hints to turn this feature on and off on a statement level as well (STATEMENT_QUEUING and NO_STATEMENT_QUEUING).

So why did I mention Exadata again? Mixed workloads require a mechanism that forces parallel queries to play nicely with others. (kind of like kindergarten) Parallel Queuing provides a relatively simple mechanism for doing just that. I also mentioned Resource Manager (RM) in passing. That’s another option at our disposal. It does have the ability to curb parallel processing as well and it’s enabled by default in 11gR2. RM, along with it’s kissing cousin IORM, are key components in consolidation strategies on Exadata as well. But we’ll have to leave that for another post. Some much to do and so little time.

Exadata Offload - The Secret Sauce

osborne — Wed, 16 Jun 2010 04:12:10 +0000

The “Secret Sauce” for Exadata is it’s ability to offload processing to the storage tier. Offloading means that the storage servers can apply predicate filters at the storage layer, instead of shipping every possible block back to the database server(s). Another thing that happens with offloading is that the volume of data returned can be further reduced by column projection (i.e. if you only select 1 column from a 100 column table, there is no need to return the other 99 columns). Offloading is geared to long running queries that access a large amount of data. Offloading only works if the Oracle decides to use it’s direct path read mechanism. Direct path reads have traditionally been done by parallel query slaves but can also be done by serial queries. In fact, as of 11g, Oracle has changed the decision making process resulting in more aggressive use of serial direct path reads. I’ve seen this feature described as “serial direct path reads” and “adaptive direct path reads”.

I’ll digress here a bit to discuss this feature since direct path reads are critical to Exadata Offloading. Direct path reads do not load blocks into Oracle’s buffer cache. Instead, the data is returned directly to the PGA of the process requesting the data. This means that the data does not have to be in Oracle block format. That means no 8K block that is only partially filled, that may only have a record or two that you’re interested in, containing every column including ones you don’t want, and with additional header information - needs to be shipped back up from the storage layer. Instead, a much more compact result set containing only the columns requested and hopefully only the rows you need are returned. As I said, direct path reads are traditionally used by parallel query slaves. They are also used in a few other instances such as LOB access and sorts that spill over into TEMP. So the ability to use direct path reads is very important to the Exadata platform and thus the changes to the make them more attractive in 11g. Here are a few links to info on the subject of serial direct path reads:

Doug Burns has a good post on 11g serial direct path reads.
Alex Fatkulin has a very good post on some of the factors controlling adaptive direct path reads.
There is a note on MOS (793845.1) on changes in 11g in the heuristics to choose between direct path reads or reads through the buffer cache.
You may also find MOS note (50415.1) on misleading nature of “direct path read” wait events of interest.

Also be aware that direct path reads are only available for full scans (tables or indexes). So any statement that uses an index to get to a row in a table will not use this mechanism. Also keep in mind that direct path requires extra processing to ensure that all blocks on disk are current - (i.e. a check point), so frequently modified tables will suffer some overhead before direct path reads can be initiated.

I must say that I think the changes to the heuristics in 11g may be a little on the aggressive side for non-Exadata platforms (the changes may well be driven by Exadata). And by the way, serial direct path reads are not always faster than the normal reads that go through the buffer cache. Dion Cho has a good post on a performance problem due to serial direct path reads kicking in on one node of an 11g RAC environment (not Exadata). The node doing the direct path reads was running the query much slower than the node using the normal buffer cache reads. He also has a post on turning off serial direct path reads.

But enough about the direct path reads stuff, on to the Offloading. One of the first things I wanted to know when I got my first look at a system running on Exadata was whether a particular query was eligible for offloading and if so, how much of the expected i/o was saved by the feature. So of course I wrote a little script to show me that. Turns out there is plenty of info in V$SQL to see what’s going on. I called the script fsx.sql (short for Find_Sql_eXadata). Here’s a little demo:

^?View Code NONE

SYS@LABRAT1> @test_offload
SYS@LABRAT1> -- ran but turned off output
SYS@LABRAT1> -- select /* test full */ * from kso.skew3; 
SYS@LABRAT1> 
SYS@LABRAT1> select /* test count */ count(col1) from kso.skew3;
 
COUNT(COL1)
-----------
  384000036
 
1 row selected.
 
Elapsed: 00:00:24.04
SYS@LABRAT1> select /* test count(rll) */ count(pk_col), count(col1), count(col2), count(col3), count(col4) from kso.skew3;
 
COUNT(PK_COL) COUNT(COL1) COUNT(COL2) COUNT(COL3) COUNT(COL4)
------------- ----------- ----------- ----------- -----------
    384000048   384000036   384000036   384000036   384000036
 
1 row selected.
 
Elapsed: 00:00:39.09
SYS@LABRAT1> select /* test max */ max(col1) from kso.skew3;
 
 MAX(COL1)
----------
   1000000
 
1 row selected.
 
Elapsed: 00:00:24.51
SYS@LABRAT1> select /* test max all */ max(pk_col), max(col1), max(col2), max(col3), max(col4) from kso.skew3;
 
MAX(PK_COL)  MAX(COL1) MAX(COL2)                      MAX(COL3) M
----------- ---------- ------------------------------ --------- -
 1.0000E+12    1000000 asddsadasd                     01-JAN-09 Y
 
1 row selected.
 
Elapsed: 00:00:57.32
SYS@LABRAT1> select /* test avg */ avg(col1) from kso.skew3;
 
 AVG(COL1)
----------
495000.194
 
1 row selected.
 
Elapsed: 00:00:30.00
SYS@LABRAT1> select /* test avg filter */ avg(col1) from kso.skew3 where col1 between 2 and 10000;
 
 AVG(COL1)
----------
 
 
1 row selected.
 
Elapsed: 00:00:13.97
SYS@LABRAT1> select /* test index */ avg(pk_col) from kso.skew where col1 = 136133;
 
AVG(PK_COL)
-----------
   15636133
 
1 row selected.
 
Elapsed: 00:00:00.00
SYS@LABRAT1> set echo off
SYS@LABRAT1> set timing off
SYS@LABRAT1> @fsx
Enter value for sql_text: % test %
Enter value for sql_id: 
Enter value for inst_id: 
 
 INST SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO    AVG_PIO AVG_PX OFFLOADABLE IO_SAVED_% SQL_TEXT
----- ------------- ------ ---------- ---------- ------------- ------------ ---------- ------ ----------- ---------- ----------------------------------------
    1 1767x3dw3m8pa      0 2684249835          1         57.31    1,956,226  1,956,219      0 Yes               9.35 select /* test max all */ max(pk_col), m
    1 1wp72w4c5d449      0 2684249835          1         29.98    1,956,226  1,956,219      0 Yes              71.84 select /* test avg */ avg(col1) from kso
    1 46vd0hducqms8      0 1667163222          1        422.77    2,032,691  1,956,219      0 No                 .00 select /* test full */ * from kso.skew3
    1 5ry8vd8u0n0bf      0 2684249835          1         24.03    1,956,226  1,956,219      0 Yes              71.85 select /* test count */ count(col1) from
    1 6jtf4rnxu2dq7      0 2684249835          1         24.50    1,956,226  1,956,219      0 Yes              71.85 select /* test max */ max(col1) from kso
    1 8pfxs6mdf62mq      0 3723858078          1           .00           35          0      0 No                 .00 select /* test index */ avg(pk_col) from
    1 abwyftyy395gk      0 2684249835          1         13.96    1,956,226  1,956,219      0 Yes              99.99 select /* test avg filter */ avg(col1) f
    1 fxpkhxt4jwqqm      0 2684249835          1         39.08    1,956,226  1,956,219      0 Yes               9.35 select /* test count(rll) */ count(pk_co
 
8 rows selected.

So I did several queries. Some were off-loadable and some were not. Then I ran the script to see what how the statements had been handled. There are a couple of things to note here:

I used IO_CELL_OFFLOAD_ELIGIBLE_BYTES in V$SQL to indicate whether a statement can be offloaded or not.
(i.e. decode(IO_CELL_OFFLOAD_ELIGIBLE_BYTES,0,’No’,'Yes’) Offloadable) Note, if you run this script on a non-Exadata 11gR2 database, all the SQL statements should show up as Offloadable=NO.
All the statements using the aggregate functions (max,count,avg…) are marked as off-loadable even though the functions themselves are not off-loadable. I believe that projection accounts for reduced IO (limited to columns requested) and the marking as off-loadable. Note that the statement commented “test max all” shows almost no reduction in IO (it applies the max function to each column). I’m not yet sure why there was a slight reduction in I/O even with all columns listed. And by the way, V$SQLFN_METADATA has a list of all valid functions and whether they are off-loadable or not (none of the aggregate functions are).
Statements selecting all columns with no where clause are not off-loadable (no filter predicate or column projection).
Statements using index range scans are not off-loadable (the optimizer has to pick a plan with a full scan). Check out the statement commented “test index” for an example.

It is possible to turn offloading off via a parameter, CELL_OFFLOAD_PROCESSING, or via a hint. Here’s a bit of the Oracle Exadata Storage Server Software User’s Guide:

The CELL_OFFLOAD_PROCESSING initialization parameter enables SQL processing offload to Exadata Cell. When the value of the parameter is set to TRUE, predicate evaluation can be offloaded to cells. The default value of the parameter is TRUE. If the parameter is set to FALSE at the session or system level, then the database performs all the predicate evaluation with cells serving blocks. You can set CELL_OFFLOAD_ PROCESSING dynamically with the SQL ALTER SYSTEM or ALTER SESSION command. For example:
SQL> ALTER SESSION SET CELL_OFFLOAD_PROCESSING = TRUE;
The CELL_OFFLOAD_PROCESSING parameter can also be set with the OPT_PARAM optimizer hint to enable or disable predicate filtering for a specific SQL command. For example:
– to disable cell_offload_processing for a SQL command SELECT /*+ OPT_PARAM(’cell_offload_processing’ ‘false’) */ COUNT(*) FROM EMPLOYEES;
– to enable cell_offload_processing for a SQL command SELECT /*+ OPT_PARAM(’cell_offload_processing’ ‘true’) */ COUNT(*) FROM EMPLOYEES;

And a quick example:

^?View Code NONE

SYS@LABRAT1> select /* test avg filter hint */
  2  /*+ OPT_PARAM('cell_offload_processing' 'false') */
  3  avg(col1) from kso.skew3 where col1 between 2 and 10000;
 
 AVG(COL1)
----------
 
 
1 row selected.
 
SYS@LABRAT1> @fsx
Enter value for sql_text: % test avg filter %
Enter value for sql_id: 
Enter value for inst_id: 
 
 INST SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO    AVG_PIO AVG_PX OFFLOADABLE IO_SAVED_% SQL_TEXT
----- ------------- ------ ---------- ---------- ------------- ------------ ---------- ------ ----------- ---------- ----------------------------------------
    1 0nb4tjmjuq67t      0 2684249835          1        104.89    1,956,227  1,956,219      0 No                 .00 select /* test avg filter hint */ /*+ OP
    1 abwyftyy395gk      0 2684249835          1         13.96    1,956,226  1,956,219      0 Yes              99.99 select /* test avg filter */ avg(col1) f
 
2 rows selected.

So as you can see, the hint disabled offloading and resulted in the statement taking about 10X the elapsed time of the offloaded version.

Now for one more example to whet your appetite for things to come:

^?View Code NONE

SYS@LABRAT1> select count(*) from kso.skew3 where col1 is null;
 
  COUNT(*)
----------
        12
 
1 row selected.
 
Elapsed: 00:00:00.08
SYS@LABRAT1> @fsx
Enter value for sql_text: %col1 is null%
Enter value for sql_id: 
Enter value for inst_id: 
 
 INST SQL_ID         CHILD  PLAN_HASH      EXECS     AVG_ETIME      AVG_LIO    AVG_PIO AVG_PX OFFLOADABLE IO_SAVED_% SQL_TEXT
----- ------------- ------ ---------- ---------- ------------- ------------ ---------- ------ ----------- ---------- ----------------------------------------
    1 0u1q4b7puqz6g      0 2684249835          1           .08    1,956,226  1,956,219      0 Yes             100.00 select count(*) from kso.skew3 where col
 
1 row selected.
 
Elapsed: 00:00:00.09
SYS@LABRAT1> @dplan
Enter value for sql_id: 0u1q4b7puqz6g                               
Enter value for child_no: 
 
PLAN_TABLE_OUTPUT
------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  0u1q4b7puqz6g, child number 0
-------------------------------------
select count(*) from kso.skew3 where col1 is null
 
Plan hash value: 2684249835
 
------------------------------------------------------------------------------------
| Id  | Operation                  | Name  | Rows  | Bytes | Cost (%CPU)| Time     |
------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT           |       |       |       |   533K(100)|          |
|   1 |  SORT AGGREGATE            |       |     1 |     5 |            |          |
|*  2 |   TABLE ACCESS STORAGE FULL| SKEW3 |    12 |    60 |   533K  (1)| 01:46:38 |
------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   2 - storage("COL1" IS NULL)
       filter("COL1" IS NULL)
 
 
20 rows selected.
 
Elapsed: 00:00:00.02

Yes that’s a full scan on a 384M row table that occupies about 2M blocks. And it took 8/100’s of a second. That’s 80 milliseconds - roughly the amount of time necessary to do 10-15 single block disk i/o’s.

Wow! Can you say storage index? More on that in the next installment!

Oracle Exadata Delivery Day

osborne — Wed, 26 May 2010 02:25:02 +0000

Well our new Exadata showed up this week. We had a pretty nice lab environment already. A bunch of Dell’s, some IBM’s, several Sun’s. We have a couple of EMC Sans as well (we actually threw away a whole EMC rack to make room for the Exadata). And of course we have every version of Oracle from 8i to 11gR2. It’s a good learning environment. It also let’s us try things when clients have a specific set of versions that we want to mimic. So now we have an Exadata V2 as well. We’ve had the delivery date on the calendar for several weeks. For some reason it reminded of the Weird Al Yankovik song Weasel Stomping Day.

It’s probably a sad reflection on how geeky we are that everyone is running around all excited like it’s Christmas or something.

Here’s a few pictures:

It’s a really fast machine by the way. In fact, we had trouble keeping up with it from the moment we got it off the truck.

Really fast, and slippery. Well in a couple of days we can actually turn it on (we’re supposed to let it acclimate to our environment). On Friday afternoon we’re going to have a happy hour to celebrate our newest edition. Wade calls it a sip and see. We’ll probably take a few pictures of ourselves with the little bundle of joy and sing a chorus of a festive Weird Al song, or maybe two. Come on by if you’re in the neighborhood!