Kerry Osborne's Oracle Blog

Last week an interesting issue popped up on a mission critical production app (MCPA). A statement that was run as part of a nightly batch process ran long. In fact, the statement never finished and the job had to be killed and restarted. This particular system is prone to plan stability issues due to various factors outside the scope of this post, so the first thing that the guys checked was if there had been a plan change. Surprisingly the plan_hash_value was the same as it had been for the past several months. The statement was very simple and a quick look at the xplan output showed that the plan was indeed the same with one exception. The predicate section was slightly different.

As a quick diversion, you probably already know that the plan_hash_value is calculated based on partial information about the plan. Arguably it’s the most important parts, but there are some important parts of the plan that are not included (namely the stuff that shows up in the predicate section of the plan). Randolf Geist explained which parts of the plan are used in calculating the plan_hash_value well in a post on How PLAN_HASH_VALUES Are Calculated several years ago. His summary was this:

So in summary the following conclusions can be made:

– The same PLAN_HASH_VALUE is merely an indicator that the same operations on objects of the same name are performed in the same order.

– It tells nothing about the similarity of the expected runtime performance of the execution plan, due to various reasons as demonstrated. The most significant information that is not covered by the PLAN_HASH_VALUE are the filter and access predicates, but there are other attributes, too, that are not part of the hash value calculation.

The production issue got me thinking about several things:

Could I come up with a simple test case to show a severe performance degradation between executions of a statement with the same plan_hash_value because of a change in the predicates? (the answer is it’s pretty easy actually)

What forensics could be used to determine when this situation has occurred?

How do you fix the problem?

Continue reading ‘SQL Gone Bad – But Plan Not Changed? – Part 1’ »

Thanks to everyone that submitted abstracts for our upcoming E4 conference. Unfortunately, there were more quality submissions than we had room for. Maybe next year we should expand the event to 3 days. :) But in the meantime, we have assembled what I believe is an excellent line up of speakers. I’ll just mention a few highlights here:

Tom Kyte will be doing the keynote. Enough said!

Maria Colgan and Roger MacNicol will be doing a 3 hour combined session on smart scans. Maria will attack the topic from the top down (optimizer) point of view (since she is the product manager for the optimizer) and Roger will be attacking it from the bottom up (since he is the lead developer for the smart scan code). This should be an awesome session and Tanel Poder has already said he was going to line up the night before.

Ferhat Sgonul will be talking about Turkcell’s usage of Exadata. Turkcell is one of the earliest adopters of Exadata and has had great success with it over the last several years, so this should be a very interesting case study.

Karl Arao and Tyler Muth will do a joint presentation on visualization techniques for performance data from Exadata environments. The plan is for them to compare and contrast their approaches using the same data set. Tyler usually uses R and Karl likes Tableu – may the best violin chart win.

Tyler Muth will also be doing a deep dive presentation on bloom filters and how they can be offloaded with smart scans. This is a topic about which there is little information, so it should be quite interesting.

Frits Hoogland will be doing a deep dive on how Oracle does multi-block i/o. This is of special interest with regard to Exadata because the direct path mechanism for doing multi-block i/o is a requirement for enabling smarts scans. So understanding how it works is one of the keys to getting the most out the platform.

Sue Lee (product manager for resource manager) will be doing a session on how to deal with mixed workloads. I’m really interested in this session as IORM and DBRM are critical for managing Exadata, particularly when it is used as a consolidation platform.

There are many other well known speakers including Martin Bach, Andy Colvin, Gwen Shapira, Mark Rittman, Tim Fox and Tanel Poder.

Here’s a link to see the complete line up of E4 speakers.

While we’re on the subject, I should mention that there will be several talks on hadoop related topics and the increasingly expanding role it is playing in our industry. The idea of pushing the work to the storage is not unique to Exadata. It is also the main driver behind hadoop. So I’m extremely pleased to announce that Doug Cutting will be speaking at E4 as well.

So that’s all for the marketing related stuff on E4. I hope you can join us in Dallas.

Just a quick note to remind you that the call for papers for E4 is closing in a few days (on April 30).  So if you have anything you think is interesting related to Exadata that you’d like to share we’d love to hear from you. By the way, you don’t have to be a world renowned speaker on Oracle technology to get accepted. You just need to have an interesting story to tell about your experience. Implementations, migrations and consolidation stories are all worthy of consideration. Any interaction between Exadata and Big Data platforms will also be of great interest to the attendees. Of course the more details you can provide the better.

Here’s a link to the submission page:

Submit an Abstract for Enkitec’s Extreme Exadata Expo

Feel free to shoot me an email if you have ideas for a talk that you want to discuss before formally submitting an abstract.

I got an email a few days ago . . .

Subject: Exadata / your blogs / you aren’t saying to put HINTS on everything, Are you…

. . . A peer of mine found a couple of your blogs listed below which discuss Profiles and Hints. The position he is inferring upon you is that you are stating unequivocally that we should create lots of indexes, add hints, create profiles and outlines on our Exadata machine. Having read your book and achieve actual results in line with that book, I know that pretty much the opposite is true. . . .

Here’s what I said in my reply (edited a bit to clean up typos and to be a bit more precise):

I do not advocate widespread usage of hints or hint based mechanisms (Outlines, Patches, Profiles). However, because I have written quite a bit about using these techniques, I fear I may have inadvertently left people with the impression that I think hints should be used everywhere. I do not. I view them as expedient ways to change plans or lock in specific plans that I want, but in most cases I use them as bandaids to temporarily cover up other problems. I haven’t really written a blog post specifically on my view of the proper usage of these techniques, so maybe I should do that. When I do presentations on hint based mechanisms to control execution plans, I always have a slide with a picture of a vial of morphine. I use that metaphor to caution people not to over use them. There is a presentation on my blog in the whitepapers/presentations section called “Controlling Execution Plans” (or something to that effect) that you can download which has that slide in it. Also I wrote a chapter on controlling execution plans in a book called Pro Oracle SQL that probably has some prose on when and why to use them (and not use them). But in general – I think they should only be used to give temporary relief until underlying problems can be fixed or in cases where the optimizer just can’t deal with a specific situation (like correlated columns for example) or in cases where you’re using a packaged app that has badly written code that you can’t change. The newest version of the optimizer (which should be released soon) actually stores plans and not just hints with Baselines, so I will leave out the case where you want to manage plans for stability sake.

Since you mentioned Exadata I’ll add that the issue of mixed workload DB’s on Exadata and when the optimizer uses indexes vs when it doesn’t is not straight forward. In general, the current version of the optimizer doesn’t know it’s running on Exadata so it tends to cost FTS’s too high, meaning it often picks indexes when a full scan would be faster. There are some things that can be done to push the optimizer towards full scans. Think of anything that pushes the optimizer in that direction – large db_file_multiblock_read_count for example. Also, System Stats has recently been updated to have an “Exadata” mode which basically just updates the MBRC setting to 128 which does the same sort of thing. If that’s not enough to get most of the statements behaving the way you want then exceptional measures can be taken such as making indexes invisible and using alter session for specific processing to enable the optimizer to see invisible indexes, or doing away with some indexes altogether. Hints (or hint based mechanisms) can be used as well. But my goal would be to get the optimizer to pick the right plan on it’s own in the vast majority of cases before resorting to the more extreme measures.

Hope this helps.

Here’s a copy of the slide I referenced in the email and have been using recently:

Continue reading ‘Bad Advice?’ »

I was doing a little research for an upcoming presentation on Oracle 12c Adaptive Optimization and I came across a series of posts on Dynamic Sampling by Randolf Geist (one of my favorite Oracle smart guys). I could not find a complete list of the series of posts either through Google or the search function on the site where they were posted. I guessed at a couple of the url’s and got lucky. Thought it might be useful at some point in the future so I wrote them down on my blog (which is where I often keep private notes). But decided it might be useful to others so decided to go ahead and publish it.

Dynamic Sampling – Intro – Part 1
Dynamic Sampling – Intro – Part 2
Dynamic Sampling – Controlling the Activity – Part 1
Dynamic Sampling – Controlling the Activity – Part 2
Dynamic Sampling – Real Life Data – Part 1
Dynamic Sampling – Real Life Data – Part 2

Here are a couple of other posts Randolf has done on the topic of Dynamic Sampling.

Volatile Data, Dynamic Sampling and Shared Cursors
Dynamic Sampling on Multiple Partitions – Bugs

Note these posts are all referring to 11gR2 or earlier releases.