No I’m sorry I don’t have a script like that. Usually I’m digging into SQL underlying procedures, so most of my stuff is targeted at individual SQL statements. Since statistics for procedures are recorded in a similar manner to SQL statements you can certainly pull information that is captured in AWR to see how they perform over time. awr_plan_change.sql is a simple script that is used in this post to track some performance stats by SQL_ID (which each procedure is assigned). So maybe that could provide a starting point for you.

Kerry

Mathijs

Wow – I think you get the prize for the longest comment.

Let me answer your specific questions first then I’ll make a few comments:

1. Cardinality Feedback is a new feature and it is still a bit immature. The is a parameter to turn it off if is causing you lot’s of issues. Manually created SQL Profiles based on hints in OTHER_XML column of v$sql_plan or dba_hist_sql_plan can definitely be used force (or come close to) forcing a desired plan. Those hints are not always complete enough to force one and only one plan but most times they are.

2. Yes – SQL Profiles can restore an old plan even from an older version in most cases.

3. Yes – SQL Profile do their magic with hints so if you have the plan you want (based on the optimizer parameters in place at the time it was parsed) those hints should generally give you the plan the plan that was being used when the profile was created, even if you change optimizer parameters. In fact that’s a technique that I have used many times in the past. (i.e. manipulate a test version of a statement until I get the plan I want using whatever means I can – including embedded hints and changing optimizer and even hidden parameters. Then create a SQL Profile on the new statement, test to make sure it still gets the desired plan after creating the profile, and then attach the SQL Profile to the original statement).

So I think the answer to all your specific questions is yes.

Now a couple of comments:

1. It doesn’t always work for the same reason you quoted me on, so you need to test after create on to ensure that it has been picked up, and that it hasn’t caused the plan to change to something you weren’t expecting (this would be a bug but it does happen occassionally).

2. AWR retention can be increased to however long a periord you are willing to store. See this post: Tracking Parameter Changes

Kerry

I’ve read a lot of the blogs / oracle doc regarding the changing oracle features (outlines to awr to bind variable peeking to sql_profiles to cardinality feedback to sql_baselines / sql plan management).

A few months ago I came up with an idea / desire which I now realized may be doable without much engineering on my part.

My thinking was that it would be great if oracle stored off the historical performance on queries (which I now realized it does for maybe 10 days in dba_hist_sqlstat, dba_hist_sqlplan and dba_hist_sqltext ) so that after an upgrade (or optimizer parameter change) we could deal with the sql that is not performing well and see how it did in the past.

Then maybe we could use an oracle technology (like sql_profiles) to get the optimizer to change gears and use the plan it was using before the change.

I have experience activating sql_profiles that the sql tuning advisor built. I’ve seen your post about creating a sql_profile from awr and / or the shared pool views.

Do you think this is doable or am I wasting my time because you really can’t point oracle back to a plan but only provide hints to try to get it to use a plan that you want.

I would want to be able to get oracle to use an old plan under these conditions.

1. cardinality feedback issues: query runs bad on run1 and good on run2 and beyond, but not across bounces. Want it to run good on run1 also. I had one that ran 2 days then 8 minutes. After a bounce the first run was 2 days again. This was no simple query. With a bunch of union statements it ended up to be 197 plan lines. Note also, the vendor written query has literals for dates that change all the time.

We’ve seen them vice verse where they run ok on run1 then bad on further runs.

2. Upgrade issues from let’s say 11.1.0.7 to 11.2.0.3, or from 10.2.0.4 to 11.2.0.3.
It would be nice to get the system to run as it did before. We had a vendor supplied all_objects query that went from .05 seconds to 4.75 seconds. Oracle messed it up (all_objects queries but we had to revert back to 10.2.0.4 because we could not get performance. We had code that was running the query 27 times to kick off a report where customers were standing in line waiting for a receipt. Could we get a sql_profile built and accepted from a plan from a different version?

3. Parameter change on Version 11.2 of oracle. Initially using optimizer_features_enable=10.2.0.4 after cutover issues discovered. Now wanting to cutover to optimizer_features_enable=11.2.0.3. Would be nice to know we could fix any query that ran ok before but now became slow. Could we get a sql_profile built and accepted from a plan from a different optimizer parameter.

We deal with a lot of vendor applications.

My understanding is most techniques deal with providing hints to push the optimizer to a plan. I am not sure this will work in these conditions. It would be nice if you could direct the optimizer to a good plan from the past. Not sure why Oracle does not natively allow that option.

I have read about sql_baselines / sql plan management but am concerned that the lack “sharing” limits its abiltiy to help ie., when there are literals it is unusable. And I believe the good plan has to come from your currrent environment.

After writing most of the above I just reread some of your postings and find in2009 you said:

“I have to admit that I have changed my mind about Outlines really having the ability to completely lock a plan by the way. I believe that trying to limit the optimizer to picking a single plan with hints is a difficult task at best. That’s why I’m hopeful that Baselines are taking a totally different approach (maybe you can help shed some light on that question). That said, Outlines have done a very good job for many years now. I’m just thinking Baselines will be.”

Unless you now have new knowledge in the area, you likely will say that in dealing with more complicated statements and environments ( like my list of desires above) you likely won’t be able to get the optimizer to make a plan that matches one from the past.

Thanks for your time.

1. I would check to see if ASH data shows any unusual wait events associated with that statement for the snap shots of interest. If that doesn’t show anything I’d look at O/S logs to see if the system was having any issues that might account for the long run time (overloaded i/o or cpu or memory shortage). If on 11g you can look at SQL Monitor data to see exactly what happened for an individual execution of a long running statement (this is the best way if you are on 11g and can have a look before the data is SQL Monitor dataa is flushed).

2. I don’t know of a way to do this for sure. There may be some set of hints that would do it and if so a SQL Profile or Baseline should be able to do the job.

1. assuming the filter predicates were also same then how can we figure out where exactly was the SQL spending time more time for that particular execution… (enq wait for another session, storage, cpu, network, any new v$session_wait wait_event)… dba_hist_sqlstat has these absolute and delta columns for various attributes …i just cant seem to interpret them…and thats why i am here…..i guess i want to say “for that execution the sql spent 10 secs on cpu but now it is spending 120….rest all wait times are same….hence its the cpu….”

for example i want to figure out the 108.289 secs (last line) in the following

199642 1 22-FEB-12 11.50.44.468 AM gxn7ry17dh5sq 5 14.269 29,178.4
199643 1 22-FEB-12 12.00.59.733 PM gxn7ry17dh5sq 2 16.362 22,507.5
199644 1 22-FEB-12 12.10.09.047 PM gxn7ry17dh5sq 3 16.753 19,809.7
199645 1 22-FEB-12 12.20.18.393 PM gxn7ry17dh5sq 3 18.509 56,736.7
199646 1 22-FEB-12 12.30.30.690 PM gxn7ry17dh5sq 5 108.289 33,629.8

2. if filter predicates are different how can we force an older explain plan (same plan_hash_value but diff predicates) onto the sql_id

I use a couple of scripts (which you can modify to look at whichever stats you want).

awr_plan_change.sql – lists execution stats for a SQL_ID for each snapshot in historical order
awr_plan_stats.sql – aggregates the data by plan (this is probably not useful since you have the same plan)

By the way, the plan_hash_value is calculated based on a subset of the plan data (basically the operation and object). It ignores predicate information, so although the plan_hash_value is the same the actual plan may be slightly different (but may perform very differently). You may be able to detect this by looking at the plan_hash_2 value in the OTHER_XML field in DBA_HIST_SQL_PLAN. (I have not written a script to identify this situation yet – maybe soon)

Kerry

Thanks a lot for sharing the insight on OPT_ESTIMATE hint and yes the coe_xfr by oracle script is definitely a flexible script since it lets you edit the hints or sql_text in the generated script that you can run anywhere.I see there is so much talk on the pinning a good explain plan and it works most of the time, but im having an issue for which i think it would help to take your advice or any good scripts that you have.

So i have this SQL_ID which has been running fine and recently customer started complaining about the slowness. so i went in to check the change of explain plan and turns out the explain plan hasn’t changed, but the corresponding avg_etime has spiked badly (avg_lio also). now obviously something has changed (optimizer_env, storage ) ie causing the same explain plan to take longer.
now i know i should be looking at dba_hist_sqlstat to find out which part (i/o, cpu etc) is taking longer than before , but i cant seem to write a sql which gives me a clear picture.
do you have anything at hand that you use when you have such scenario.

thanks
tarun