Comments on: Hidden SQL – why can’t I find my SQL Text?

By: Paul

Paul — Thu, 02 Jun 2011 03:40:15 +0000

Thanks for this great post. Those darn “table_#~_@%_!?#” sql statements have really been annoying – they just sit there in v$open_cursor and tell you nothing.. prev_sql_id indeed!

By: Mark

Mark — Wed, 25 May 2011 17:08:48 +0000

Hi Kerry,
I had a similar case and followed your article. But, if the “unknown” sql took place a while ago:
- the session may be gone, therefore no info on prev_sql_id, or open cursor
- the session may still be connected, so you may still find the open cursor for the sql_id, but even if it’s there, the current sql id and prev sql id may already be for the sql executed long after the one we are looking for
Unless I’m missing somehting, in the above conditions, it is more of a guessing game. The most helpfull for me was that I was able to identify an open cursor with “table_…” and based on that the actual table containing the LOB column. That combined with some knowledge of the application gave me enough info to talk to the developers.

By: Kellyn Pedersen

Kellyn Pedersen — Thu, 02 Dec 2010 02:09:45 +0000

Great article, Kerry! I’ve worked through this same problem, but you worked through it efficiently without all the dead-ends I tripped down. Wish I had found your article before I’d taken this on by myself, would have saved myself a ton of time! :)
Kellyn

By: osborne

osborne — Sat, 20 Mar 2010 05:13:42 +0000

Well, the app servers were several states way from the database servers. And the dedicated network connection between them was pretty slow. The statement was executed a lot. Several options for dealing with it presented themselves.

1. Get a bigger pipe.
2. Reducing frequency – Because the query was executed multiple times per transaction getting only one row at a time, it should have been possible to collapse it into a single multi-row get.
3. Move the app servers – to the datacenter where the database servers resided.
4. Move the data – to the datacenter where the app servers resided.
5. Alter the code – It turned out that the amount of data in the vast majority of the clob’s was only a couple of hundred bytes, but for some reason, SQL*Net padded the field to appx. 80K. Putting a to_char function around the column in the select statement caused the filed to not be padded. I think this may be related to the java variable type they were selecting into. At any rate, another possible solution would have been to alter the code to add the to_char function.
6. Alter the design to get rid of the XML/Clob – since the clob rarely had more than a couple of hundred bytes, there were many options. All the way from eliminating the XML altogether in favor of columns and rows, to just changing the clob to a varchar2.

In the end, the decision was number 4, moving the data closer to the app servers. This was done by replicating the data via Streams to another instance in the datacenter where the app servers are located. Not my favorite solution, but it required no code changes. We’re still discussion changing the code via to_char and redesign, but the wheels of justice turn slowly.

Kerry

By: abcd_1234

abcd_1234 — Sat, 20 Mar 2010 03:09:22 +0000

It’s an awesome article …. I am in the same situation in trying to figure out why CPU is getting killed ….and troubleshooted based on your article. I was able to figure out the offending sql.

I am quite interested to know how you tuned that query after finding the bottleneck … Can you share that information ?…

By: LOB vs. SQL*Trace « Dion Cho – Oracle Performance Storyteller

LOB vs. SQL*Trace « Dion Cho – Oracle Performance Storyteller — Fri, 19 Jun 2009 02:22:34 +0000

[...] followings are related posts which describe the same phenomenon in the different ways. http://kerryosborne.oracle-guy.com/2009/04/hidden-sql-why-cant-i-find-my-sql-text/ [...]

By: Log Buffer #143: a Carnival of the Vanities for DBAs | Pythian Group Blog

Log Buffer #143: a Carnival of the Vanities for DBAs | Pythian Group Blog — Fri, 24 Apr 2009 16:45:03 +0000

[...] Osborne wonders about Hidden SQL – why can’t I find my SQL Text?, beginning, “A very frequently executed statement on a database was generating some waits on [...]

By: osborne

osborne — Mon, 20 Apr 2009 13:47:19 +0000

Dion,

Thanks for the pointer on the x$kglob. I’m with you, I don’t want to look at any dumps unless I have too. I hadn’t tried to reproduce this issue until this morning after your comment. But I didn’t have too many problems doing so. The problems for me were all related to doing the test in SQL*Plus. The first was that SQL*Plus spits out DBMS_OUTPUT calls unless you set serveroutput off (so this wiped out the prev_sql_id field in v$session). The second issue is that when the statement completes, the sql_id field in v$session is wiped out. The entries still exist in v$open_cursor and the prev_sql_id field still has the original sql_id of the statement which generated the access to the lob. So in order to see the behavior we had in production, my SQL*Plus test required that I catch the session in the act of executing the statement. I haven’t had time to look into that yet. I have captured the data and will add it to the original post as I’m not sure how the formatting would work in the comments section.

Thanks for your comments.

Kerry

By: LOB, sql_id and LCO « Dion Cho - Oracle Performance Storyteller

LOB, sql_id and LCO « Dion Cho - Oracle Performance Storyteller — Mon, 20 Apr 2009 10:40:02 +0000

[...] leave a comment » Kerry Osborne wrote an interesting case on LOB here. [...]

By: Dion Cho

Dion Cho — Mon, 20 Apr 2009 09:08:20 +0000

Kerry.

It’s very interesting case even though I could not reproduce your case(SQL*Net more data from client thing).

Anyway, query on x$kglob would have shown some data like:

select * from sys.x$kglob where kglobt03 = ”;

And library cache dump would show the depedent table for the LOB LCO.

But still, hard to decode. :)