OCI-21500: internal error code, arguments: [kpcerecv: auth], [9], [], [], [], [], [], []

Those days, for one of my customers, I was having a bizarre oracle connection error, where the Oracle Client connecting using sqlplus was calling a package procedure and this error was being returned:

[user@machine ~]$ sqlplus /@dbname

SQL*Plus: Release 18.0.0.0.0 - Production on Thu Sep 10 00:07:43 2020
Version 18.3.0.0.0

Copyright (c) 1982, 2018, Oracle. All rights reserved.


Connected to:
Oracle Database 11g EE Extreme Perf Release 11.2.0.4.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, Oracle Label Security,
OLAP, Data Mining and Real Application Testing options

SQL> exec dwxp.p_xxxx;
Errors in file :
OCI-21500: internal error code, arguments: [kpcerecv: auth], [9], [], [], [], [], [], []


----- Call Stack Trace -----
calling call entry argument values in hex
location type point (? means dubious value)
-------------------- -------- -------------------- ----------------------------
skgudmp()+171 call kgdsdst() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 000000000 ?
kgerinv_internal()+ call skgudmp() 000000000 ? 000000000 ?
115 000000000 ? 000000000 ?
000000000 ? 000000000 ?
kgerinv()+40 call kgerinv_internal() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 000000000 ?
kgeasnmierr()+150 call kgerinv() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 000000000 ?
kpcerecv()+3828 call kgeasnmierr() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 000000000 ?
start_thread()+209 call kpcerecv() 000000000 ? 000000000 ?
000000000 ? 000000000 ?
000000000 ? 000000000 ?

Call stack signature: 0x8f11d4ecfccf723a



call stack performance statistics:
total : 0.010000 sec
setup : 0.000000 sec
stack unwind : 0.000000 sec
symbol translation : 0.010000 sec
printing the call stack: 0.000000 sec
printing frame data : 0.000000 sec
printing argument data : 0.000000 sec


----- End of Call Stack Trace -----

Errors in file :
OCI-21500: internal error code, arguments: [kgepop: no error frame to pop to], [], [], [], [], [], [], []
OCI-21500: internal error code, arguments: [kpcerecv: auth], [9], [], [], [], [], [], []

As I was using a Oracle 18c client to connect on Oracle 11g, my first try was to use the 11g client so I would remove completely a possible compatible matrix issue from the table. However, the issue persisted.

The biggest problem is that this issue was so intermittent that was so hard do reproduce when we were together with Oracle Support in a call. And the worst thing is that even though this error was returned to the client, the package execution continued on the server side.

Finally after +1 month of Oracle Support investigation, they came with a solution:

I came across an internal bug where the client was getting similar errors when AQ_HA_NOTIFICATION is turned on on services.
In your scenario, this database hasAQ_HA_NOTIFICATION turned on and the rest of the databases- where the issue does not exhibit, does not have it turned on.

Can you please turn off AQ_HA_NOTIFICATION for this service and monitor.
You can use dbms_service.modify_service to turn off AQ_HA_NOTIFICATION .

So the issue was finally fixed when we disable the AQ_HA_NOTIFICATION on the service.

SQL> execute dbms_service.modify_service( SERVICE_NAME => 'myservice', AQ_HA_NOTIFICATIONS => FALSE);

This one is not easy to spot on MOS or over the internet. Hope this helps if you face the same.

+1