ORA-27301 OS failure message: No buffer space available

If you get this error

ORA-27300: OS system dependent operation:sendmsg failed with status: 105
ORA-27301: OS failure message: No buffer space available
ORA-27302: failure occurred at: sskgxpsnd2

The solution is to:

  1. Lower MTU to 16436. To do this, add following to /etc/sysconfig/network-scripts/ifcfg-lo or replace the value if MTU is already set in this configuration file.
    MTU=16436
  2. Save the file, exit and restart the network service by running:
    systemctl restart network.service
  3. Calculate the value for vm.min_free_kbytes to be 0.4 % of the total physical memory and multiply by the number of NUMA nodes.
    total=$(awk ‘/MemTotal:(.)/{print $2 }’ /proc/meminfo) numa_count=$(lscpu | awk ‘/^NUMA node(s)(.)/{print $3 }’)
    min_free_kb=$(expr $total / 250 * 1) # this is the
  4. Increase the value of vm.min_free_kbytes accordingly. This can be done by adding following a new sysctl configuration.
    echo “vm.min_free_kbytes=$min_free_kb” > /etc/sysctl.d/99-vm_min_free_kbytes.conf

This will take the value of $min_free_kb from the previous step!

  1. load the new values

sysctl –system

  1. Verify the new value has been applied correctly.
    sysctl vm.min_free_kbytes

ORA-00600 [KSLGES_3]/ORA-27300

When using RHEL 7.2 you may encounter some database crash with the errors below:

ORA-27157: OS post/wait facility removed
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed
ORA-27302: failure occurred at: sskgpwwait1
ORA-00600: internal error code, arguments: [KSLGES_3], [], [], [], [], [], [], [], [], [], [], []
ORA-27300: OS system dependent operation:semop failed with status: 43
ORA-27301: OS failure message: Identifier removed

Redhat 7.2, systemd-logind service introduced a new feature to remove all IPC objects when a user fully logs out.
The feature is controled by the option RemoveIPC in the /etc/systemd/logind.conf configuration file, see man logind.conf(5) for details.

The default value for RemoveIPC in RHEL7.2 is yes.

As a result, the OS may removes shared memory segments and semaphores for those users.
As Oracle ASM and Databases use shared memory segments for SGA, removing shared memory segments will crash the Oracle ASM and database instances.

A new systemd feature was introduced in Red Hat Enterprise Linux 7.2: cleanup of all allocated inter-process communication (IPC) resources with the last session a user finishes. A session can be an administrative cron job or an interactive session.

This behavior can cause daemons running under the same user, and using the same resources, to terminate unexpectedly.

To work around this problem, edit the file /etc/systemd/logind.conf and add the following line:
RemoveIPC=no

Then, execute the following command, so that the change is put into effect:
systemctl restart systemd-logind.service
After performing these steps, daemons no longer crash in the described situation.

“can’t open the fdp output file linkorderfile” Error

“can’t open the fdp output file linkorderfile” warning is seen during oracle binary link on HP Unix.

This problem can happen if you invoke “make -f $ORACLE_HOME/rdbms/lib/ins_rdbms_mk ioracle” from directory different from $ORACLE_HOME/rdbms/lib
The problem can happen on scripts tha call “ins_rdbms.mk ioracle”, for example, the “chopt” script can hit this problem.

Workaround
“cd $ORACLE_HOME/rdbms/lib” and rerun the command.

Le licensing Oracle

je peux remarquer assez souvent que pas mal de personnes ne sont pas au courant du fonctionnement du licensing Oracle qui est, il faut l’avouer assez complexe en fonction de la licence que l’on possède.

Les règles sont assez méconnu lorsque l’on parle de virtualisation, surtout avec VMWare. J’aborderai ce point dans un autre post.

Lire la suite!

Using variables in RMAN

When using RMAN catalog it is preferred to use backup script stored in catalog. It is best if scripts can be reused for different retention.

You can use variables in RMAN script this way:

CREATE global script global_backup_full_database 
comment 'backup full of database. Parameter sysdate+n,tag db, tag archivelog'
{
allocate channel c1 TYPE SBT_TAPE PARMS 'BLKSIZE=1048576,SBT_LIBRARY=/usr/lib/ddbda/libddboostora.so, ENV=(CONFIG_FILE=/opt/ddbda/config/oracle_ddbda.cfg)' FORMAT '%d_%U';
backup AS backupset DATABASE filesperset 1 keep until TIME '&1' tag &2;
backup AS backupset archivelog ALL DELETE INPUT filesperset=1 keep until TIME '&1' tag &3;
}
Lire la suite!

Calendar expressions for dbms_scheduler

Here are some examples of Calendar type instructions to schedule job executions for the Oracle dbms_scheduler scheduler:

Replace line repeat_interval with the desired repeat interval.

BEGIN
sys.dbms_scheduler.create_job( job_name => ‘”SYSTEM”.”TEST”‘,
job_type => ‘PLSQL_BLOCK’,
job_action => ‘begin
— Insert PL/SQL code here
end;’
,
repeat_interval => ‘FREQ=YEARLY; BYDATE=0715;’,
start_date => systimestamp at time zone ‘Europe/Paris’,
job_class => ‘”DEFAULT_JOB_CLASS”‘,
auto_drop => FALSE,
enabled => TRUE);
END;
/

Every day, monday to friday at 8PM
FREQ=DAILY; BYDAY=MON,TUE,WED,THU,FRI; BYHOUR=20; BYMINUTE=0; BYSECOND=0;

Every hour
FREQ=HOURLY;INTERVAL=1;

Every 5 minutes
FREQ=MINUTELY;INTERVAL=5;

Every monday at 10AM:
FREQ=DAILY; BYDAY=MON; BYHOUR=10; BYMINUTE=0; BYSECOND=0;
FREQ=WEEKLY; BYDAY=MON; BYHOUR=10; BYMINUTE=0; BYSECOND=0;
FREQ=YEARLY; BYDAY=MON; BYHOUR=10; BYMINUTE=0; BYSECOND=0;

One Monday out of two:
FREQ=WEEKLY; INTERVAL=2; BYDAY=MON;

Every monday on week 3, 6 et 9 every year:
FREQ=YEARLY; BYWEEKNO=3,6,9; BYDAY=MON;

Last day of the month:
FREQ=MONTHLY; BYMONTHDAY=-1;

Beofre last day of every month:
FREQ=MONTHLY; BYMONTHDAY=-2;

15 July:
FREQ=YEARLY; BYMONTH=JUL; BYMONTHDAY=15;
FREQ=YEARLY; BYDATE=0715;

Every  1,2,3,4 andet 5 february:
FREQ=YEARLY; BYDATE=0201,0202,0203,0204,0205
FREQ=YEARLY; BYDATE=0201+SPAN:5D;

Every 10 days:
FREQ=DAILY; INTERVAL=10;

Very day at 16h15, 17h15 and 18h15:
FREQ=DAILY; BYHOUR=16,17,18; BYMINUTE=15; BYSECOND=0;

Every 15, one month out of two:
FREQ=MONTHLY; INTERVAL=2; BYMONTHDAY=15;

Every 17 of each month:
FREQ=MONTHLY; BYMONTHDAY=17;

Second Thursday of each month:
FREQ=MONTHLY; BYDAY=2THU;

Last Thursday of the month:
FREQ=YEARLY; BYDAY=-1THU;

Every 40 hours:
FREQ=HOURLY; INTERVAL=40;

Last day of the month one month out of two:
FREQ=MONTHLY; INTERVAL=2; BYMONTHDAY=-1;

Every hours the two first day of the month:
FREQ=HOURLY; BYMONTHDAY=1,2;

Run the 60th and 120th day of the year:
FREQ=YEARLY; BYYEARDAY=60,120;

Monday to Friday the last day of work of the month:
FREQ=MONTHLY; BYDAY=MON,TUE,WED,THU,FRI; BYSETPOS=-1

DRA: Data Recovery Advisor


The data recovery advisor makes it possible to detect corruption or loss of data, determines the actions to be carried out and suggests the actions to be taken to repair the database.
This can saves you lots of hours if you are not sure what you can do.

LIST FAILURE
The LIST FAILURE command allows you to display problems that are in OPEN status with different levels of criticality.
 
RMAN> LIST FAILURE;

List of Database Failures
=========================

Failure ID Priority Status    Time Detected Summary
———- ——– ——— ————- ——-
48         HIGH     OPEN      22-FEB-12     One or more non-system datafiles are missing

ADVISE FAILURE

ADVISE FAILURE as its name indicates, allows you to give advice for repairing the problems listed by the LIST FAILURE command. If possible it will create a script to fix the problem.
If another failure appears after the list failure, this will not be included in the advice given by the ADVISE FAILURE command. You must redo a FAILURE LIST for this to be taken into account.


RMAN> advise failure;


List of Database Failures
=========================


Failure ID Priority Status    Time Detected Summary
———- ——– ——— ————- ——-
48         HIGH     OPEN      22-FEB-12     One or more non-system datafiles are missing


analyzing automatic repair options; this may take some time
using channel ORA_DISK_1
analyzing automatic repair options complete


Mandatory Manual Actions
========================
no manual actions available


Optional Manual Actions
=======================
1. If file /oracle/oradata/orcl/datafile/o1_mf_users_7n6xprq9_.dbf was unintentionally renamed or moved, restore it


Automated Repair Options
========================
Option Repair Description
—— ——————
1 Restore and recover datafile 4
Strategy: The repair includes complete media recovery with no data loss
Repair script: /oracle/diag/rdbms/orcl/ORCL/hm/reco_1941492637.hm

REPAIR FAILURE

REPAIR FAILURE runs the script generated by the ADVISE FAILURE command. The preview option allows you to see the contents of the file without applying it.
 

RMAN> REPAIR FAILURE PREVIEW;


Strategy: The repair includes complete media recovery with no data loss
Repair script: /oracle/diag/rdbms/orcl/ORCL/hm/reco_1941492637.hm


contents of repair script:
# restore and recover datafile
sql ‘alter database datafile 4 offline’;
restore datafile 4;
recover datafile 4;
sql ‘alter database datafile 4 online’;


By default, the REPAIR FAILURE command requests confirmation to apply the corrections. This can be overridden with the NOPROMPT option.


RMAN> REPAIR FAILURE NOPROMPT;


Strategy: The repair includes complete media recovery with no data loss
Repair script: /oracle/diag/rdbms/orcl/orcl/hm/reco_1941492637.hm


contents of repair script:
# restore and recover datafile
sql ‘alter database datafile 4 offline’;
restore datafile 4;
recover datafile 4;
sql ‘alter database datafile 4 online’;
executing repair script


sql statement: alter database datafile 4 offline


Starting restore at 22-FEB-12
using channel ORA_DISK_1


channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00004 to /oracle/oradata/ORCL/datafile/o1_mf_users_7n6xprq9_.dbf
channel ORA_DISK_1: reading from backup piece /oracle/flash_recovery_area/backups/ORCL/ORCL_db_gqn3uqh2.rmanbck
channel ORA_DISK_1: piece handle=/oracle/flash_recovery_area/backups/ORCL/ORCL_db_gqn3uqh2.rmanbck tag=TAG20120222T095753
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
Finished restore at 22-FEB-12


Starting recover at 22-FEB-12
using channel ORA_DISK_1


starting media recovery
media recovery complete, elapsed time: 00:00:00
Finished recover at 22-FEB-12
sql statement: alter database datafile 4 online
repair failure complete