[root@ocasxxxt00dbre1 audit]# ls -ltrh | tail -n 30\r\n-r--------. 1 root root 6.1M Aug 15 10:24 audit.log.29\r\n-r--------. 1 root root 6.1M Aug 15 14:08 audit.log.28\r\n-r--------. 1 root root 6.1M Aug 15 18:00 audit.log.27\r\n-r--------. 1 root root 6.1M Aug 15 21:48 audit.log.26\r\n-r--------. 1 root root 6.1M Aug 16 01:25 audit.log.25\r\n-r--------. 1 root root 6.1M Aug 16 05:20 audit.log.24\r\n-r--------. 1 root root 6.1M Aug 16 09:15 audit.log.23\r\n-r--------. 1 root root 6.1M Aug 16 11:42 audit.log.22\r\n-r--------. 1 root root 6.1M Aug 16 12:59 audit.log.21\r\n-r--------. 1 root root 6.1M Aug 16 14:15 audit.log.20\r\n-r--------. 1 root root 6.1M Aug 16 15:32 audit.log.19\r\n-r--------. 1 root root 6.0M Aug 16 16:49 audit.log.18\r\n-r--------. 1 root root 6.1M Aug 16 18:03 audit.log.17\r\n-r--------. 1 root root 6.1M Aug 16 19:18 audit.log.16\r\n-r--------. 1 root root 6.1M Aug 16 20:34 audit.log.15\r\n-r--------. 1 root root 6.1M Aug 16 21:51 audit.log.14\r\n-r--------. 1 root root 6.1M Aug 16 23:06 audit.log.13\r\n-r--------. 1 root root 6.1M Aug 17 00:17 audit.log.12\r\n-r--------. 1 root root 6.1M Aug 17 01:32 audit.log.11\r\n-r--------. 1 root root 6.1M Aug 17 02:48 audit.log.10\r\n-r--------. 1 root root 6.1M Aug 17 04:03 audit.log.9\r\n-r--------. 1 root root 6.1M Aug 17 05:16 audit.log.8\r\n-r--------. 1 root root 6.1M Aug 17 06:32 audit.log.7\r\n-r--------. 1 root root 6.1M Aug 17 07:46 audit.log.6\r\n-r--------. 1 root root 6.1M Aug 17 09:03 audit.log.5\r\n-r--------. 1 root root 6.1M Aug 17 10:18 audit.log.4\r\n-r--------. 1 root root 6.1M Aug 17 11:35 audit.log.3\r\n-r--------. 1 root root 6.1M Aug 17 12:53 audit.log.2\r\n-r--------. 1 root root 6.1M Aug 17 14:10 audit.log.1\r\n-rw-------. 1 root root 740K Aug 17 14:20 audit.log\r\n[root@ocasxxxt00dbre1 audit]# \r\n<\/pre>\nAuditd, responsible for those files, is a important service that keeps track of changes on your OS that could potentially be a security risk. Disabling it was not an option.<\/p>\n
This server had multiple databases.. checking the content of those audit files, there were thousands and thousands of entries caused mostly by:<\/p>\n
- \n
- $ORACLE_HOME\/bin\/tnslsnr<\/strong><\/li>\n
- $ORACLE_HOME\/bin\/oracle<\/strong><\/li>\n<\/ul>\n
80 MBs\/day was not a huge problem. The filesystem had more than 30GBs free space so it could handle those files until some space alert triggered, someone deleted them or the retention cleaned them up.<\/p>\n
The major issue I faced was when for some reason the archivelog<\/strong> area run out of space. The \/var\/log\/audit<\/strong><\/span> started growing so fast, but so fast, that in only\u00a05 hours<\/span> my filesystem run out of space. This caused sshd to become inaccessible, boot to fail and I had to clean up my filesystem manually attaching it on a working machine.<\/p>\n
The Bombflow was:<\/p>\n
Archivelog Full -> Auditd out of control -> “\/” out of space -> sshd unavailable -> server unable to boot.<\/strong><\/p>\n
After some investigation, I solved this issue by appending to \/etc\/audit\/audit.rules<\/strong> the following lines:<\/p>\n
# Oracle Binaries – Ignore
\nw never,exclude -F path=\/u01\/app\/oracle\/ -k exclude\u00a0<\/span><\/p>\nWhere\u00a0\/u01\/app\/oracle\/<\/strong> is my ORACLE_BASE<\/strong>.<\/p>\n
Restarting the service via “service auditd restart<\/strong><\/span>” didn’t work as the configuration was immutable. So reboot was required<\/span> to change audit rules (this protection happens when there is a “-e 2<\/strong>” in\u00a0audit.rules file).<\/p>\n
References:<\/p>\n
- $ORACLE_HOME\/bin\/oracle<\/strong><\/li>\n<\/ul>\n
