背景介紹

一套在RHEL的集群上運行的Oracle實例,是用systemd服務啟動Oracle實例的(方便集群的切換操作).在測試過程中發現標準大頁沒有被用上.
具體情況如下所示:

$ grep HugePages /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    2034
HugePages_Free:     2034
HugePages_Rsvd:        0
HugePages_Surp:        0

原因分析

如上所示,HugePages_Free的值為2034, HugePages_Total的值也是2034,也就是說標準大頁完全沒有使用.

Linux服務器的基本信息如下

$ more /etc/redhat-release 
Red Hat Enterprise Linux release 8.10 (Ootpa)
$ free -m
              total        used        free      shared  buff/cache   available
Mem:          11697        4929        5986          17         780        6612
Swap:         16383           0       16383

檢查數據庫的參數如下, 完全符合條件:

SQL> select banner from v$version;

BANNER
--------------------------------------------------------------------------------
Oracle Database 19c Enterprise Edition Release 19.0.0.0.0 - Production

SQL> col name for a20;
SQL> col value for a32;
SQL> select name, value from v$parameter 
  2  where name in ('memory_target','sga_target','use_large_pages');

NAME                 VALUE
-------------------- --------------------------------
use_large_pages      TRUE
sga_target           4261412864
memory_target        0

SQL>

內核參數vm.nr_hugepages也是正確設置的,具體如下所示

$ grep vm.nr_hugepages /etc/sysctl.conf 
vm.nr_hugepages = 2034
$ ./hugepages_settings.sh

This script is provided by Doc ID 401749.1 from My Oracle Support
(http://support.oracle.com) where it is intended to compute values for
the recommended HugePages/HugeTLB configuration for the current shared
memory segments on Oracle Linux. Before proceeding with the execution please note following:
 * For ASM instance, it needs to configure ASMM instead of AMM.
 * The 'pga_aggregate_target' is outside the SGA and
   you should accommodate this while calculating the overall size.
 * In case you changes the DB SGA size,
   as the new SGA will not fit in the previous HugePages configuration,
   it had better disable the whole HugePages,
   start the DB with new SGA size and run the script again.
And make sure that:
 * Oracle Database instance(s) are up and running
 * Oracle Database 11g Automatic Memory Management (AMM) is not setup
   (See Doc ID 749851.1)
 * The shared memory segments can be listed by command:
     # ipcs -m


Press Enter to proceed...

Recommended setting: vm.nr_hugepages = 2034

資源限制配置文件limits.conf中的memlock的設置也是正確的.如下所示:

# grep memlock /etc/security/limits.conf 
#        - memlock - max locked-in-memory address space (KB)
oracle   soft   memlock    10485760
oracle   hard   memlock    10485760
# su - oracle
Last login: Fri Aug  8 13:54:36 CST 2025 on pts/0
$ ulimit -l
10485760
$ grep memlock /etc/security/limits.conf 
#        - memlock - max locked-in-memory address space (KB)
oracle   soft   memlock    10485760
oracle   hard   memlock    10485760

很是納悶為什么配置都是正確,但是Oracle就是不用標準大頁,于是重啟一下Oracle實例,在告警日志中發現了蛛絲馬跡,如下所示:

**********************************************************************
2025-08-08T13:50:16.662256+08:00
Dump of system resources acquired for SHARED GLOBAL AREA (SGA) 

2025-08-08T13:50:16.662285+08:00
 Domain name: system.slice/bpsdbsvr.service
2025-08-08T13:50:16.662302+08:00
 Per process system memlock (soft) limit = 64K
2025-08-08T13:50:16.662318+08:00
 Expected per process system memlock (soft) limit to lock
 instance MAX SHARED GLOBAL AREA (SGA) into memory: 4066M
2025-08-08T13:50:16.662356+08:00
 Available system pagesizes:
  4K, 2048K 
2025-08-08T13:50:16.662387+08:00
 Supported system pagesize(s):
2025-08-08T13:50:16.662404+08:00
  PAGESIZE  AVAILABLE_PAGES  EXPECTED_PAGES  ALLOCATED_PAGES  ERROR(s)
2025-08-08T13:50:16.662421+08:00
        4K       Configured              11          1040395        NONE
2025-08-08T13:50:16.662450+08:00
     2048K             2034            2033                0        NONE
2025-08-08T13:50:16.662466+08:00
RECOMMENDATION:
2025-08-08T13:50:16.662483+08:00
 1. Increase per process memlock (soft) limit to at least 4066MB
 to lock 100% of SHARED GLOBAL AREA (SGA) pages into physical memory
2025-08-08T13:50:16.662514+08:00
**********************************************************************

檢查Oracle進程的限制,發現進程的Max locked memory為65536,也就是64K.

# oracle_pid=$(pgrep -f "_pmon_")
# cat /proc/$oracle_pid/limits
Limit                     Soft Limit           Hard Limit           Units     
Max cpu time              unlimited            unlimited            seconds   
Max file size             unlimited            unlimited            bytes     
Max data size             unlimited            unlimited            bytes     
Max stack size            33554432             unlimited            bytes     
Max core file size        0                    unlimited            bytes     
Max resident set          unlimited            unlimited            bytes     
Max processes             46635                46635                processes 
Max open files            262144               262144               files     
Max locked memory         65536                65536                bytes     
Max address space         unlimited            unlimited            bytes     
Max file locks            unlimited            unlimited            locks     
Max pending signals       46635                46635                signals   
Max msgqueue size         819200               819200               bytes     
Max nice priority         0                    0                    
Max realtime priority     0                    0                    
Max realtime timeout      unlimited            unlimited            us        

也就是說systemd服務啟動Oracle實例時,由于某些原因memlock依然是64K,即日志中的提示"Per process system memlock (soft) limit = 64K"

于是改用手工啟動數據庫實例,檢查發現標準大頁被Oracle使用了,但是systemd服務啟動Oracle實例就會出現上面標準大頁不被使用的情況

$ grep HugePages /proc/meminfo
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
FileHugePages:         0 kB
HugePages_Total:    2034
HugePages_Free:        4
HugePages_Rsvd:        3
HugePages_Surp:        0

后面和同事查資料,發現systemctl啟動的服務默認不讀取資源限制配置文件(limits.conf). limits.conf中的限制是針對用戶會話級別的資源控制，
由PAM模塊在用戶登錄時生效。而systemd服務是通過systemd進程直接啟動的，屬于非登錄會話，默認不會觸發PAM的pam_limits.so模塊，
因此/etc/security/limits.conf 中為oracle用戶設置的所有資源限制都不會自動應用到通過 systemd 服務啟動的進程.

解決方案

如果想讓systemd服務啟動Oracle實例時memlock限制生效，可以在oracle.service 中直接配置 memlock 限制或者通過PAM讓服務讀取limits.conf.
網上資料推薦直接在systemd服務文件中配置(這是 systemd 推薦的方式，比依賴limits.conf更可靠), 如下所示:

oracle.service原始的配置

[Unit]
Description=Oracle Database Service
After=network.target

[Service]
Type=forking
User=oracle
Group=oinstall
ExecStart=/home/oracle/xxxx/ora19c.sh start
ExecStop=/home/oracle/xxxx/ora19c.sh shutdown
StandardOutput=append:/var/log/rhcs_resource_logs/xxx/xxx.log
RemainAfterExit=yes
KillMode=none

[Install]
WantedBy=multi-user.target

注意: oracle.service配置做了一點混淆,不影響大家理解.

oracle.service修改后配置

[Unit]
Description=Oracle Database Service
After=network.target

[Service]
Type=forking
User=oracle
Group=oinstall
ExecStart=/home/oracle/xxxx/ora19c.sh start
ExecStop=/home/oracle/xxxx/ora19c.sh shutdown
StandardOutput=append:/var/log/rhcs_resource_logs/xxx/xxx.log
RemainAfterExit=yes
KillMode=none

# oracle /etc/security/limits.conf
LimitNPROC=16384
LimitNOFILE=65536
LimitSTACK=10485760
LimitMEMLOCK=10737418240

[Install]
WantedBy=multi-user.target

這樣配置后,通過systemd服務啟動Oracle實例就能正常使用標準大頁了.問題Troubleshooting完美解決. 究其原因還是因為對Linux的systemd服務了解得不夠多.不夠深入!