wso2~分布式集群部署總結

集群模式需要有關注這塊，如果是all-in-one模式，你部署多個節點，這不叫集群模式，它會有多種狀態信息不一致，會有多種問題，如果希望將網關tw,平臺acp(admin,consumer,publisher)，密鑰管理km，限流管理tm分開部署多份，需要使用Distributed Deployment 模式(https://apim.docs.wso2.com/en/latest/install-and-setup/setup/kubernetes-deployment/kubernetes/am-pattern-3-acp-tm-gw/)。

https://www.yenlo.com/blogs/custom-rate-limiting-with-wso2-apim/
https://apim.docs.wso2.com/en/4.1.0/design/rate-limiting/advanced-topics/configuring-rate-limiting-api-gateway-cluster/
https://apim.docs.wso2.com/en/latest/manage-apis/design/rate-limiting/advanced-topics/configuring-rate-limiting-api-gateway-cluster/
https://github.com/wso2/api-manager/issues/3961
https://github.com/wso2/api-manager/issues/359

helm-distribute分布式部署

1. 部署順序

control-plane
gateway
traffic-manager
key-manager[未在私有倉庫找到景象，目前它集成到control-plane里]

2. 各子服務Dockerfile添加jdbc

FROM wso2/wso2am-tm:4.5.0

ARG USER=wso2carbon
ARG USER_HOME=/home/${USER}
ARG WSO2_SERVER_NAME=wso2am-tm
ARG WSO2_SERVER_VERSION=4.5.0
ARG WSO2_SERVER=${WSO2_SERVER_NAME}-${WSO2_SERVER_VERSION}
ARG WSO2_SERVER_HOME=${USER_HOME}/${WSO2_SERVER}

# Copy JDBC MySQL driver
ADD --chown=wso2carbon:wso2 https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.28/mysql-connector-java-8.0.28.jar ${WSO2_SERVER_HOME}/repository/components/lib

3. gateway中values.yaml中配置acp的服務名和實例名

wso2:
 apim:
   configurations
      throttling:
        # -- Traffic manager loadbalancer service url for throttling.
        # If the traffic manager is separated from control-plane, all urls should be pointed to traffic-manager ones.
        serviceUrl: "wso2am-acp-service" # 對應control-plane的主服務名
      eventhub:
        # -- Event hub (control plane) enabled.
        # This should be enabled if the traffic manager is separated from the control-plane.
        enabled: false
        # -- Event hub (control plane) loadbalancer service url
        serviceUrl: "wso2am-acp-service"

4. 各服務的deployment.yaml文件添加本地時區變量

control-plane\templates\control-plane\instance-1\wso2am-cp-deployment.yaml
control-plane\templates\control-plane\instance-2\wso2am-cp-deployment.yaml
gateway\templates\gateway\wso2am-gateway-deployment.yaml
traffic-manager\templates\traffic-manager\instance-1\wso2am-tm-deployment.yaml
traffic-manager\templates\traffic-manager\instance-2\wso2am-tm-deployment.yaml

spec:
  template:
    spec:
      containers:
      - name: wso2am-traffic-manager
        image: {{ .Values.wso2.deployment.image.registry }}/{{ .Values.wso2.deployment.image.repository }}@{{ .Values.wso2.deployment.image.digest }}
        imagePullPolicy: {{ .Values.wso2.deployment.imagePullPolicy }}
        env:
        - name: PROFILE_NAME
          value: traffic-manager
        - name: NODE_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: JVM_MEM_OPTS
          value: "-Xms{{ .Values.wso2.deployment.resources.jvm.memory.xms }} -Xmx{{ .Values.wso2.deployment.resources.jvm.memory.xmx }}"
        - name: TZ
          value: Asia/Shanghai

5. 多節點集群

cp(control-plane) 支持多副本，多個deployment
gateway 無狀態，直接使用deployment多副本功能
traffic-manager 支持多副本，多個deployment

redis throttle相關【需再調研】

分布式部署問題點說明

兩個acp配置問題，acp需要配置自己和對方的發布消息的端口9611和9711，他們需要通訊，配置對方的5672的mq端口
一個gateway沒有正常配置兩個acp，gateway需要配置所有acp的5672，9611，9711端口，和一個公共的acp服務的9433端口
兩個tm配置問題，tm需要配置對方tm的5672端口，和一個公共的acp服務的9433端口

apim-acp 兩個實例，配置時1配置2，2配置1

wso2am-acp-1配置

[apim.event_hub]
enable = true
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
event_listening_endpoints = ["tcp://localhost:5672"]
event_duplicate_url = ["tcp://wso2am-acp-2-service:5672"]

[[apim.event_hub.publish.url_group]]
urls = ["tcp://wso2am-acp-1-service:9611"]
auth_urls = ["ssl://wso2am-acp-1-service:9711"]
[[apim.event_hub.publish.url_group]]
urls = ["tcp://wso2am-acp-2-service:9611"]
auth_urls = ["ssl://wso2am-acp-2-service:9711"]

wso2am-acp-2配置

[apim.event_hub]
enable = true
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
event_listening_endpoints = ["tcp://localhost:5672"]
event_duplicate_url = ["tcp://wso2am-acp-1-service:5672"]

[[apim.event_hub.publish.url_group]]
urls = ["tcp://wso2am-acp-1-service:9611"]
auth_urls = ["ssl://wso2am-acp-1-service:9711"]
[[apim.event_hub.publish.url_group]]
urls = ["tcp://wso2am-acp-2-service:9611"]
auth_urls = ["ssl://wso2am-acp-2-service:9711"]

apim-gateway 兩個副本，配置acp1和acp2

# Traffic Manager configurations
[apim.throttling]
username= "$ref{super_admin.username}"
password= "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
throttle_decision_endpoints = ["tcp://wso2am-acp-1-service:5672","tcp://wso2am-acp-2-service:5672"]
enable_unlimited_tier = true
enable_header_based_throttling = true
enable_jwt_claim_based_throttling = true
enable_query_param_based_throttling = true


[[apim.throttling.url_group]]
traffic_manager_urls = ["tcp://wso2am-acp-1-service:9611","tcp://wso2am-acp-2-service:9611"]
traffic_manager_auth_urls = ["ssl://wso2am-acp-1-service:9711","ssl://wso2am-acp-2-service:9711"]
type = "loadbalance"

[apim.event_hub]
enable = true
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
event_listening_endpoints = ["tcp://wso2am-acp-1-service:5672","tcp://wso2am-acp-2-service:5672"]

apim-tm 兩個實例，配置時1配置2，2配置1

wso2am-tm-1配置

[apim.key_manager]
service_url = "https://wso2am-acp-service:9443/services/"
# Traffic Manager configurations
[apim.throttling]
event_duplicate_url = ["tcp://wso2am-tm-2-service:5672"]
service_url = "https://wso2am-tm-service:9443/services/"
throttle_decision_endpoints = ["tcp://localhost:5672"]

[apim.event_hub]
enable = true
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
event_listening_endpoints = ["tcp://wso2am-acp-2-service:5672"]

wso2am-tm-2配置

[apim.key_manager]
service_url = "https://wso2am-acp-service:9443/services/"
# Traffic Manager configurations
[apim.throttling]
event_duplicate_url = ["tcp://wso2am-tm-1-service:5672"]
service_url = "https://wso2am-tm-service:9443/services/"
throttle_decision_endpoints = ["tcp://localhost:5672"]

[apim.event_hub]
enable = true
username = "$ref{super_admin.username}"
password = "$ref{super_admin.password}"
service_url = "https://wso2am-acp-service:9443/services/"
event_listening_endpoints = ["tcp://wso2am-acp-1-service:5672"]

posted @ 2025-07-22 13:29 張占嶺閱讀(56) 評論(0) 收藏舉報

刷新頁面返回頂部