Arthas 是一款線上監控診斷產品，通過全局視角實時查看應用 load、內存、gc、線程的狀態信息，并能在不修改應用代碼的情況下，對業務問題進行診斷，包括查看方法調用的出入參、異常，監測方法執行耗時，類加載信息等，大大提升線上問題排查效率

一、啟動安裝
##1.下載
wget https://arthas.aliyun.com/arthas-boot.jar;
##2.運行
java -jar arthas-boot.jar;
例如： Windows上使用參考

1.執行java -jar arthas-boot.jar

D:\Soft\arthas-bin>java -jar arthas-boot.jar
[INFO] arthas-boot version: 3.5.4
[INFO] Process 10668 already using port 3658
[INFO] Process 10668 already using port 8563
[INFO] Found existing java process, please choose one and input the serial number of the process, eg : 1. Then hit ENTER.
* [1]: 10668 org.jboss.Main
[2]: 5952
[3]: 2240
2.10668為需要監控的進程，是一個啟動的Jboss服務，輸入10668 前方的序號1回車，開始診斷監控，如下

[INFO] Process 10668 already using port 8563
[INFO] Found existing java process, please choose one and input the serial number of the process, eg : 1. Then hit ENTER.
* [1]: 10668 org.jboss.Main
[2]: 5952
[3]: 2240
1
[INFO] arthas home: D:\Soft\arthas-bin
[INFO] The target process already listen port 3658, skip attach.
[INFO] arthas-client connect 127.0.0.1 3658
,---. ,------. ,--------.,--. ,--. ,---. ,---.
/ O \ | .--. ''--. .--'| '--' | / O \ ' .-'
| .-. || '--'.' | | | .--. || .-. |`. `-.
| | | || |\ \ | | | | | || | | |.-' |
`--' `--'`--' '--' `--' `--' `--'`--' `--'`-----'


wiki https://arthas.aliyun.com/doc
tutorials https://arthas.aliyun.com/doc/arthas-tutorials.html
version 3.5.4
main_class
pid 10668
time 2022-12-23 16:33:09
二、能使用Arthas(阿爾薩斯）做什么
1. 全局查看系統的運行狀況
dashboard 命令，查看當前系統實時數據面板

參數說明

-i : 指刷新的時間間隔

-n: 指刷新次數

例如：

[arthas@10668]$ dashboard
ID NAME GROUP PRIORITY STATE %CPU DELTA_TIME TIME INTERRUPTED DAE
129 Thread-24 jboss 5 TIMED_WAITING 0.0 0.000 4:23.345 false fal
14 ScannerThread jboss 5 TIMED_WAITING 0.0 0.000 4:22.596 false tru
157 http-0.0.0.0-8091-1 jboss 5 WAITING 0.0 0.000 0:34.304 false tru
-1 C2 CompilerThread0 - -1 - 0.0 0.000 0:27.019 false tru
-1 C2 CompilerThread1 - -1 - 0.0 0.000 0:26.894 false tru
-1 VM Thread - -1 - 0.0 0.000 0:19.734 false tru
-1 GC task thread#2 (ParallelGC) - -1 - 0.0 0.000 0:10.420 false tru
-1 GC task thread#4 (ParallelGC) - -1 - 0.0 0.000 0:10.405 false tru
-1 GC task thread#7 (ParallelGC) - -1 - 0.0 0.000 0:10.233 false tru
-1 GC task thread#3 (ParallelGC) - -1 - 0.0 0.000 0:10.186 false tru
-1 GC task thread#6 (ParallelGC) - -1 - 0.0 0.000 0:9.968 false tru
-1 GC task thread#0 (ParallelGC) - -1 - 0.0 0.000 0:9.906 false tru
-1 GC task thread#5 (ParallelGC) - -1 - 0.0 0.000 0:9.765 false tru
-1 GC task thread#1 (ParallelGC) - -1 - 0.0 0.000 0:9.672 false tru
52 JCA PoolFiller jboss 5 WAITING 0.0 0.000 0:7.113 false fal
154 arthas-command-execute system 5 RUNNABLE 0.0 0.000 0:5.210 false tru
25 Thread-5 jboss 5 TIMED_WAITING 0.0 0.000 0:4.882 false fal
153 arthas-NettyHttpTelnetBootstrap-3-2 system 5 RUNNABLE 0.0 0.000 0:1.762 false tru
88 IdleRemover jboss 5 TIMED_WAITING 0.0 0.000 0:0.374 false tru
-1 VM Periodic Task Thread - -1 - 0.0 0.000 0:0.296 false tru
49 DefaultQuartzScheduler_QuartzSchedulerThread QuartzScheduler:DefaultQuart 5 TIMED_WAITING 0.0 0.000 0:0.280 false fal
24 Thread-4 jboss 5 TIMED_WAITING 0.0 0.000 0:0.234 false tru
133 AWT-Windows system 6 RUNNABLE 0.0 0.000 0:0.093 false tru
3 Finalizer system 8 WAITING 0.0 0.000 0:0.062 false tru
10 DestroyJavaVM main 5 RUNNABLE 0.0 0.000 0:0.062 false fal
2 Reference Handler system 10 WAITING 0.0 0.000 0:0.046 false tru
145 logback-1 system 5 WAITING 0.0 0.000 0:0.046 false tru
35 ContainerBackgroundProcessor[StandardEngine[jboss.web]] jboss 5 TIMED_WAITING 0.0 0.000 0:0.046 false tru
39 DefaultQuartzScheduler_Worker-0 jboss 5 TIMED_WAITING 0.0 0.000 0:0.046 false fal
40 DefaultQuartzScheduler_Worker-1 jboss 5 TIMED_WAITING 0.0 0.000 0:0.046 false fal
70 Framework Active Thread jboss 5 TIMED_WAITING 0.0 0.000 0:0.031 false fal
138 http-0.0.0.0-8091-Acceptor-0 jboss 5 RUNNABLE 0.0 0.000 0:0.031 false tru
160 stariboss-payment.data jboss 5 TIMED_WAITING 0.0 0.000 0:0.031 false fal
5 Attach Listener system 5 RUNNABLE 0.0 0.000 0:0.015 false tru
19 GC Daemon system 2 TIMED_WAITING 0.0 0.000 0:0.015 false tru
146 arthas-NettyHttpTelnetBootstrap-3-1 system 5 RUNNABLE 0.0 0.000 0:0.015 false tru
Memory used total max usage GC
heap 201M 346M 455M 44.31% gc.ps_scavenge.count 660
ps_eden_space 31M 83M 169M 18.46% gc.ps_scavenge.time(ms) 10991
ps_survivor_space 448K 1024K 1024K 43.75% gc.ps_marksweep.count 69
ps_old_gen 169M 262M 341M 49.85% gc.ps_marksweep.time(ms) 18641
nonheap 175M 178M 304M 57.80%
code_cache 7M 10M 48M 16.48%
ps_perm_gen 167M 168M 256M 65.55%
direct 0K 0K - 0.00%
mapped 1M 1M - 100.00%
Runtime
os.name Windows 7
os.version 6.1
java.version 1.7.0_80
java.home C:\Java\jdk1.7.0_80\jre
systemload.average -1.00
processors 8
timestamp/uptime Mon Dec 26 10:56:25 CST 2022/239168s
[arthas@10668]$
說明

ID：Java級別的線程ID，注意這個ID不能跟jstack中的nativeID一一對應

NAME：線程名

GROUP：線程組名

PRIORITY：線程優先級, 1~10之間的數字，越大表示優先級越高

STATE：線程的狀態

CPU%：線程消耗的cpu占比，采樣100ms，將所有線程在這100ms內的cpu使用量求和，再算出每個線程的cpu使用占比。

TIME：線程運行總時間，數據格式為

分：秒
INTERRUPTED：線程當前的中斷位狀態

DAEMON：是否是daemon線程

2. 查看指定線程信息
Thread命令

參數說明

n：指定最忙的錢N個線程

b：找出當前是阻塞其他線程的線程

i ：指定CPU占比統計的采樣時間，單位毫秒

例如 ：

thread 顯示所有線程信息

thread 1 顯示1號線程的運行堆棧

thread -b 查看阻塞的線程信息

thread -n 3 查看CPU使用率最高的3個線程

thread -i 1000 -n 3 ,每過1000秒采用，顯示CPU使用最高的線程

#查看最耗CPU的兩個線程

[arthas@10668]$ thread -n
"Reference Handler" Id=2 cpuUsage=0.0% deltaTime=0ms time=46ms WAITING on java.lang.ref.Reference$Lock@2ff15375
at java.lang.Object.wait(Native Method)
- waiting on java.lang.ref.Reference$Lock@2ff15375
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)

"http-0.0.0.0-8091-1" Id=157 cpuUsage=0.0% deltaTime=0ms time=9812ms RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.DataPacket.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1104)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1075)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:480)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:219)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:970)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1190)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3370)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3454)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:365)
at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:824)
at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:818)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:589)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:818)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:874)
at com.star.sms.dao.utils.SmsJdbcTemplate.update(SmsJdbcTemplate.java:119)
at com.star.sms.dao.utils.SmsJdbcTemplate.modify(SmsJdbcTemplate.java:227)
at com.star.sms.dao.customer.jdbc.CustomerInfoDaoJdbcImpl.modifyCustomer(CustomerInfoDaoJdbcImpl.java:216)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at com.star.common.interceptor.ThreadCacheInterceptor.invoke(ThreadCacheInterceptor.java:66)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
#查看157號線程信息

[arthas@10668]$ thread 157
"http-0.0.0.0-8091-1" Id=157 RUNNABLE (in native)
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.read(SocketInputStream.java:152)
at java.net.SocketInputStream.read(SocketInputStream.java:122)
at oracle.net.ns.Packet.receive(Unknown Source)
at oracle.net.ns.DataPacket.receive(Unknown Source)
at oracle.net.ns.NetInputStream.getNextPacket(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.net.ns.NetInputStream.read(Unknown Source)
at oracle.jdbc.driver.T4CMAREngine.unmarshalUB1(T4CMAREngine.java:1104)
at oracle.jdbc.driver.T4CMAREngine.unmarshalSB1(T4CMAREngine.java:1075)
at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:480)
at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:219)
at oracle.jdbc.driver.T4CPreparedStatement.executeForRows(T4CPreparedStatement.java:970)
at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1190)
at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3370)
at oracle.jdbc.driver.OraclePreparedStatement.executeUpdate(OraclePreparedStatement.java:3454)
at org.jboss.resource.adapter.jdbc.WrappedPreparedStatement.executeUpdate(WrappedPreparedStatement.java:365)
at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:824)
at org.springframework.jdbc.core.JdbcTemplate$2.doInPreparedStatement(JdbcTemplate.java:818)
at org.springframework.jdbc.core.JdbcTemplate.execute(JdbcTemplate.java:589)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:818)
at org.springframework.jdbc.core.JdbcTemplate.update(JdbcTemplate.java:874)
at com.star.sms.dao.utils.SmsJdbcTemplate.update(SmsJdbcTemplate.java:119)
at com.star.sms.dao.utils.SmsJdbcTemplate.modify(SmsJdbcTemplate.java:227)
at com.star.sms.dao.customer.jdbc.CustomerInfoDaoJdbcImpl.modifyCustomer(CustomerInfoDaoJdbcImpl.java:216)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at com.star.common.interceptor.ThreadCacheInterceptor.invoke(ThreadCacheInterceptor.java:66)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy1908.modifyCustomer(Unknown Source)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
如果堆棧信息過大， 可以輸出日志到文件

默認情況下arthas執行結果是不保存日志的，當然我們通過arthas提供的命令行options進行開啟:

關閉日志打印

[arthas@10668]$ options save-result false NAME BEFORE-VALUE AFTER-VALUE

save-result true false
開啟日志打印 [arthas@10668]$ options save-result true NAME BEFORE-VALUE AFTER-VALUE

save-result false true

[arthas@10668]$
日志會異步保存在 {user.home}/logs/arthas-cache/result.log;

3.排查為什么沒有執行到修改的代碼
jad 命令

參數說明

jad 包名.類名 反編譯類
jad包名.類名 方法名 只反編譯某個方法
例如：

反編譯com.star.sms.service.accept2.dto.NewInstallAcceptDTO如下。jad命令不僅可以查看到ClassLoader還能看到運行的Location信息。 這樣可以很清楚了解到運行的是那個包下的類。類覆蓋問題就變的很容易發現。

[arthas@10668]$
[arthas@10668]$ jad com.star.sms.service.accept2.dto.NewInstallAcceptDTO

ClassLoader:
+-com.star.osgi.patch.classloader.BundlePatchURLClassLoader[stariboss-accept_interface]
+-org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader@4911704b[stariboss-accept_interface:1.0.0(id=99)]
+-org.eclipse.osgi.baseadaptor.BaseAdaptor$1@3983efa2

Location:
/E:/Jboss/jboss-4.2.3-8.2.1R1-8091-liaoning_821-5g/server/default/./deploy/stariboss.war/tmp/.patchtmp/1671784240888/a_780103_server_22112711_DB-9452.stariboss-accept_interface.jar

/*
* Decompiled with CFR.
*
* Could not load the following classes:
* com.star.common.lang.Money
* com.star.sms.model.customer.Customer
* com.star.sms.service.accept2.dto.AcceptSheetDTO
* com.star.sms.service.accept2.dto.BatchNoteInfoDTO
*/
package com.star.sms.service.accept2.dto;

import com.star.common.lang.Money;
import com.star.sms.model.customer.Customer;
import com.star.sms.service.accept2.dto.AcceptSheetDTO;
import com.star.sms.service.accept2.dto.BatchNoteInfoDTO;
import java.io.Serializable;
import java.util.Map;

/*
* This class specifies class file version 49.0 but uses Java 6 signatures. Assumed Java 6.
*/
public class NewInstallAcceptDTO
implements Serializable {
private static final long serialVersionUID = 5229054864090922612L;
private int rowNum;
private AcceptSheetDTO acceptSheetDTO;
private Customer customer;
private Money businessFee;
private Money preSavingFee;
private Money transferFee;
private Map<String, BatchNoteInfoDTO> notedtos;
private boolean contactTelRepeatAble = false;

public boolean getContactTelRepeatAble() {
/*86*/ return this.contactTelRepeatAble;
}

public void setContactTelRepeatAble(boolean contactTelRepeatAble) {
/*90*/ this.contactTelRepeatAble = contactTelRepeatAble;
}

public Customer getCustomer() {
/*38*/ return this.customer;
}

public void setCustomer(Customer customer) {
/*42*/ this.customer = customer;
}

public AcceptSheetDTO getAcceptSheetDTO() {
/*30*/ return this.acceptSheetDTO;
}

public void setAcceptSheetDTO(AcceptSheetDTO acceptSheetDTO) {
/*34*/ this.acceptSheetDTO = acceptSheetDTO;
}

public Money getBusinessFee() {
/*46*/ return this.businessFee;
}

public void setBusinessFee(Money businessFee) {
/*50*/ this.businessFee = businessFee;
}

public Money getPreSavingFee() {
/*54*/ return this.preSavingFee;
}

public void setPreSavingFee(Money preSavingFee) {
/*58*/ this.preSavingFee = preSavingFee;
}

public Money getTransferFee() {
/*62*/ return this.transferFee;
}

public void setTransferFee(Money transferFee) {
/*66*/ this.transferFee = transferFee;
}

public Map<String, BatchNoteInfoDTO> getNotedtos() {
/*70*/ return this.notedtos;
}

public void setNotedtos(Map<String, BatchNoteInfoDTO> notedtos) {
/*74*/ this.notedtos = notedtos;
}

public int getRowNum() {
/*78*/ return this.rowNum;
}

public void setRowNum(int rowNum) {
/*82*/ this.rowNum = rowNum;
}
}
如果類過大，反編譯內容會很多，建議只反編譯某個方法，例如

[arthas@10668]$
[arthas@10668]$ jad com.star.sms.service.support.GroupParamContainer getCustomer
Affect(row-cnt:0) cost in 37 ms.
No class found for: com.star.sms.service.support.GroupParamContainer
[arthas@10668]$ jad com.star.sms.business.customer.CustomerInfoService getCustomer

ClassLoader:
+-com.star.osgi.patch.classloader.BundlePatchURLClassLoader[stariboss-customer]
+-org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader@4089eba3[stariboss-customer:1.0.0(id=124)]
+-org.eclipse.osgi.baseadaptor.BaseAdaptor$1@3983efa2

Location:
/E:/Jboss/jboss-4.2.3-8.2.1R1-8091-liaoning_821-5g/server/default/./deploy/stariboss.war/tmp/.patchtmp/1671784240888/a_780108_server_22112201_DB-9505.stariboss-customer.jar

private Customer getCustomer(CustomerInfoDTO result, Long certificateTypeId, String certificateNum, SubscriberResultDTO subscriberResultDTO) {
Customer customer = new Customer();
/*730*/ if (this.getCustomerCodeGenerator() == null) {
throw new MsrRuntimeException("customer.code.is.required", false, false, null);
}
/*731*/ this.getCustomerCodeGenerator().generateAndSetCustomerCode(customer);
/*736*/ customer.setName(result.getCUST_NAME());
/*748*/ customer.setCertificate(this.getCertificate(result, certificateTypeId, certificateNum));
/*752*/ customer.setContactAddress("");
/*753*/ customer.setContactTel(result.getCONT_NUMBER());
/*755*/ customer.setEnrolDate(new Date());
/*758*/ customer.setMobileTel(result.getCONT_NUMBER());
/*760*/ customer.setCustomerAddress("");
/*762*/ customer.setSaleChannel((SaleChannel)ApplicationSessionHolder.getApplicationSession().getValue("currentSaleChannel"));
/*765*/ Sex sex = Sex.getByCode((String)result.getGENDER());
/*766*/ customer.setSex(sex == null ? Sex.MAN : sex);
/*769*/ customer.setStatus(CustomerStatus.EXISTING);
/*771*/ customer.setType(CustomerType.INDIVIDUAL);
/*773*/ SaleArea saleArea = this.getSaleAreaByGw(subscriberResultDTO.getCITY_CODE());
/*774*/ customer.setSaleArea(saleArea);
/*777*/ Address address = this.getAddress(saleArea.getId());
/*778*/ customer.setAddress(address);
/*779*/ customer.setContactMan(result.getCUST_NAME());
/*785*/ customer.setDetailAddressCode("");
/*796*/ customer.setRemark("createByGwInfo,");
/*797*/ return customer;
}
4.方法調用路徑，以及性能分析
trace 命令： 方法內部調用路徑，并輸出方法路徑上的每個節點上耗時

例如： 查看GwPaymentService#callFeePay 調用路徑和耗時

[arthas@10668]$ trace com.star.sms.business.payment.gw.GwPaymentService callFeePay
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 184 ms, listenerId: 3
`---ts=2022-12-26 13:37:22;thread_name=http-0.0.0.0-8091-1;id=9d;is_daemon=true;priority=5;TCCL=org.jboss.web.tomc
`---[41.232898ms] com.star.sms.business.payment.gw.GwPaymentService:callFeePay()
+---[0.020813ms] com.star.sms.business.payment.gw.helper.GwPaymentConverter:getInstance() #65
+---[0.018817ms] com.star.sms.business.payment.gw.helper.GwPaymentConverter:callFeePayConvert() #65
+---[0.019102ms] com.star.sms.gw.payment.GwPayDTO:getPayRecordId() #66
`---[41.047583ms] com.star.sms.business.payment.gw.GwPaymentService:doPayFee() #66
trace 命令只會 trace 匹配到的函數里的子調用，并不會向下 trace 多層。因為 trace 是代價比較貴的

可以用正則表匹配路徑上的多個類和函數，一定程度上達到多層 trace 的效果

例如： trace -E com.test.ClassA|org.test.ClassB method1|method2|method3

[arthas@13688]$ trace -E com.star.sms.business.provision.send.InstructionSender sendInstruction|sendGroupByGroup
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 2) cost in 218 ms, listenerId: 7
`---ts=2023-01-06 13:52:59;thread_name=http-0.0.0.0-8089-2;id=a2;is_daemon=true;priority=5;TCCL=org.jboss.web.tomcat.service.WebAppClassLoader@2c08c092
`---[3.387799ms] com.star.sms.business.provision.send.InstructionSender:sendInstruction()
+---[0.028225ms] org.springframework.util.CollectionUtils:isEmpty() #24
+---[0.01397ms] com.star.sms.model.product.core.business.OpenDevice:getOpenDeviceType() #27
+---[0.007413ms] com.star.sms.model.provision.enums.OpenDeviceType:name() #27
+---[0.005987ms] com.star.sms.model.product.core.business.OpenDevice:getCode() #27
+---[0.29422ms] org.apache.commons.logging.Log:info() #27
+---[0.01511ms] com.star.sms.business.provision.send.util.InstructionGroupUtil:groupInstructionByServiceStr() #29
+---[0.137417ms] com.star.sms.business.provision.send.InstructionSender:translate2Req() #30
+---[2.620605ms] com.star.sms.business.provision.send.InstructionSender:sendGroupByGroup() #31
| `---[2.603785ms] com.star.sms.business.provision.send.InstructionSender:sendGroupByGroup()
| `---[2.538497ms] com.star.sms.business.provision.send.InstructionSender:sendOneGroupReq() #52
+---[0.003421ms] com.star.sms.model.product.core.business.OpenDevice:getOpenDeviceType() #32
+---[0.003136ms] com.star.sms.model.provision.enums.OpenDeviceType:name() #32
+---[0.002851ms] com.star.sms.model.product.core.business.OpenDevice:getCode() #32
`---[0.118885ms] org.apache.commons.logging.Log:info() #32
5.線上 debug
watch命令檢測函數入參返回值，拋出異常等信息

參數說明、

params : 表示所有參數數組 （因為不確定是幾個參數）
returnObject:表示返回值
-b 方法調用之前觀察
-e 方法異常后觀察
-s 方法返回后觀察
-f 方法結束后觀察
-x 屬性深度
例如：

watch com.star.sms.business.payment.gw.GwPayService payOrder returnObj 觀察返回值

watch com.star.sms.business.payment.gw.GwPayService payOrder params 觀察輸入參數

watch com.star.sms.business.payment.gw.GwPayService payOrder "{params, returnObj}" -x 3

[arthas@10668]$ watch com.star.sms.business.payment.gw.GwPayService payOrder returnObj
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 694 ms, listenerId: 1
method=com.star.sms.business.payment.gw.GwPayService.payOrder location=AtExit
ts=2022-12-23 16:39:28; [cost=85.028278ms] result=@GwPayRecord[
serialVersionUID=@Long[6159375491528104642],
USER_ID=null,
groupNumber=null,
phoneNumber=@String[19200000078],
operationType=@GwPayOperaType[CALL_FEE_PAY],
payMethod=@DictionaryData[com.star.sms.model.system.basicparameter.DictionaryData@13d7],
payChannelType=@PayChannelType[SCAN_PAY],
payFee=@Money[錕? 10.00],
payDate=@Date[2022-12-23 00:00:00,000],
tradeNo=@String[202291392],
outTradeNo=@String[221223163927000244],
barCode=null,
buyerId=null,
thirdPayStatus=@PayStatus[SUCCESS],
bossPayStatus=@ScanPaymentSign[UNPAYED],
developOperator=null,
status=@Status[VALID],
acceptSheetId=null,
resp_code=@String[000000],
resp_desc=@String[璋冪敤鎴愬姛],
order_no=@String[12],
order_pay_status=@String[2],
pay_page_url=null,
scan_url=@String[http://www.baidu.com],
sign_info=@String[hhggr],
business_id=null,
gw_pos_trade_no=null,
prov_mer_order_no=null,
posPayStatus=null,
posRespDesc=null,
payNo=@String[13214],
id=@Long[893],
name=null,
createDate=@Timestamp[2022-12-23 16:39:27,292],
modifyDate=@Timestamp[2022-12-23 16:39:28,000],
createOperator=@Operator[com.star.sms.model.system.organization.Operator@5d],
modifyOperator=@Operator[com.star.sms.model.system.organization.Operator@5d],
remark=null,
operationTerminal=@Terminal[com.star.sms.model.system.organization.Terminal@2d660cb5],
operationSpot=@SaleChannel[com.star.sms.model.system.organization.SaleChannel@9],
version=@Integer[0],
]
[arthas@10668]$
[arthas@10668]$ watch com.star.sms.business.payment.gw.GwPayService payOrder "{params, returnObj}" -x 3
Press Q or Ctrl+C to abort.
Affect(class count: 1 , method count: 1) cost in 129 ms, listenerId: 4
method=com.star.sms.business.payment.gw.GwPayService.payOrder location=AtExit
ts=2022-12-26 13:56:10; [cost=43.709849ms] result=@ArrayList[
@Object[][
@GwPayRecord[
serialVersionUID=@Long[6159375491528104642],
USER_ID=null,
groupNumber=null,
phoneNumber=@String[19200007899],
operationType=@GwPayOperaType[CALL_FEE_PAY],
payMethod=@DictionaryData[com.star.sms.model.system.basicparameter.DictionaryData@13d7],
payChannelType=@PayChannelType[SCAN_PAY],
payFee=@Money[錕? 99.00],
payDate=@Date[2022-12-26 00:00:00,000],
tradeNo=@String[202299042],
outTradeNo=@String[221226135609002605],
barCode=null,
buyerId=null,
thirdPayStatus=@PayStatus[SUCCESS],
bossPayStatus=@ScanPaymentSign[UNPAYED],
developOperator=null,
status=@Status[VALID],
acceptSheetId=null,
resp_code=@String[000000],
resp_desc=@String[璋冪敤鎴愬姛],
order_no=@String[12],
order_pay_status=@String[2],
pay_page_url=null,
scan_url=@String[http://www.baidu.com],
sign_info=@String[hhggr],
business_id=null,
gw_pos_trade_no=null,
prov_mer_order_no=null,
posPayStatus=null,
posRespDesc=null,
payNo=@String[13214],
id=@Long[898],
name=null,
createDate=@Timestamp[2022-12-26 13:56:09,981],
modifyDate=@Timestamp[2022-12-26 13:56:10,000],
createOperator=@Operator[com.star.sms.model.system.organization.Operator@5d],
modifyOperator=@Operator[com.star.sms.model.system.organization.Operator@5d],
remark=@String[TRACE TEST],
operationTerminal=@Terminal[com.star.sms.model.system.organization.Terminal@7a05dcd9],
operationSpot=@SaleChannel[com.star.sms.model.system.organization.SaleChannel@9],
version=@Integer[0],
],
],
@GwPayRecord[
serialVersionUID=@Long[6159375491528104642],
USER_ID=null,
groupNumber=null,
phoneNumber=@String[19200007899],
operationType=@GwPayOperaType[
CALL_FEE_PAY=@GwPayOperaType[CALL_FEE_PAY],
BUSINESS_PAY=@GwPayOperaType[BUSINESS_PAY],
DAMAGES_PAY=@GwPayOperaType[DAMAGES_PAY],
$VALUES=@GwPayOperaType[][isEmpty=false;size=3],
name=@String[CALL_FEE_PAY],
ordinal=@Integer[0],
],
payMethod=@DictionaryData[
serialVersionUID=@Long[4048790182402733107],
code=@String[2800],
displayPos=@Integer[0],
type=@DictionaryDataType[com.star.sms.model.system.basicparameter.DictionaryDataType@1b5b],
status=@Status[VALID],
operateLevel=null,
saleAreas=null,
synchronizeStatus=@SynchronizeStatus[UNSYNCHRONIZED],
synchronizedt=null,
gw5GCorpOrgCode=null,
regionID=null,
regionExtId=null,
id=@Long[5079],
name=@String[鐜伴噾鏀粯],
createDate=@Timestamp[2022-03-30 12:13:49,000],
modifyDate=null,
createOperator=@Operator[com.star.sms.model.system.organization.Operator@710107e1],
modifyOperator=@Operator[com.star.sms.model.system.organization.Operator@7aac7ebc],
remark=null,
operationTerminal=@Terminal[com.star.sms.model.system.organization.Terminal@52b4a78f],
operationSpot=@SaleChannel[com.star.sms.model.system.organization.SaleChannel@5264c7b4],
version=@Integer[0],
],
payChannelType=@PayChannelType[
CARD_SWIP=@PayChannelType[CARD_SWIP],
SCAN_PAY=@PayChannelType[SCAN_PAY],
$VALUES=@PayChannelType[][isEmpty=false;size=2],
name=@String[SCAN_PAY],
ordinal=@Integer[1],
],
payFee=@Money[
MOVE_CODE=@Integer[32],
serialVersionUID=@Long[3256444694312726840],
CENTS=@int[][isEmpty=false;size=4],
HanDigiStr=@String[][isEmpty=false;size=10],
HanDiviStr=@String[][isEmpty=false;size=24],
ZERO=@Money[錕? 0.00],
CURRENCY_SYMBOL_POSTFIX=@String[ ],
amount=@Long[9900],
currency=@Currency[CNY],
currentCurrency=@Currency[CNY],
],
payDate=@Date[2022-12-26 00:00:00,000],
tradeNo=@String[202299042],
outTradeNo=@String[221226135609002605],
barCode=null,
buyerId=null,
thirdPayStatus=@PayStatus[
SUCCESS=@PayStatus[SUCCESS],
FAIL=@PayStatus[FAIL],
UNKOWN=@PayStatus[UNKOWN],
CANCEL=@PayStatus[CANCEL],
NOTPAY=@PayStatus[NOTPAY],
$VALUES=@PayStatus[][isEmpty=false;size=5],
name=@String[SUCCESS],
ordinal=@Integer[0],
],
bossPayStatus=@ScanPaymentSign[
UNPAYED=@ScanPaymentSign[UNPAYED],
PAYED=@ScanPaymentSign[PAYED],
$VALUES=@ScanPaymentSign[][isEmpty=false;size=2],
name=@String[UNPAYED],
ordinal=@Integer[0],
],
developOperator=null,
status=@Status[
INVALID=@Status[INVALID],
VALID=@Status[VALID],
$VALUES=@Status[][isEmpty=false;size=2],
name=@String[VALID],
ordinal=@Integer[1],
],
acceptSheetId=null,
resp_code=@String[000000],
resp_desc=@String[璋冪敤鎴愬姛],
order_no=@String[12],
order_pay_status=@String[2],
pay_page_url=null,
scan_url=@String[http://www.baidu.com],
sign_info=@String[hhggr],
business_id=null,
gw_pos_trade_no=null,
prov_mer_order_no=null,
posPayStatus=null,
posRespDesc=null,
payNo=@String[13214],
id=@Long[898],
name=null,
createDate=@Timestamp[2022-12-26 13:56:09,981],
modifyDate=@Timestamp[2022-12-26 13:56:10,000],
createOperator=@Operator[
PROPERTY_USERNAME=@String[code],
PROPERTY_PASSWORD=@String[password],
DYNAMIC_PASSWORD=@String[dynamicPassword],
serialVersionUID=@Long[3256443599062578999],
code=@String[0147],
password=null,
dynamicPassword=null,
type=@OperatorType[COMMON_OP],
tel=null,
familyAddress=null,
email=null,
validDateRange=null,
sex=null,
identityCard=null,
education=null,
department=@Department[com.star.sms.model.system.organization.Department@3],
level=@OperationLevel[com.star.sms.model.system.privilege.OperationLevel@1],
address=@Address[com.star.sms.model.system.resource.Address@1],
rayNode=null,
saleAreas=@ArrayList[isEmpty=false;size=13],
roles=@ArrayList[isEmpty=false;size=16],
saleChannels=@ArrayList[isEmpty=false;size=15],
channelBindings=@ArrayList[isEmpty=false;size=15],
segment=null,
synchronizeStatus=null,
synchronizedt=null,
gw5GOperCode=@String[OPER0001],
storeHouses=null,
terminals=null,
mustBindTerminal=@Boolean[false],
status=null,
selectedSaleChannel=null,
dynamicPasswordCheck=null,
isVIPOperator=@Boolean[false],
passwordLastModifyDate=@Date[2021-11-02 00:00:00,000],
imei=null,
jobType=null,
id=@Long[93],
name=@String[huangzw],
createDate=null,
modifyDate=null,
createOperator=null,
modifyOperator=null,
remark=null,
operationTerminal=null,
operationSpot=null,
version=@Integer[0],
],
//略

version=@Integer[0],
],
]