<output id="qn6qe"></output>

    1. <output id="qn6qe"><tt id="qn6qe"></tt></output>
    2. <strike id="qn6qe"></strike>

      亚洲 日本 欧洲 欧美 视频,日韩中文字幕有码av,一本一道av中文字幕无码,国产线播放免费人成视频播放,人妻少妇偷人无码视频,日夜啪啪一区二区三区,国产尤物精品自在拍视频首页,久热这里只有精品12

      優(yōu)先級(jí)反轉(zhuǎn)那些事兒

      作者:崔曉兵

      從一個(gè)線上問(wèn)題說(shuō)起

      最近在線上遇到了一些[HMDConfigManager remoteConfigWithAppID:]卡死

      初步分析

      觀察了下主線程堆棧,用到的鎖是讀寫(xiě)鎖

      圖片

      隨后又去翻了下持有著鎖的子線程,有各種各樣的情況,且基本都處于正常的執(zhí)行狀態(tài),例如有的處于打開(kāi)文件狀態(tài),有的處于read狀態(tài),有的正在執(zhí)行NSUserDefaults的方法…圖片圖片圖片通過(guò)觀察發(fā)現(xiàn),出問(wèn)題的線程都有QOS:BACKGROUND標(biāo)記。整體看起來(lái)持有鎖的子線程仍然在執(zhí)行,只是留給主線程的時(shí)間不夠了。為什么這些子線程在持有鎖的情況下,需要執(zhí)行這么久,直到主線程的8s卡死?一種情況就是真的如此耗時(shí),另一種則是出現(xiàn)了優(yōu)先級(jí)反轉(zhuǎn)。

      解決辦法

      在這個(gè)案例里面,持有讀寫(xiě)鎖且優(yōu)先級(jí)低的線程遲遲得不到調(diào)度(又或者得到調(diào)度的時(shí)候又被搶占了,或者得到調(diào)度的時(shí)候時(shí)間已然不夠了),而具有高優(yōu)先級(jí)的線程由于拿不到讀寫(xiě)鎖,一直被阻塞,所以互相死鎖。iOS8之后引入了QualityOfService的概念,類似于線程的優(yōu)先級(jí),設(shè)置不同的QualityOfService的值后系統(tǒng)會(huì)分配不同的CPU時(shí)間、網(wǎng)絡(luò)資源和硬盤(pán)資源等,因此我們可以通過(guò)這個(gè)設(shè)置隊(duì)列的優(yōu)先級(jí) 。

      方案一:去除對(duì)NSOperationQueue的優(yōu)先級(jí)設(shè)置

      在 Threading Programming Guide 文檔中,蘋(píng)果給出了提示:

      Important: It is generally a good idea to leave the priorities of your threads at their default values. Increasing the priorities of some threads also increases the likelihood of starvation among lower-priority threads. If your application contains high-priority and low-priority threads that must interact with each other, the starvation of lower-priority threads may block other threads and create performance bottlenecks.

      蘋(píng)果的建議是不要隨意修改線程的優(yōu)先級(jí),尤其是這些高低優(yōu)先級(jí)線程之間存在臨界資源競(jìng)爭(zhēng)的情況。所以刪除相關(guān)優(yōu)先級(jí)設(shè)置代碼即可解決問(wèn)題。

      方案二:臨時(shí)修改線程優(yōu)先級(jí)

      在 pthread_rwlock_rdlock(3pthread) 發(fā)現(xiàn)了如下提示:

      Realtime applications may encounter priority inversion when using read-write locks. The problem occurs when a high priority thread “l(fā)ocks” a read-write lock that is about to be “unlocked” by a low priority thread, but the low priority thread is preempted by a medium priority thread. This scenario leads to priority inversion; a high priority thread is blocked by lower priority threads for an unlimited period of time. During system design, realtime programmers must take into account the possibility of this kind of priority inversion. They can deal with it in a number of ways, such as by having critical sections that are guarded by read-write locks execute at a high priority, so that a thread cannot be preempted while executing in its critical section.

      盡管針對(duì)的是實(shí)時(shí)系統(tǒng),但是還是有一些啟示和幫助。按照提示,對(duì)有問(wèn)題的代碼進(jìn)行了修改:在線程通過(guò)pthread_rwlock_wrlock拿到_rwlock的時(shí)候,臨時(shí)提升其優(yōu)先級(jí),在釋放_rwlock之后,恢復(fù)其原先的優(yōu)先級(jí)

      - (id)remoteConfigWithAppID:(NSString *)appID
      {
          .......
          pthread_rwlock_rdlock(&_rwlock);
          HMDHeimdallrConfig *result = ....... // get existing config
          pthread_rwlock_unlock(&_rwlock);
          
          if(result == nil) {
              result = [[HMDHeimdallrConfig alloc] init]; // make a new config
              pthread_rwlock_wrlock(&_rwlock);
              
              qos_class_t oldQos = qos_class_self();
              BOOL needRecover = NO;
              
              // 臨時(shí)提升線程優(yōu)先級(jí)
              if (_enablePriorityInversionProtection && oldQos < QOS_CLASS_USER_INTERACTIVE) {
                  int ret = pthread_set_qos_class_self_np(QOS_CLASS_USER_INTERACTIVE, 0);
                  needRecover = (ret == 0);
              }
                  
              ......
      
              pthread_rwlock_unlock(&_rwlock);
              
              // 恢復(fù)線程優(yōu)先級(jí)
              if (_enablePriorityInversionProtection && needRecover) {
                  pthread_set_qos_class_self_np(oldQos, 0);
              }
          }
          
          return result;
      }

      值得注意的是,這里只能使用pthreadapiNSThread提供的API是不可行的

      Demo 驗(yàn)證

      為了驗(yàn)證上述的手動(dòng)調(diào)整線程優(yōu)先級(jí)是否有一定的效果,這里通過(guò)demo進(jìn)行本地實(shí)驗(yàn):定義了2000個(gè)operation(目的是為了CPU繁忙),優(yōu)先級(jí)設(shè)置NSQualityOfServiceUserInitiated,且對(duì)其中可以被100整除的operation的優(yōu)先級(jí)調(diào)整為NSQualityOfServiceBackground,在每個(gè)operation執(zhí)行相同的耗時(shí)任務(wù),然后對(duì)這被選中的10個(gè)operation進(jìn)行耗時(shí)統(tǒng)計(jì)。

      for (int j = 0; j < 2000; ++j) {
          NSOperationQueue *operation = [[NSOperationQueue alloc] init];
          operation.maxConcurrentOperationCount = 1;
          operation.qualityOfService = NSQualityOfServiceUserInitiated;
          
          // 模塊1
          // if (j % 100 == 0) {
          //    operation.qualityOfService = NSQualityOfServiceBackground;
          // }
          // 模塊1
          
          [operation addOperationWithBlock:^{
              // 模塊2
              // qos_class_t oldQos = qos_class_self();
              // pthread_set_qos_class_self_np(QOS_CLASS_USER_INITIATED, 0);
              // 模塊2
              
              NSTimeInterval start = CFAbsoluteTimeGetCurrent();
              double sum = 0;
              for (int i = 0; i < 100000; ++i) {
                  sum += sin(i) + cos(i) + sin(i*2) + cos(i*2);
              }
              start = CFAbsoluteTimeGetCurrent() - start;
              if (j % 100 == 0) {
                  printf("%.8f\n", start * 1000);
              }
              
              // 模塊2
              // pthread_set_qos_class_self_np(oldQos, 0);
              // 模塊2
          }];
      }

      統(tǒng)計(jì)信息如下圖所示

      A B C
      (注釋模塊1和模塊2代碼) (只打開(kāi)模塊1代碼) (同時(shí)打開(kāi)模塊1和模塊2代碼)
      11.8190561 94.70210189 15.04005137

      可以看到

      1. 正常情況下,每個(gè)任務(wù)的平均耗時(shí)為:11.8190561;
      2. 當(dāng)operation被設(shè)置為低優(yōu)先級(jí)時(shí),其耗時(shí)大幅度提升為:94.70210189;
      3. 當(dāng)operation被設(shè)置為低優(yōu)先級(jí)時(shí),又在Block中手動(dòng)恢復(fù)其原有的優(yōu)先級(jí),其耗時(shí)已經(jīng)大幅度降低:15.04005137( 耗時(shí)比正常情況高,大家可以思考下為什么)

      通過(guò)Demo可以發(fā)現(xiàn),通過(guò)手動(dòng)調(diào)整其優(yōu)先級(jí),低優(yōu)先級(jí)任務(wù)的整體耗時(shí)得到大幅度的降低,這樣在持有鎖的情況下,可以減少對(duì)主線程的阻塞時(shí)間。

      上線效果

      圖片

      該問(wèn)題的驗(yàn)證過(guò)程分為2個(gè)階段:

      1. 第一個(gè)階段如第1個(gè)紅框所示,從36號(hào)開(kāi)始在版本19.7上有較大幅度的下降,主要原因:堆棧中被等待的隊(duì)列信息由QOS:BACKGROUND變?yōu)榱?code>com.apple.root.default-qos,隊(duì)列的優(yōu)先級(jí)從QOS_CLASS_BACKGROUND提升為QOS_CLASS_DEFAULT,相當(dāng)于實(shí)施了方案一,使用了默認(rèn)優(yōu)先級(jí)。
      2. 第二個(gè)階段如第2個(gè)紅框所示,從424號(hào)在版本20.3上開(kāi)始驗(yàn)證。目前看起來(lái)效果暫時(shí)不明顯,推測(cè)一個(gè)主要原因是:demo中是把優(yōu)先級(jí)從QOS_CLASS_BACKGROUND提升為QOS_CLASS_USER_INITIATED,而線上相當(dāng)于把隊(duì)列的優(yōu)先級(jí)從默認(rèn)的優(yōu)先級(jí)QOS_CLASS_DEFAULT提升為QOS_CLASS_USER_INITIATED所以相對(duì)來(lái)說(shuō),線上的提升相對(duì)有限。
        1. QOS_CLASS_BACKGROUNDMach層級(jí)優(yōu)先級(jí)數(shù)是4;
        2. QOS_CLASS_DEFAULTMach層級(jí)優(yōu)先級(jí)數(shù)是31;
        3. QOS_CLASS_USER_INITIATEDMach層級(jí)優(yōu)先級(jí)數(shù)是37;

      深刻理解優(yōu)先級(jí)反轉(zhuǎn)

      那么是否所有鎖都需要像上文一樣,手動(dòng)提升持有鎖的線程優(yōu)先級(jí)?系統(tǒng)是否會(huì)自動(dòng)調(diào)整線程的優(yōu)先級(jí)?如果有這樣的機(jī)制,是否可以覆蓋所有的鎖?要理解這些問(wèn)題,需要深刻認(rèn)識(shí)優(yōu)先級(jí)反轉(zhuǎn)。

      什么是優(yōu)先級(jí)反轉(zhuǎn)?

      優(yōu)先級(jí)反轉(zhuǎn),是指某同步資源被較低優(yōu)先級(jí)的進(jìn)程/線程所擁有,較高優(yōu)先級(jí)的進(jìn)程/線程競(jìng)爭(zhēng)該同步資源未獲得該資源,而使得較高優(yōu)先級(jí)進(jìn)程/線程反而推遲被調(diào)度執(zhí)行的現(xiàn)象。根據(jù)阻塞類型的不同,優(yōu)先級(jí)反轉(zhuǎn)又被分為Bounded priority inversionUnbounded priority inversion。這里借助 Introduction to RTOS - Solution to Part 11 (Priority Inversion) 的圖進(jìn)行示意。

      Bounded priority inversion

      如圖所示,高優(yōu)先級(jí)任務(wù)(Task H)被持有鎖的低優(yōu)先級(jí)任務(wù)(Task L)阻塞,由于阻塞的時(shí)間取決于低優(yōu)先級(jí)任務(wù)在臨界區(qū)的時(shí)間(持有鎖的時(shí)間),所以被稱為bounded priority inversion。只要Task L一直持有鎖,Task H就會(huì)一直被阻塞,低優(yōu)先級(jí)的任務(wù)運(yùn)行在高優(yōu)先級(jí)任務(wù)的前面,優(yōu)先級(jí)被反轉(zhuǎn)。

      這里的任務(wù)也可以理解為線程

      圖片

      Unbounded priority inversion

      Task L持有鎖的情況下,如果有一個(gè)中間優(yōu)先級(jí)的任務(wù)(Task M)打斷了Task L,前面的bounded就會(huì)變?yōu)?code>unbounded,因?yàn)?code>Task M只要搶占了Task LCPU,就可能會(huì)阻塞Task H任意多的時(shí)間(Task M可能不止1個(gè))

      圖片

      優(yōu)先級(jí)反轉(zhuǎn)常規(guī)解決思路

      目前解決Unbounded priority inversion2種方法:一種被稱作優(yōu)先權(quán)極限(priority ceiling protocol),另一種被稱作優(yōu)先級(jí)繼承(priority inheritance)。

      Priority ceiling protocol

      在優(yōu)先權(quán)極限方案中,系統(tǒng)把每一個(gè)臨界資源與1個(gè)極限優(yōu)先權(quán)相關(guān)聯(lián)。當(dāng)1個(gè)任務(wù)進(jìn)入臨界區(qū)時(shí),系統(tǒng)便把這個(gè)極限優(yōu)先權(quán)傳遞給這個(gè)任務(wù),使得這個(gè)任務(wù)的優(yōu)先權(quán)最高;當(dāng)這個(gè)任務(wù)退出臨界區(qū)后,系統(tǒng)立即把它的優(yōu)先權(quán)恢復(fù)正常,從而保證系統(tǒng)不會(huì)出現(xiàn)優(yōu)先權(quán)反轉(zhuǎn)的情況。該極限優(yōu)先權(quán)的值是由所有需要該臨界資源的任務(wù)的最大優(yōu)先級(jí)來(lái)決定的。

      如圖所示,鎖的極限優(yōu)先權(quán)是3。當(dāng)Task L持有鎖的時(shí)候,它的優(yōu)先級(jí)將會(huì)被提升到3,和Task H一樣的優(yōu)先級(jí)。這樣就可以阻止Task M(優(yōu)先級(jí)是2)的運(yùn)行,直到Task LTask H不再需要該鎖。

      圖片

      Priority inheritance

      在優(yōu)先級(jí)繼承方案中,大致原理是:高優(yōu)先級(jí)任務(wù)在嘗試獲取鎖的時(shí)候,如果該鎖正好被低優(yōu)先級(jí)任務(wù)持有,此時(shí)會(huì)臨時(shí)把高優(yōu)先級(jí)線程的優(yōu)先級(jí)轉(zhuǎn)移給擁有鎖的低優(yōu)先級(jí)線程,使低優(yōu)先級(jí)線程能更快的執(zhí)行并釋放同步資源,釋放同步資源后再恢復(fù)其原來(lái)的優(yōu)先級(jí)。

      圖片

      priority ceiling protocolpriority inheritance都會(huì)在釋放鎖的時(shí)候,恢復(fù)低優(yōu)先級(jí)任務(wù)的優(yōu)先級(jí)。同時(shí)要注意,以上2種方法只能阻止Unbounded priority inversion,而無(wú)法阻止Bounded priority inversionTask H必須等待Task L執(zhí)行完畢才能執(zhí)行,這個(gè)反轉(zhuǎn)是無(wú)法避免的)。

      可以通過(guò)以下幾種發(fā)生來(lái)避免或者轉(zhuǎn)移Bounded priority inversion

      1. 減少臨界區(qū)的執(zhí)行時(shí)間,減少Bounded priority inversion的反轉(zhuǎn)耗時(shí);
      2. 避免使用會(huì)阻塞高優(yōu)先級(jí)任務(wù)的臨界區(qū)資源;
      3. 專門(mén)使用一個(gè)隊(duì)列來(lái)管理資源,避免使用鎖。

      優(yōu)先級(jí)繼承必須是可傳遞的。舉個(gè)栗子:當(dāng)T1阻塞在被T2持有的資源上,而T2又阻塞在T3持有的一個(gè)資源上。如果T1的優(yōu)先級(jí)高于T2T3的優(yōu)先級(jí),T3必須通過(guò)T2繼承T1的優(yōu)先級(jí)。否則,如果另外一個(gè)優(yōu)先級(jí)高于T2T3,小于T1的線程T4,將搶占T3,引發(fā)相對(duì)于T1的優(yōu)先級(jí)反轉(zhuǎn)。因此,線程所繼承的優(yōu)先級(jí)必須是直接或者間接阻塞的線程的最高優(yōu)先級(jí)。

      如何避免優(yōu)先級(jí)反轉(zhuǎn)?

      QoS 傳遞

      iOS 系統(tǒng)主要使用以下兩種機(jī)制來(lái)在不同線程(或 queue)間傳遞 QoS

      • 機(jī)制1:dispatch_async
        • dispatch_async() automatically propagates the QoS from the calling thread, though it will translate User Interactive to User Initiated to avoid assigning that priority to non-main threads.
        • Captured at time of block submission, translate user interactive to user initiated. Used if destination queue does not have a QoS and does not lower the QoS (ex dispatch_async back to the main thread)
      • 機(jī)制2:基于 XPC 的進(jìn)程間通信(IPC

      系統(tǒng)的 QoS 傳遞規(guī)則比較復(fù)雜,主要參考以下信息:

      • 當(dāng)前線程的 QoS
      • 如果是使用 dispatch_block_create() 方法生成的 dispatch_block,則考慮生成 block 時(shí)所調(diào)用的參數(shù)
      • dispatch_async 或 IPC 的目標(biāo) queue 或線程的 QoS

      調(diào)度程序會(huì)根據(jù)這些信息決定 block 以什么優(yōu)先級(jí)運(yùn)行。

      1. 如果沒(méi)有其他線程同步地等待此 block,則 block 就按上面所說(shuō)的優(yōu)先級(jí)來(lái)運(yùn)行。
      2. 如果出現(xiàn)了線程間同步等待的情況,則調(diào)度程序會(huì)根據(jù)情況調(diào)整線程的運(yùn)行優(yōu)先級(jí)。

      如何觸發(fā)優(yōu)先級(jí)反轉(zhuǎn)避免機(jī)制?

      如果當(dāng)前線程因等待某線程(線程1)上正在進(jìn)行的操作(如 block1)而受阻,而系統(tǒng)知道 block1 所在的目標(biāo)線程(owner),系統(tǒng)會(huì)通過(guò)提高相關(guān)線程的優(yōu)先級(jí)來(lái)解決優(yōu)先級(jí)反轉(zhuǎn)的問(wèn)題。反之如果系統(tǒng)不知道 block1 所在目標(biāo)線程,則無(wú)法知道應(yīng)該提高誰(shuí)的優(yōu)先級(jí),也就無(wú)法解決反轉(zhuǎn)問(wèn)題;

      記錄了持有者信息(owner)的系統(tǒng) API 如下:

      1. pthread mutexos_unfair_lock、以及基于這二者實(shí)現(xiàn)的上層 API
        1. dispatch_once 的實(shí)現(xiàn)是基于 os_unfair_lock 的
        2. NSLockNSRecursiveLock@synchronized 等的實(shí)現(xiàn)是基于 pthread mutex
      2. dispatch_syncdispatch_wait
      3. xpc_connection_send_with_message_sync

      使用以上這些 API 能夠在發(fā)生優(yōu)先級(jí)反轉(zhuǎn)時(shí)使系統(tǒng)啟用優(yōu)先級(jí)反轉(zhuǎn)避免機(jī)制

      基礎(chǔ)API驗(yàn)證

      接下來(lái)對(duì)前文提到的各種「基礎(chǔ)系統(tǒng)API」進(jìn)行驗(yàn)證

      測(cè)試驗(yàn)證環(huán)境:模擬器 iOS15.2

      pthread mutex

      pthread mutex的數(shù)據(jù)結(jié)構(gòu)pthread_mutex_s其中有一個(gè)m_tid字段,專門(mén)來(lái)記錄持有該鎖的線程Id

      // types_internal.h
      struct pthread_mutex_s {
              long sig;
              _pthread_lock lock;
              union {
                      uint32_t value;
                      struct pthread_mutex_options_s options;
              } mtxopts;
              int16_t prioceiling;
              int16_t priority;
      #if defined(__LP64__)
              uint32_t _pad;
      #endif
              union {
                      struct {
                              uint32_t m_tid[2]; // thread id of thread that has mutex locked
                              uint32_t m_seq[2]; // mutex sequence id
                              uint32_t m_mis[2]; // for misaligned locks m_tid/m_seq will span into here
                      } psynch;
                      struct _pthread_mutex_ulock_s ulock;
              };
      #if defined(__LP64__)
              uint32_t _reserved[4];
      #else
              uint32_t _reserved[1];
      #endif
      };

      代碼來(lái)驗(yàn)證一下:線程優(yōu)先級(jí)是否會(huì)被提升?

      // printThreadPriority用來(lái)打印線程的優(yōu)先級(jí)信息
      void printThreadPriority() {
        thread_t cur_thread = mach_thread_self();
        mach_port_deallocate(mach_task_self(), cur_thread);
        mach_msg_type_number_t thread_info_count = THREAD_INFO_MAX;
        thread_info_data_t thinfo;
        kern_return_t kr = thread_info(cur_thread, THREAD_EXTENDED_INFO, (thread_info_t)thinfo, &thread_info_count);
        if (kr != KERN_SUCCESS) {
          return;
        }
        thread_extended_info_t extend_info = (thread_extended_info_t)thinfo;
        printf("pth_priority: %d, pth_curpri: %d, pth_maxpriority: %d\n", extend_info->pth_priority, extend_info->pth_curpri, extend_info->pth_maxpriority);
      }

      先在子線程上鎖并休眠,然后主線程請(qǐng)求該鎖

      dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0), ^{
        printf("begin : \n");
        printThreadPriority();
        printf("queue before lock \n");
        pthread_mutex_lock(&_lock); //確保 backgroundQueue 先得到鎖
        printf("queue lock \n");
        printThreadPriority();
        dispatch_async(dispatch_get_main_queue(), ^{
          printf("before main lock\n");
          pthread_mutex_lock(&_lock);
          printf("in main lock\n");
          pthread_mutex_unlock(&_lock);
          printf("after main unlock\n");
        });
        sleep(10);
        printThreadPriority();
        printf("queue unlock\n");
        pthread_mutex_unlock(&_lock);
        printf("queue after unlock\n");
      });
      begin : 
      pth_priority: 4, pth_curpri: 4, pth_maxpriority: 63
      queue before lock 
      queue lock 
      pth_priority: 4, pth_curpri: 4, pth_maxpriority: 63
      before main lock
      pth_priority: 47, pth_curpri: 47, pth_maxpriority: 63
      queue unlock
      in main lock
      after main unlock
      queue after unlock

      可以看到,低優(yōu)先級(jí)子線程先持有鎖,當(dāng)時(shí)的優(yōu)先級(jí)為4,而該鎖被主線程請(qǐng)求的時(shí)候,子線程的優(yōu)先級(jí)被提升為47

      os_unfair_lock

      os_unfair_lock用來(lái)替換OSSpinLock,解決優(yōu)先級(jí)反轉(zhuǎn)問(wèn)題。等待os_unfair_lock鎖的線程會(huì)處于休眠狀態(tài),從用戶態(tài)切換到內(nèi)核態(tài),而并非忙等。os_unfair_lock將線程ID保存到了鎖的內(nèi)部,鎖的等待者會(huì)把自己的優(yōu)先級(jí)讓出來(lái),從而避免優(yōu)先級(jí)反轉(zhuǎn)。驗(yàn)證一下:

      dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0), ^{
          printf("begin : \n");
          printThreadPriority();
          printf("queue before lock \n");
          os_unfair_lock_lock(&_unfair_lock); //確保 backgroundQueue 先得到鎖
          printf("queue lock \n");
          printThreadPriority();
          dispatch_async(dispatch_get_main_queue(), ^{
            printf("before main lock\n");
            os_unfair_lock_lock(&_unfair_lock);
            printf("in main lock\n");
            os_unfair_lock_unlock(&_unfair_lock);
            printf("after main unlock\n");
          });
          sleep(10);
          printThreadPriority();
          printf("queue unlock\n");
          os_unfair_lock_unlock(&_unfair_lock);
          printf("queue after unlock\n");
        });
      begin : 
      pth_priority: 4, pth_curpri: 4, pth_maxpriority: 63
      queue before lock 
      queue lock 
      pth_priority: 4, pth_curpri: 4, pth_maxpriority: 63
      before main lock
      pth_priority: 47, pth_curpri: 47, pth_maxpriority: 63
      queue unlock
      in main lock
      after main unlock
      queue after unlock

      結(jié)果和pthread mutex一致

      pthread_rwlock_t

      在 pthread_rwlock_init 有如下提示:

      Caveats: Beware of priority inversion when using read-write locks. A high-priority thread may be blocked waiting on a read-write lock locked by a low-priority thread. The microkernel has no knowledge of read-write locks, and therefore can’t boost the low-priority thread to prevent the priority inversion.

      大意是內(nèi)核不感知讀寫(xiě)鎖,無(wú)法提升低優(yōu)先級(jí)線程的優(yōu)先級(jí),從而無(wú)法避免優(yōu)先級(jí)反轉(zhuǎn)。通過(guò)查詢定義發(fā)現(xiàn):pthread_rwlock_s包含了字段rw_tid,專門(mén)來(lái)記錄持有寫(xiě)鎖的線程,這不由令人好奇:為什么pthread_rwlock_sowner信息卻仍然無(wú)法避免優(yōu)先級(jí)反轉(zhuǎn)?

      struct pthread_rwlock_s {
              long sig;
              _pthread_lock lock;
              uint32_t
                      unused:29,
                      misalign:1,
                      pshared:2;
              uint32_t rw_flags;
      #if defined(__LP64__)
              uint32_t _pad;
      #endif
              uint32_t rw_tid[2]; // thread id of thread that has exclusive (write) lock
              uint32_t rw_seq[4]; // rw sequence id (at 128-bit aligned boundary)
              uint32_t rw_mis[4]; // for misaligned locks rw_seq will span into here
      #if defined(__LP64__)
              uint32_t _reserved[34];
      #else
              uint32_t _reserved[18];
      #endif
      };

      https://news.ycombinator.com/item?id=21751269 鏈接中提到:

      xnu supports priority inheritance through “turnstiles”, a kernel-internal mechani** which is used by default by a number of locking primitives (list at [1]), including normal pthread mutexes (though not read-write locks [2]), as well as the os_unfair_lock API (via the ulock syscalls). With pthread mutexes, you can actually explicitly request priority inheritance by calling pthread_mutexattr_setprotocol [3] with PTHREAD_PRIO_INHERIT; the Apple implementation supports it, but currently ignores the protocol setting and just gives all mutexes priority inheritance.

      大意是:XNU使用turnstiles內(nèi)核機(jī)制進(jìn)行優(yōu)先級(jí)繼承,這種機(jī)制被應(yīng)用在pthread mutexos_unfair_lock上。

      順藤摸瓜,在ksyn_wait方法中找到了_kwq_use_turnstile的調(diào)用,其中的注釋對(duì)讀寫(xiě)鎖解釋的比較委婉,添加了at least sometimes

      pthread mutexes and rwlocks both (at least sometimes)  know their owner and can use turnstiles. Otherwise, we pass NULL as the tstore to the shims so they wait on the global waitq.

      // libpthread/kern/kern_synch.c
      int
      ksyn_wait(ksyn_wait_queue_t kwq, kwq_queue_type_t kqi, uint32_t lockseq,
                      int fit, uint64_t abstime, uint16_t kwe_flags,
                      thread_continue_t continuation, block_hint_t block_hint)
      {
              thread_t th = current_thread();
              uthread_t uth = pthread_kern->get_bsdthread_info(th);
              struct turnstile **tstore = NULL;
              int res;
      
              assert(continuation != THREAD_CONTINUE_NULL);
      
              ksyn_waitq_element_t kwe = pthread_kern->uthread_get_uukwe(uth);
              bzero(kwe, sizeof(*kwe));
              kwe->kwe_count = 1;
              kwe->kwe_lockseq = lockseq & PTHRW_COUNT_MASK;
              kwe->kwe_state = KWE_THREAD_INWAIT;
              kwe->kwe_uth = uth;
              kwe->kwe_thread = th;
              kwe->kwe_flags = kwe_flags;
      
              res = ksyn_queue_insert(kwq, kqi, kwe, lockseq, fit);
              if (res != 0) {
                      //panic("psynch_rw_wrlock: failed to enqueue\n"); // XXX                ksyn_wqunlock(kwq);
                      return res;
              }
      
              PTHREAD_TRACE(psynch_mutex_kwqwait, kwq->kw_addr, kwq->kw_inqueue,
                              kwq->kw_prepost.count, kwq->kw_intr.count);
      
              if (_kwq_use_turnstile(kwq)) {
                      // pthread mutexes and rwlocks both (at least sometimes) know their                
                      // owner and can use turnstiles. Otherwise, we pass NULL as the                
                      // tstore to the shims so they wait on the global waitq.                
                      tstore = &kwq->kw_turnstile;
              }
              ......
      }

      再去查看_kwq_use_turnstile的定義,代碼還是很誠(chéng)實(shí)的,只有在KSYN_WQTYPE_MTX才會(huì)啟用turnstile進(jìn)行優(yōu)先級(jí)反轉(zhuǎn)保護(hù),而讀寫(xiě)鎖的類型為KSYN_WQTYPE_RWLOCK,這說(shuō)明讀寫(xiě)鎖不會(huì)使用_kwq_use_turnstile,所以無(wú)法避免優(yōu)先級(jí)反轉(zhuǎn)。

      #define KSYN_WQTYPE_MTX         0x01
      #define KSYN_WQTYPE_CVAR        0x02
      #define KSYN_WQTYPE_RWLOCK      0x04
      #define KSYN_WQTYPE_SEMA        0x08
      
      static inline bool
      _kwq_use_turnstile(ksyn_wait_queue_t kwq)
      {
              // If we had writer-owner information from the
              // rwlock then we could use the turnstile to push on it. For now, only
              // plain mutexes use it.
              return (_kwq_type(kwq) == KSYN_WQTYPE_MTX);
      }

      另外在_pthread_find_owner也可以看到,讀寫(xiě)鎖的owner0

      void
      _pthread_find_owner(thread_t thread,
                      struct stackshot_thread_waitinfo * waitinfo)
      {
              ksyn_wait_queue_t kwq = _pthread_get_thread_kwq(thread);
              switch (waitinfo->wait_type) {
                      case kThreadWaitPThreadMutex:
                              assert((kwq->kw_type & KSYN_WQTYPE_MASK) == KSYN_WQTYPE_MTX);
                              waitinfo->owner  = thread_tid(kwq->kw_owner);
                              waitinfo->context = kwq->kw_addr;
                              break;
                      /* Owner of rwlock not stored in kernel space due to races. Punt
                       * and hope that the userspace address is helpful enough. */
                      case kThreadWaitPThreadRWLockRead:
                      case kThreadWaitPThreadRWLockWrite:
                              assert((kwq->kw_type & KSYN_WQTYPE_MASK) == KSYN_WQTYPE_RWLOCK);
                              waitinfo->owner  = 0;
                              waitinfo->context = kwq->kw_addr;
                              break;
                      /* Condvars don't have owners, so just give the userspace address. */
                      case kThreadWaitPThreadCondVar:
                              assert((kwq->kw_type & KSYN_WQTYPE_MASK) == KSYN_WQTYPE_CVAR);
                              waitinfo->owner  = 0;
                              waitinfo->context = kwq->kw_addr;
                              break;
                      case kThreadWaitNone:
                      default:
                              waitinfo->owner = 0;
                              waitinfo->context = 0;
                              break;
              }
      }

      把鎖更換為讀寫(xiě)鎖,驗(yàn)證一下前面的理論是否正確:

      pthread_rwlock_init(&_rwlock, NULL);
      dispatch_async(dispatch_get_global_queue(DISPATCH_QUEUE_PRIORITY_BACKGROUND, 0), ^{
        printf("begin : \n");
        printThreadPriority();
        printf("queue before lock \n");
        pthread_rwlock_rdlock(&_rwlock); //確保 backgroundQueue 先得到鎖
        printf("queue lock \n");
        printThreadPriority();
        dispatch_async(dispatch_get_main_queue(), ^{
          printf("before main lock\n");
          pthread_rwlock_wrlock(&_rwlock);
          printf("in main lock\n");
          pthread_rwlock_unlock(&_rwlock);
          printf("after main unlock\n");
        });
        sleep(10);
        printThreadPriority();
        printf("queue unlock\n");
        pthread_rwlock_unlock(&_rwlock);
        printf("queue after unlock\n");
      });
      begin : 
      pth_priority: 4, pth_curpri: 4, pth_maxpriority: 63
      queue before lock 
      queue lock 
      pth_priority: 4, pth_curpri: 4, pth_maxpriority: 63
      before main lock
      pth_priority: 4, pth_curpri: 4, pth_maxpriority: 63
      queue unlock
      queue after unlock
      in main lock
      after main unlock

      可以看到讀寫(xiě)鎖不會(huì)發(fā)生優(yōu)先級(jí)提升

      dispatch_sync

      這個(gè)API都比較熟悉了,這里直接驗(yàn)證:

      // 當(dāng)前線程為主線程
      dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(DISPATCH_QUEUE_SERIAL, QOS_CLASS_BACKGROUND, 0);
      _queue = dispatch_queue_create("com.demo.test", qosAttribute);
      printThreadPriority();
      dispatch_async(_queue, ^{
          printf("dispatch_async before dispatch_sync : \n");
          printThreadPriority();
      });
      dispatch_sync(_queue, ^{
          printf("dispatch_sync: \n");
          printThreadPriority();
      });
      dispatch_async(_queue, ^{
          printf("dispatch_async after dispatch_sync: \n");
          printThreadPriority();
      });
      pth_priority: 47, pth_curpri: 47, pth_maxpriority: 63 
      dispatch_async before dispatch_sync : 
      pth_priority: 47, pth_curpri: 47, pth_maxpriority: 63
      dispatch_sync: 
      pth_priority: 47, pth_curpri: 47, pth_maxpriority: 63
      dispatch_async after dispatch_sync: 
      pth_priority: 4, pth_curpri: 4, pth_maxpriority: 63

      _queue是一個(gè)低優(yōu)先級(jí)隊(duì)列(QOS_CLASS_BACKGROUND),可以看到dispatch_sync調(diào)用壓入隊(duì)列的任務(wù),以及在這之前dispatch_async壓入的任務(wù),都被提升到較高的優(yōu)先級(jí)47(和主線程一致),而最后一個(gè)dispatch_async的任務(wù)則以優(yōu)先級(jí)4來(lái)執(zhí)行。

      dispatch_wait

      // 當(dāng)前線程為主線程
      dispatch_queue_attr_t qosAttribute = dispatch_queue_attr_make_with_qos_class(DISPATCH_QUEUE_SERIAL, QOS_CLASS_BACKGROUND, 0);
      _queue = dispatch_queue_create("com.demo.test", qosAttribute);
      printf("main thread\n");
      printThreadPriority();
      dispatch_block_t block = dispatch_block_create(DISPATCH_BLOCK_INHERIT_QOS_CLASS, ^{
          printf("sub thread\n");
          sleep(2);
          printThreadPriority();
      });
      dispatch_async(_queue, block);
      dispatch_wait(block, DISPATCH_TIME_FOREVER);

      _queue是一個(gè)低優(yōu)先級(jí)隊(duì)列(QOS_CLASS_BACKGROUND),當(dāng)在當(dāng)前主線程使用dispatch_wait進(jìn)行等待時(shí),輸出如下,低優(yōu)先級(jí)的任務(wù)被提升到優(yōu)先級(jí)47

      main thread
      pth_priority: 47, pth_curpri: 47, pth_maxpriority: 63
      sub thread
      pth_priority: 47, pth_curpri: 47, pth_maxpriority: 63

      而如果將dispatch_wait(block, DISPATCH_TIME_FOREVER)注釋掉之后,輸出如下:

      main thread
      pth_priority: 47, pth_curpri: 47, pth_maxpriority: 63
      sub thread
      pth_priority: 4, pth_curpri: 4, pth_maxpriority: 63

      值得注意的是,dispatch_wait是一個(gè)宏(C11的泛型),或者是一個(gè)入口函數(shù),它可以接受dispatch_block_tdispatch_group_tdispatch_semaphore_t 3種類型的參數(shù),但是這里的具體含義應(yīng)該是指dispatch_block_wait,只有dispatch_block_wait會(huì)調(diào)整優(yōu)先級(jí),避免優(yōu)先級(jí)反轉(zhuǎn)。

      intptr_t
      dispatch_wait(void *object, dispatch_time_t timeout);
      #if __has_extension(c_generic_selections)
      #define dispatch_wait(object, timeout) \
                      _Generic((object), \
                              dispatch_block_t:dispatch_block_wait, \
                              dispatch_group_t:dispatch_group_wait, \
                              dispatch_semaphore_t:dispatch_semaphore_wait \
                      )((object),(timeout))
      #endif

      神秘的信號(hào)量

      dispatch_semaphore

      之前對(duì)dispatch_semaphore的認(rèn)知非常淺薄,經(jīng)常把二值信號(hào)量和互斥鎖劃等號(hào)。但是通過(guò)調(diào)研后發(fā)現(xiàn):dispatch_semaphore 沒(méi)有 QoS 的概念,沒(méi)有記錄當(dāng)前持有信號(hào)量的線程(owner),所以有高優(yōu)先級(jí)的線程在等待鎖時(shí),內(nèi)核無(wú)法知道該提高哪個(gè)線程的調(diào)試優(yōu)先級(jí)(QoS)。如果鎖持有者優(yōu)先級(jí)比其他線程低,高優(yōu)先級(jí)的等待線程將一直等待。Mutex vs Semaphore: What’s the Difference? 一文詳細(xì)比對(duì)了MutexSemaphore之間的區(qū)別。

      Semaphores are for signaling (sames a condition variables, events) while mutexes are for mutual exclusion. Technically, you can also use semaphores for mutual exclusion (a mutex can be thought as a binary semaphore) but you really shouldn’t.Right, but libdispatch doesn’t have a mutex. It has semaphores and queues. So if you’re trying to use libdispatch and you don’t want the closure-based aspect of queues, you might be tempted to use a semaphore instead. Don’t do that, use os_unfair_lock or pthread_mutex (or a higher-level construct like NSLock) instead.

      這些是一些警示,可以看到dispatch_semaphore十分危險(xiǎn),使用需要特別小心。

      這里通過(guò)蘋(píng)果官方提供的demo進(jìn)行解釋:

      __block NSString *taskName = nil;
      dispatch_semaphore_t sema = dispatch_semaphore_create(0); 
      [self.connection.remoteObjectProxy requestCurrentTaskName:^(NSString *task) { 
           taskName = task; 
           dispatch_semaphore_signal(sema); 
      }]; 
      dispatch_semaphore_wait(sema, DISPATCH_TIME_FOREVER); 
      return taskName;
      1. 假設(shè)在主線程執(zhí)行這段代碼,那么當(dāng)前線程的優(yōu)先級(jí)是QOS_CLASS_USER_INTERACTIVE
      2. 由于從主線程進(jìn)行了異步,異步任務(wù)隊(duì)列的QoS將會(huì)被提升為QOS_CLASS_USER_INITIATED
      3. 主線程被信號(hào)量sema阻塞,而負(fù)責(zé)釋放該信號(hào)量的異步任務(wù)的優(yōu)先級(jí)QOS_CLASS_USER_INITIATED低于主線程的優(yōu)先級(jí)QOS_CLASS_USER_INTERACTIVE,因此可能會(huì)發(fā)生優(yōu)先級(jí)反轉(zhuǎn)。

      值得一提的是,Clang專門(mén)針對(duì)這種情況進(jìn)行了靜態(tài)檢測(cè):

      https://github.com/llvm-mirror/clang/blob/master/lib/StaticAnalyzer/Checkers/GCDAntipatternChecker.cpp

      static auto findGCDAntiPatternWithSemaphore() -> decltype(compoundStmt()) {
      
        const char *SemaphoreBinding = "semaphore_name";
        auto SemaphoreCreateM = callExpr(allOf(
            callsName("dispatch_semaphore_create"),
            hasArgument(0, ignoringParenCasts(integerLiteral(equals(0))))));
      
        auto SemaphoreBindingM = anyOf(
            forEachDescendant(
                varDecl(hasDescendant(SemaphoreCreateM)).bind(SemaphoreBinding)),
            forEachDescendant(binaryOperator(bindAssignmentToDecl(SemaphoreBinding),
                           hasRHS(SemaphoreCreateM))));
      
        auto HasBlockArgumentM = hasAnyArgument(hasType(
                  hasCanonicalType(blockPointerType())
                  ));
      
        auto ArgCallsSignalM = hasAnyArgument(stmt(hasDescendant(callExpr(
                allOf(
                    callsName("dispatch_semaphore_signal"),
                    equalsBoundArgDecl(0, SemaphoreBinding)
                    )))));
      
        auto HasBlockAndCallsSignalM = allOf(HasBlockArgumentM, ArgCallsSignalM);
      
        auto HasBlockCallingSignalM =
          forEachDescendant(
            stmt(anyOf(
              callExpr(HasBlockAndCallsSignalM),
              objcMessageExpr(HasBlockAndCallsSignalM)
                 )));
      
        auto SemaphoreWaitM = forEachDescendant(
          callExpr(
            allOf(
              callsName("dispatch_semaphore_wait"),
              equalsBoundArgDecl(0, SemaphoreBinding)
            )
          ).bind(WarnAtNode));
      
        return compoundStmt(
            SemaphoreBindingM, HasBlockCallingSignalM, SemaphoreWaitM);
      }

      如果想使用該功能,只需要打開(kāi)xcode設(shè)置即可:

      圖片

      另外,dispatch_group 跟 semaphore 類似,在調(diào)用 enter() 方法時(shí),無(wú)法預(yù)知誰(shuí)會(huì)調(diào)用 leave(),所以系統(tǒng)也無(wú)法知道其 owner是誰(shuí),所以同樣不會(huì)有優(yōu)先級(jí)提升的問(wèn)題。

      信號(hào)量卡死現(xiàn)身說(shuō)法

      dispatch_semaphore給筆者的印象非常深刻,之前寫(xiě)過(guò)一段這樣的代碼:使用信號(hào)量在主線程同步等待相機(jī)授權(quán)結(jié)果。

      __block BOOL auth = NO;
      dispatch_semaphore_t semaphore = dispatch_semaphore_create(0);
      [KTAuthorizeService requestAuthorizationWithType:KTPermissionsTypeCamera completionHandler:^(BOOL allow) {
        auth = allow;
        dispatch_semaphore_signal(semaphore);
      }];
      dispatch_semaphore_wait(semaphore, DISPATCH_TIME_FOREVER);

      上線后長(zhǎng)期占據(jù)卡死top1,當(dāng)時(shí)百思不得其解,在深入了解到信號(hào)量無(wú)法避免優(yōu)先級(jí)反轉(zhuǎn)后,終于豁然開(kāi)朗,一掃之前心中的陰霾。這類問(wèn)題一般通過(guò)2種方式來(lái)解決:

      1. 使用同步API
      BOOL auth = [KTAuthorizeService authorizationWithType:KTPermissionsTypeCamera];
      // do something next
      1. 異步回調(diào),不要在當(dāng)前線程等待
      [KTAuthorizeService requestAuthorizationWithType:KTPermissionsTypeCamera completionHandler:^(BOOL allow) {
          BOOL auth = allow;
          // do something next via callback
      }];

      幾個(gè)概念

      turnstile

      前文提到XNU使用turnstile進(jìn)行優(yōu)先級(jí)繼承,這里對(duì)turnstile機(jī)制進(jìn)行簡(jiǎn)單的描述和理解。在XNU內(nèi)核中,存在著大量的同步對(duì)象(例如lck_mtx_t),為了解決優(yōu)先級(jí)反轉(zhuǎn)的問(wèn)題,每個(gè)同步對(duì)象都必須對(duì)應(yīng)一個(gè)分離的數(shù)據(jù)結(jié)構(gòu)來(lái)維護(hù)大量的信息,例如阻塞在這個(gè)同步對(duì)象上的線程隊(duì)列。可以想象一下,如果每個(gè)同步對(duì)象都要分配一個(gè)這樣的數(shù)據(jù)結(jié)構(gòu),將造成極大的內(nèi)存浪費(fèi)。為了解決這個(gè)問(wèn)題,XNU采用了turnstile機(jī)制,一種空間利用率很高的解決方案。該方案的提出依據(jù)是同一個(gè)線程在同一時(shí)刻不能同時(shí)阻塞于多個(gè)同步對(duì)象上。這一事實(shí)允許所有同步對(duì)象只需要保留一個(gè)指向turnstile的指針,且在需要的時(shí)候去分配一個(gè)turnstile即可,而turnstile則包含了操作一個(gè)同步對(duì)象需要的所有信息,例如阻塞線程的隊(duì)列、擁有這個(gè)同步對(duì)象的線程指針。turnstile是從池中動(dòng)態(tài)分配的,這個(gè)池的大小會(huì)隨著系統(tǒng)中已分配的線程數(shù)目增加而增加,所以turnstile總數(shù)將始終低于或等于線程數(shù),這也決定了turnstile的數(shù)目是可控的。turnstile由阻塞在該同步對(duì)象上的第一個(gè)線程負(fù)責(zé)分配,當(dāng)沒(méi)有更多線程阻塞在該同步對(duì)象上,turnstile會(huì)被釋放,回收到池中。turnstile的數(shù)據(jù)結(jié)構(gòu)如下:

      struct turnstile {
          struct waitq                  ts_waitq;              /* waitq embedded in turnstile */
          turnstile_inheritor_t         ts_inheritor;          /* thread/turnstile inheriting the priority (IL, WL) */
          union {
              struct turnstile_list ts_free_turnstiles;    /* turnstile free list (IL) */
              SLIST_ENTRY(turnstile) ts_free_elm;          /* turnstile free list element (IL) */
          };
          struct priority_queue_sched_max ts_inheritor_queue;    /* Queue of turnstile with us as an inheritor (WL) */
          union {
              struct priority_queue_entry_sched ts_inheritor_links;    /* Inheritor queue links */
              struct mpsc_queue_chain   ts_deallocate_link;    /* thread deallocate link */
          };
          SLIST_ENTRY(turnstile)        ts_htable_link;        /* linkage for turnstile in global hash table */
          uintptr_t                     ts_proprietor;         /* hash key lookup turnstile (IL) */
          os_refcnt_t                   ts_refcount;           /* reference count for turnstiles */
          _Atomic uint32_t              ts_type_gencount;      /* gen count used for priority chaining (IL), type of turnstile (IL) */
          uint32_t                      ts_port_ref;           /* number of explicit refs from ports on send turnstile */
          turnstile_update_flags_t      ts_inheritor_flags;    /* flags for turnstile inheritor (IL, WL) */
          uint8_t                       ts_priority;           /* priority of turnstile (WL) */
      
      #if DEVELOPMENT || DEBUG
          uint8_t                       ts_state;              /* current state of turnstile (IL) */
          queue_chain_t                 ts_global_elm;         /* global turnstile chain */
          thread_t                      ts_thread;             /* thread the turnstile is attached to */
          thread_t                      ts_prev_thread;        /* thread the turnstile was attached before donation */
      #endif
      };

      優(yōu)先級(jí)數(shù)值

      在驗(yàn)證環(huán)節(jié)有一些優(yōu)先級(jí)數(shù)值,這里借助「Mac OS? X and iOS Internals 」解釋一下:實(shí)驗(yàn)中涉及到的優(yōu)先級(jí)數(shù)值都是相對(duì)于Mach層而言的,且都是用戶線程數(shù)值

      1. 用戶線程的優(yōu)先級(jí)是0~63;
        1. NSQualityOfServiceBackgroundMach層級(jí)優(yōu)先級(jí)數(shù)是4;
        2. NSQualityOfServiceUtilityMach層級(jí)優(yōu)先級(jí)數(shù)是20;
        3. NSQualityOfServiceDefaultMach層級(jí)優(yōu)先級(jí)數(shù)是31;
        4. NSQualityOfServiceUserInitiatedMach層級(jí)優(yōu)先級(jí)數(shù)是37;
        5. NSQualityOfServiceUserInteractiveMach層級(jí)優(yōu)先級(jí)是47;
      2. 內(nèi)核線程的優(yōu)先級(jí)是80~95;
      3. 實(shí)時(shí)系統(tǒng)線程的優(yōu)先級(jí)是96~127;
      4. 64~79被保留給系統(tǒng)使用;

      圖片

      總結(jié)

      本文主要闡述了優(yōu)先級(jí)反轉(zhuǎn)的一些概念和解決思路,并結(jié)合iOS平臺(tái)的幾種鎖進(jìn)行了詳細(xì)的調(diào)研。通過(guò)深入的理解,可以去規(guī)避一些不必要的優(yōu)先級(jí)反轉(zhuǎn),從而進(jìn)一步避免卡死異常。字節(jié)跳動(dòng) APM團(tuán)隊(duì)也針對(duì)線程的優(yōu)先級(jí)做了監(jiān)控處理,進(jìn)而達(dá)到發(fā)現(xiàn)和預(yù)防優(yōu)先級(jí)反轉(zhuǎn)的目的。

      加入我們

      字節(jié)跳動(dòng) APM 中臺(tái)致力于提升整個(gè)集團(tuán)內(nèi)全系產(chǎn)品的性能和穩(wěn)定性表現(xiàn),技術(shù)棧覆蓋iOS/Android/Server/Web/Hybrid/PC/游戲/小程序等,工作內(nèi)容包括但不限于性能穩(wěn)定性監(jiān)控,問(wèn)題排查,深度優(yōu)化,防劣化等。長(zhǎng)期期望為業(yè)界輸出更多更有建設(shè)性的問(wèn)題發(fā)現(xiàn)和深度優(yōu)化手段。

      歡迎對(duì)字節(jié)APM團(tuán)隊(duì)職位感興趣的同學(xué)投遞簡(jiǎn)歷到郵箱 xushuangqing@bytedance.com 。

      參考文檔

      posted @ 2022-11-28 15:54  字節(jié)跳動(dòng)終端技術(shù)  閱讀(775)  評(píng)論(0)    收藏  舉報(bào)
      主站蜘蛛池模板: 国产一区二区三区黄网| 亚洲一精品一区二区三区| 国产精品小粉嫩在线观看| 国产区图片区小说区亚洲区 | 国产裸体美女视频全黄| 97久久久亚洲综合久久| 欧美成人精品手机在线| 国产91精品一区二区亚洲| 免费吃奶摸下激烈视频| 韩国无码AV片午夜福利| 大乳丰满人妻中文字幕日本| 亚洲区综合区小说区激情区| 午夜国产小视频| 久久中文字幕日韩无码视频| 成人中文在线| 招远市| 欧美黑吊大战白妞| 337P日本欧洲亚洲大胆精品555588| 潜山县| 人妻无码中文专区久久app| 18成人片黄网站www| 久久亚洲精品亚洲人av| 精品乱码一区二区三四五区| 日韩av综合免费在线| 国产欧美日韩免费看AⅤ视频| 97久久精品无码一区二区| 国产一区二区黄色激情片| 开心婷婷五月激情综合社区| 国产精品久久久久久久专区| 精品国产午夜福利在线观看| 日本高清一区免费中文视频| 国产精品毛片av999999| 亚洲成av人片无码不卡播放器| 国产亚洲精品一区二区不卡| 欧美另类videossexo高潮| 看全色黄大黄大色免费久久| 小污女小欲女导航| 国产重口老太和小伙| 一区二区在线观看成人午夜| gogogo高清免费观看| 久久亚洲精品中文字幕波多野结衣|