Bulkdelete 高效,穩(wěn)定刪除上億條數(shù)據(jù)。
以下是一個基于 Entity Framework Extensions 的 BulkDelete 高效刪除海量數(shù)據(jù)的方案,包含性能優(yōu)化、穩(wěn)定性保障和錯誤處理機(jī)制:
csharp
using System;
using System.Linq;
using Z.EntityFramework.Plus;
public class BulkDeleteService
{
private const int BatchSize = 50000; // 根據(jù)服務(wù)器內(nèi)存和網(wǎng)絡(luò)調(diào)整
private const int MaxRetryAttempts = 3;
private readonly TimeSpan RetryDelay = TimeSpan.FromSeconds(5);
public void BulkDeleteRecords(MyDbContext context, DateTime deleteBefore)
{
var query = context.YourEntities
.Where(x => x.CreatedDate < deleteBefore)
.OrderBy(x => x.Id); // 按主鍵排序確保分頁穩(wěn)定
int totalDeleted = 0;
bool hasMoreData = true;
var deleteOptions = new BulkDeleteOptions
{
BatchSize = BatchSize,
UseTransaction = false, // 關(guān)閉事務(wù)提升性能
CommandTimeout = 600, // 適當(dāng)延長超時時間
ExecuteHook = (x) =>
{
// 內(nèi)存優(yōu)化配置
x.BatchTimeout = 60;
x.TrackingEntities = false;
}
};
while (hasMoreData)
{
try
{
// 分頁刪除(使用子查詢確保分頁準(zhǔn)確)
var subQuery = query.Take(BatchSize);
int deletedCount = RetryPolicy(() => subQuery.DeleteFromQuery(deleteOptions));
totalDeleted += deletedCount;
hasMoreData = deletedCount == BatchSize;
Console.WriteLine($"Deleted {totalDeleted:N0} records, LastID: {GetLastProcessedId(context)}");
// 定期釋放資源
if (totalDeleted % (BatchSize * 10) == 0)
{
context.Dispose();
context = new MyDbContext();
query = context.YourEntities.Where(...); // 重建查詢
}
}
catch (Exception ex)
{
LogError(ex, totalDeleted);
throw; // 根據(jù)業(yè)務(wù)需求決定是否終止
}
}
}
private int RetryPolicy(Func<int> action)
{
int attempts = 0;
while (true)
{
try
{
return action();
}
catch (SqlException ex) when (attempts < MaxRetryAttempts)
{
if (IsTransientError(ex.Number))
{
attempts++;
Thread.Sleep(RetryDelay);
continue;
}
throw;
}
}
}
private bool IsTransientError(int errorNumber)
{
// SQL Server 瞬態(tài)錯誤代碼列表
int transientErrors = { 4060, 40197, 40501, 40613, 49918, 49919, 49920, 4221 };
return transientErrors.Contains(errorNumber);
}
private long GetLastProcessedId(MyDbContext context)
{
return context.YourEntities
.OrderByDescending(x => x.Id)
.Select(x => x.Id)
.FirstOrDefault();
}
private void LogError(Exception ex, int processedCount)
{
// 記錄錯誤和當(dāng)前進(jìn)度到持久化存儲
File.AppendAllText("delete_log.txt",
$"{DateTime.UtcNow:u} Error after {processedCount} records: {ex}\n");
}
}
關(guān)鍵優(yōu)化點(diǎn):
- 分頁策略優(yōu)化:
- 使用
Take(BatchSize).DeleteFromQuery()確保準(zhǔn)確分頁 - 按主鍵排序避免數(shù)據(jù)重復(fù)或遺漏
- 通過子查詢實(shí)現(xiàn)穩(wěn)定分頁
- 性能增強(qiáng):
- 禁用實(shí)體跟蹤 (TrackingEntities = false)
- 批量提交而非事務(wù)包裹整個操作
- 定期重建 DbContext 釋放內(nèi)存
- 調(diào)整批量大小(建議5萬-10萬/批)
- 穩(wěn)定性保障:
- 瞬態(tài)錯誤重試機(jī)制(支持SQL超時、連接中斷等)
- 進(jìn)度記錄到文件(支持?jǐn)帱c(diǎn)續(xù)刪)
- 獨(dú)立的錯誤日志記錄
- 內(nèi)存泄漏預(yù)防(定期釋放上下文)
- 數(shù)據(jù)庫優(yōu)化建議:
sql
-- 執(zhí)行前建議操作
ALTER DATABASE YourDB SET RECOVERY SIMPLE;
EXEC sp_configure 'max server memory', '4096'; -- 根據(jù)服務(wù)器調(diào)整
ALTER DATABASE YourDB SET DELAYED_DURABILITY = FORCED;
-- 執(zhí)行后建議
UPDATE STATISTICS YourTable WITH FULLSCAN;
ALTER INDEX ALL ON YourTable REBUILD;
- 執(zhí)行策略:
- 建議在低峰時段執(zhí)行
- 提前備份目標(biāo)數(shù)據(jù)
- 使用數(shù)據(jù)庫鏡像/AlwaysOn在從庫執(zhí)行
- 監(jiān)控鎖狀態(tài)(sp_who2, sys.dm_tran_locks)
- 高級方案:
csharp
// 分區(qū)并行刪除(需要物理分區(qū))
Parallel.ForEach(GetPartitionRanges(), range =>
{
using var ctx = new MyDbContext();
ctx.YourEntities
.Where(x => x.Id > range.Start && x.Id <= range.End)
.DeleteFromQuery();
});
// 時間范圍分段刪除
var dateRanges = GenerateDateChunks(deleteBefore);
foreach (var range in dateRanges)
{
context.YourEntities
.Where(x => x.CreatedDate >= range.Start && x.CreatedDate < range.End)
.DeleteFromQuery();
}
注意事項(xiàng):
- 建議先在測試環(huán)境驗(yàn)證刪除邏輯
- 確保刪除條件字段有合適的索引(CreatedDate或ID)
- 監(jiān)控事務(wù)日志增長(特別是FULL恢復(fù)模式)
- 考慮使用Table Partitioning處理持續(xù)的大數(shù)據(jù)刪除
對于超大規(guī)模數(shù)據(jù)(超過1億條),建議結(jié)合數(shù)據(jù)庫原生工具(如SQL Server的Partition Switching)實(shí)現(xiàn)秒級數(shù)據(jù)刪除。

浙公網(wǎng)安備 33010602011771號