ml.net例子筆記1
1 ml.net例子概要
https://github.com/feiyun0112/machinelearning-samples.zh-cn/tree/master
https://gitee.com/mirrors_feiyun0112/machinelearning-samples.zh-cn
根據(jù)場景和機(jī)器學(xué)習(xí)問題/任務(wù)，官方ML.NET示例被分成多個類別，可通過下表訪問：
二元分類
| 二元分類 | ||
|---|---|---|
 |
||
 情緒分析C#** **F# |
 |
|
 垃圾信息檢測C#** **F# |
 |
|
 **信用卡欺詐識別 |
||
| (Binary Classification)C# **F# | ||
 |
||
 心臟病預(yù)測C# |
多類分類
| 多類分類 | ||
|---|---|---|
 |
||
 GitHub Issues 分類C#** **F# |
 |
|
 鳶尾花分類C#** **F# |
 |
|
 手寫數(shù)字識別C# |
建議
| 建議 | ||
|---|---|---|
 |
||
 產(chǎn)品推薦C# |
 |
|
 **電影推薦 |
||
| (Matrix Factorization)**C# |  |
|
 **電影推薦 |
||
| (Field Aware Factorization Machines)**C# |
回歸
| 回歸 | ||
|---|---|---|
 |
||
 價格預(yù)測C#** **F# |
||
 |
||
 銷售預(yù)測C# |
 |
|
 需求預(yù)測C#** **F# |
時間序列預(yù)測
| 時間序列預(yù)測 | ||
|---|---|---|
 |
||
 銷售預(yù)測C# |
異常情況檢測
| 異常情況檢測 | ||
|---|---|---|
 |
||
銷售高峰檢測 ** C#  **C# |
 |
|
 電力異常檢測C# |
 |
|
 **信用卡欺詐檢測 |
||
| (Anomaly Detection)**C# |
聚類分析
| 聚類分析 | ||
|---|---|---|
 |
||
 客戶細(xì)分C#** **F# |
 |
|
 鳶尾花聚類C#** **F# |
排名
| 排名 | ||
|---|---|---|
 |
||
 排名搜索引擎結(jié)果C# |
計算機(jī)視覺
| 計算機(jī)視覺 | ||
|---|---|---|
 **圖像分類訓(xùn)練 |
||
(High-Level API) C# F# ** |
 **圖像分類預(yù)測 |
|
(Pretrained TensorFlow model scoring) C# F#  **C# |
 **圖像分類訓(xùn)練 |
|
(TensorFlow Featurizer Estimator) C# **F# |
||
 **對象檢測 |
||
(ONNX model scoring) C#  **C# |
跨領(lǐng)域方案
| 跨領(lǐng)域方案 | ||
|---|---|---|
 |
||
 **Web API上的可擴(kuò)展模型 |
||
| **C# |  |
|
 **Razor Web應(yīng)用程序上的可擴(kuò)展模型 |
||
| **C# |  |
|
 **Azure Functions上的可擴(kuò)展模型 |
||
| **C# | ||
 |
||
 **Blazor Web應(yīng)用程序上的可擴(kuò)展模型 |
||
| **C# |  |
|
 **大數(shù)據(jù)集 |
||
| **C# |  |
|
 **使用DatabaseLoader加載數(shù)據(jù) |
||
| **C# | ||
 |
||
 **使用LoadFromEnumerable加載數(shù)據(jù) |
||
| **C# |  |
|
 **模型可解釋性 |
||
| **C# |  |
|
 **導(dǎo)出到ONNX |
||
| **C# |
2 ml.net例子筆記
工程目錄結(jié)構(gòu)，按照ml.net的使用類別， 以CLI modelbuilder csharp等進(jìn)行了分類，分別對應(yīng)ml.net的命令行自動學(xué)習(xí)；GUI方式的學(xué)習(xí)；API方式的學(xué)習(xí)
Vs缺少組件的自動安裝
避免 windows路徑太長的問題報錯可以設(shè)置如下：
本地策略組編輯器
計算機(jī)配置/管理模板/系統(tǒng)/文件系統(tǒng)
3 二元分類
4 情緒分析
5 訓(xùn)練數(shù)據(jù)
這個問題集中在預(yù)測客戶的評論是否具有正面或負(fù)面情緒。我們將使用小型的wikipedia-detox-datasets（一個用于訓(xùn)練的數(shù)據(jù)集，一個用于模型的準(zhǔn)確性評估的數(shù)據(jù)集），這些數(shù)據(jù)集已經(jīng)由人工處理過，并且每個評論都被分配了一個情緒標(biāo)簽：
- 0 - 好評/正面
- 1 - 差評/負(fù)面
我們將使用這些數(shù)據(jù)集構(gòu)建一個模型，在預(yù)測時將分析字符串并預(yù)測情緒值為0或1。
訓(xùn)練數(shù)據(jù)類似如下：
Label rev_id comment year logged_in ns sample split
0 666674821.0 " He is a Rapist!!!!! Please edit the article to include this important fact. Thank You. — Preceding unsigned comment added by ? " 2015 True article blocked train
0 24297552.0 The other two films Hitch and Magnolia are also directly related to the community in question, and may be of interest to those who see those films. So why not link to them? 2005 False article random train
6 數(shù)據(jù)結(jié)構(gòu)
public class SentimentIssue
{
[LoadColumn(0)]
public bool Label { get; set; }
[LoadColumn(2)]
public string Text { get; set; }
}
只選取了數(shù)據(jù)中的0，2這兩個列
IDataView dataView = mlContext.Data.LoadFromTextFile<SentimentIssue>(DataPath, hasHeader: true);
TrainTestData trainTestSplit = mlContext.Data.TrainTestSplit(dataView, testFraction: 0.2);
IDataView trainingData = trainTestSplit.TrainSet;
IDataView testData = trainTestSplit.TestSet;
使用定義的輸入數(shù)據(jù)類型加載數(shù)據(jù)，并以2-8方式(0.2)切分訓(xùn)練和驗(yàn)證數(shù)據(jù)
7 訓(xùn)練
// STEP 2: Common data process configuration with pipeline data transformations
var dataProcessPipeline = mlContext.Transforms.Text.FeaturizeText(outputColumnName: "Features", inputColumnName: nameof(SentimentIssue.Text));
// STEP 3: Set the training algorithm, then create and config the modelBuilder
var trainer = mlContext.BinaryClassification.Trainers.SdcaLogisticRegression(labelColumnName: "Label", featureColumnName: "Features");
var trainingPipeline = dataProcessPipeline.Append(trainer);
// STEP 4: Train the model fitting to the DataSet
ITransformer trainedModel = trainingPipeline.Fit(trainingData);
dataProcessPipelin變量是一個數(shù)據(jù)處理管道，通過調(diào)用mlContext.Transforms.Text.FeaturizeText方法來創(chuàng)建。這個方法的參數(shù)有兩個，分別是outputColumnName和inputColumnName。
outputColumnName表示輸出的特征列的名稱，這里設(shè)置為"Features"。
inputColumnName表示輸入的文本列的名稱，這里使用了nameof(SentimentIssue.Text)，它是一個C#語言的語法，表示SentimentIssue.Text的名稱。
SdcaLogisticRegression方法創(chuàng)建一個二分類訓(xùn)練器trainer，并設(shè)置標(biāo)簽列名為"Label"，特征列名為"Features"。這個訓(xùn)練器使用Sdca算法實(shí)現(xiàn)邏輯回歸模型。
8 評估模型
// STEP 5: Evaluate the model and show accuracy stats
var predictions = trainedModel.Transform(testData);
var metrics = mlContext.BinaryClassification.Evaluate(data: predictions, labelColumnName: "Label", scoreColumnName: "Score");
ConsoleHelper.PrintBinaryClassificationMetrics(trainer.ToString(), metrics);
// STEP 6: Save/persist the trained model to a .ZIP file
mlContext.Model.Save(trainedModel, trainingData.Schema, ModelPath);
Console.WriteLine("The model is saved to {0}", ModelPath);
使用測試數(shù)據(jù)評價模型的訓(xùn)練結(jié)果并保存模型
9 工程運(yùn)行結(jié)果
10 垃圾信息檢測
11 訓(xùn)練數(shù)據(jù)
近6000條被分類為“垃圾信息”或“ham”（不是垃圾信息）的消息。
下載的數(shù)據(jù)類似如下
ham Go until jurong point, crazy.. Available only in bugis n great world la e buffet... Cine there got amore wat...
ham Ok lar... Joking wif u oni...
spam Free entry in 2 a wkly comp to win FA Cup final tkts 21st May 2005. Text FA to 87121 to receive entry question(std txt rate)T&C's apply 08452810075over18's
ham U dun say so early hor... U c already then say...
12 數(shù)據(jù)結(jié)構(gòu)
class SpamInput
{
[LoadColumn(0)]
public string Label { get; set; }
[LoadColumn(1)]
public string Message { get; set; }
}
對應(yīng)數(shù)據(jù)中的2列，都是字符串類型
13 訓(xùn)練
// Create the estimator which converts the text label to boolean, featurizes the text, and adds a linear trainer.
// Data process configuration with pipeline data transformations
var dataProcessPipeline = mlContext.Transforms.Conversion.MapValueToKey("Label", "Label")
.Append(mlContext.Transforms.Text.FeaturizeText("FeaturesText", new Microsoft.ML.Transforms.Text.TextFeaturizingEstimator.Options
{
WordFeatureExtractor = new Microsoft.ML.Transforms.Text.WordBagEstimator.Options { NgramLength = 2, UseAllLengths = true },
CharFeatureExtractor = new Microsoft.ML.Transforms.Text.WordBagEstimator.Options { NgramLength = 3, UseAllLengths = false },
Norm = Microsoft.ML.Transforms.Text.TextFeaturizingEstimator.NormFunction.L2,
}, "Message"))
.Append(mlContext.Transforms.CopyColumns("Features", "FeaturesText"))
.AppendCacheCheckpoint(mlContext);
由于機(jī)器學(xué)習(xí)默認(rèn)處理的都是數(shù)字，因此文字內(nèi)容需要處理
將"Label"列的值映射為鍵值對。
對"Message"列進(jìn)行文本向量化，使用WordBagEstimator將文本轉(zhuǎn)換為特征向量。WordBagEstimator的選項(xiàng)參數(shù)包括：NgramLength：ngram的長度。UseAllLengths：是否使用所有長度的ngram。Norm：特征向量的歸一化方式。
復(fù)制"FeaturesText"列為"Features"列。
在管道中添加一個緩存檢查點(diǎn)。【在機(jī)器學(xué)習(xí)模型的訓(xùn)練過程中，數(shù)據(jù)處理是非常耗時的操作。緩存檢查點(diǎn)是一種優(yōu)化技術(shù)，可以將數(shù)據(jù)處理的結(jié)果緩存起來，以避免在后續(xù)訓(xùn)練迭代中重復(fù)處理同樣的數(shù)據(jù)。這樣可以顯著提高訓(xùn)練速度，減少訓(xùn)練時間。在預(yù)測過程中也可以使用緩存檢查點(diǎn)，以加速數(shù)據(jù)處理的過程。】
// Set the training algorithm
var trainer = mlContext.MulticlassClassification.Trainers.OneVersusAll(mlContext.BinaryClassification.Trainers.AveragedPerceptron(labelColumnName: "Label", numberOfIterations: 10, featureColumnName: "Features"), labelColumnName: "Label")
.Append(mlContext.Transforms.Conversion.MapKeyToValue("PredictedLabel", "PredictedLabel"));
var trainingPipeLine = dataProcessPipeline.Append(trainer);
// Evaluate the model using cross-validation.
// Cross-validation splits our dataset into 'folds', trains a model on some folds and
// evaluates it on the remaining fold. We are using 5 folds so we get back 5 sets of scores.
// Let's compute the average AUC, which should be between 0.5 and 1 (higher is better).
Console.WriteLine("=============== Cross-validating to get model's accuracy metrics ===============");
var crossValidationResults = mlContext.MulticlassClassification.CrossValidate(data: data, estimator: trainingPipeLine, numberOfFolds: 5);
ConsoleHelper.PrintMulticlassClassificationFoldsAverageMetrics(trainer.ToString(), crossValidationResults);
// Now let's train a model on the full dataset to help us get better results
var model = trainingPipeLine.Fit(data);
- 設(shè)置訓(xùn)練算法：這里使用的是 OneVersusAll，一種常用的多類別分類策略，基本上是每次將某一類別作為正例，其他所有類別作為負(fù)例，訓(xùn)練一個二分類器。使用的訓(xùn)練器是 AveragedPerceptron，這是一種常用的二分類器。最后通過 Append 方法將轉(zhuǎn)換操作（將標(biāo)簽列映射為預(yù)測標(biāo)簽列）添加到訓(xùn)練器中。
- 將訓(xùn)練器添加到數(shù)據(jù)預(yù)處理管道中：dataProcessPipeline.Append(trainer) 這行代碼將訓(xùn)練器添加到數(shù)據(jù)預(yù)處理管道中，這樣在訓(xùn)練模型時，會首先對數(shù)據(jù)進(jìn)行預(yù)處理，然后再進(jìn)行訓(xùn)練。
- 進(jìn)行交叉驗(yàn)證：通過 MulticlassClassification.CrossValidate 方法進(jìn)行交叉驗(yàn)證，該方法會按照設(shè)定的折數(shù)（這里為 5）將數(shù)據(jù)集分割成若干部分，用一部分?jǐn)?shù)據(jù)進(jìn)行訓(xùn)練，用另一部分?jǐn)?shù)據(jù)評估模型的性能。這樣可以得到每折的評分，然后計算平均 AUC（Area Under the Curve）。
- 在全數(shù)據(jù)集上訓(xùn)練模型：trainingPipeLine.Fit(data) 這行代碼在全數(shù)據(jù)集上訓(xùn)練模型，以得到更好的結(jié)果。
14 工程運(yùn)行結(jié)果
15 建議
16 產(chǎn)品推薦
ProductRecommender.csproj將這個工程手工加入進(jìn)來
該工程是基于購買的歷史，推薦產(chǎn)品相關(guān)購買
17 訓(xùn)練數(shù)據(jù)
如下是脫敏的數(shù)據(jù)
ProductID ProductID_Copurchased
0 1
0 2
0 3
0 4
0 5
1 0
1 2
1 4
1 5
1 15
18 數(shù)據(jù)結(jié)構(gòu)
public class ProductEntry
{
[KeyType(count : 262111)]
public uint ProductID { get; set; }
[KeyType(count : 262111)]
public uint CoPurchaseProductID { get; set; }
}
訓(xùn)練數(shù)據(jù)中產(chǎn)品類別就是262111個
19 訓(xùn)練
//STEP 2: Read the trained data using TextLoader by defining the schema for reading the product co-purchase dataset
// Do remember to replace amazon0302.txt with dataset from [https://snap.stanford.edu/data/amazon0302.html](https://snap.stanford.edu/data/amazon0302.html)
var traindata = mlContext.Data.LoadFromTextFile(path:TrainingDataLocation,
columns: new[]
{
new TextLoader.Column("Label", DataKind.Single, 0),
new TextLoader.Column(name:nameof(ProductEntry.ProductID), dataKind:DataKind.UInt32, source: new [] { new TextLoader.Range(0) }, keyCount: new KeyCount(262111)),
new TextLoader.Column(name:nameof(ProductEntry.CoPurchaseProductID), dataKind:DataKind.UInt32, source: new [] { new TextLoader.Range(1) }, keyCount: new KeyCount(262111))
},
hasHeader: true,
separatorChar: '\t');
//STEP 3: Your data is already encoded so all you need to do is specify options for MatrxiFactorizationTrainer with a few extra hyperparameters
// LossFunction, Alpa, Lambda and a few others like K and C as shown below and call the trainer.
MatrixFactorizationTrainer.Options options = new MatrixFactorizationTrainer.Options();
options.MatrixColumnIndexColumnName = nameof(ProductEntry.ProductID);
options.MatrixRowIndexColumnName = nameof(ProductEntry.CoPurchaseProductID);
options.LabelColumnName= "Label";
options.LossFunction = MatrixFactorizationTrainer.LossFunctionType.SquareLossOneClass;
options.Alpha = 0.01;
options.Lambda = 0.025;
// For better results use the following parameters
//options.K = 100;
//options.C = 0.00001;
//Step 4: Call the MatrixFactorization trainer by passing options.
var est = mlContext.Recommendation().Trainers.MatrixFactorization(options);
//STEP 5: Train the model fitting to the DataSet
//Please add Amazon0302.txt dataset from [https://snap.stanford.edu/data/amazon0302.html](https://snap.stanford.edu/data/amazon0302.html) to Data folder if FileNotFoundException is thrown.
ITransformer model = est.Fit(traindata);
Matrix Factorization訓(xùn)練器是一種用于訓(xùn)練推薦系統(tǒng)的工具。它主要用于處理用戶行為數(shù)據(jù)，并學(xué)習(xí)用戶興趣和物品之間的潛在關(guān)系。這種訓(xùn)練器可以應(yīng)用于多種推薦場景，如電影推薦、商品推薦等。
Matrix Factorization訓(xùn)練器通過將用戶行為數(shù)據(jù)表示為一個矩陣，并使用矩陣分解技術(shù)對矩陣進(jìn)行分解，從而挖掘出用戶興趣和物品之間的潛在關(guān)系。這種訓(xùn)練器通常采用隨機(jī)梯度下降（SGD）等優(yōu)化算法進(jìn)行訓(xùn)練，并使用正則化技術(shù)來防止過擬合。
20 工程運(yùn)行結(jié)果
修改了下原工程代碼，多推薦幾個：
var pes = new ProductEntry[]
{
new ProductEntry() {
ProductID = 3,
CoPurchaseProductID = 63
},
new ProductEntry() {
ProductID = 3,
CoPurchaseProductID = 20
},
new ProductEntry() {
ProductID = 262108,
CoPurchaseProductID = 262109
},
new ProductEntry() {
ProductID = 262108,
CoPurchaseProductID = 3
}
};
var predictionengine = mlContext.Model.CreatePredictionEngine<ProductEntry, Copurchase_prediction>(model);
foreach (var p in pes)
{
var prediction = predictionengine.Predict(p);
Console.WriteLine($"For ProductID = {p.ProductID} and CoPurchaseProductID = {p.CoPurchaseProductID}");
Console.WriteLine(" the predicted score is " + Math.Round(prediction.Score, 1));
}
可見這個算法偏差不小
機(jī)器學(xué)習(xí)的過程不同的算法差別可能會很大
21 電影推薦-矩陣分解
22 訓(xùn)練數(shù)據(jù)
電影評分，標(biāo)題，流派等信息
movieId,title,genres
1,Toy Story (1995),Adventure|Animation|Children|Comedy|Fantasy
2,Jumanji (1995),Adventure|Children|Fantasy
3,Grumpier Old Men (1995),Comedy|Romance
4,Waiting to Exhale (1995),Comedy|Drama|Romance
recommendation-ratings-train.csv
userId,movieId,rating,timestamp
1,1,4,964982703
1,3,4,964981247
1,6,4,964982224
1,47,5,964983815
23 數(shù)據(jù)結(jié)構(gòu)
public class MovieRating
{
[LoadColumn(0)]
public float userId;
[LoadColumn(1)]
public float movieId;
[LoadColumn(2)]
public float Label;
}
24 訓(xùn)練
//STEP 3: Transform your data by encoding the two features userId and movieID. These encoded features will be provided as input
// to our MatrixFactorizationTrainer.
var dataProcessingPipeline = mlcontext.Transforms.Conversion.MapValueToKey(outputColumnName: "userIdEncoded", inputColumnName: nameof(MovieRating.userId))
.Append(mlcontext.Transforms.Conversion.MapValueToKey(outputColumnName: "movieIdEncoded", inputColumnName: nameof(MovieRating.movieId)));
//Specify the options for MatrixFactorization trainer
MatrixFactorizationTrainer.Options options = new MatrixFactorizationTrainer.Options();
options.MatrixColumnIndexColumnName = "userIdEncoded";
options.MatrixRowIndexColumnName = "movieIdEncoded";
options.LabelColumnName = "Label";
options.NumberOfIterations = 20;
options.ApproximationRank = 100;
//STEP 4: Create the training pipeline
var trainingPipeLine = dataProcessingPipeline.Append(mlcontext.Recommendation().Trainers.MatrixFactorization(options));
//STEP 5: Train the model fitting to the DataSet
Console.WriteLine("=============== Training the model ===============");
ITransformer model = trainingPipeLine.Fit(trainingDataView);
25 工程運(yùn)行結(jié)果
26 電影推薦-場景感知分解機(jī)
這個例子將訓(xùn)練和使用分開來寫
samples\csharp\end-to-end-apps\Recommendation-MovieRecommender\MovieRecommender_Model 這個是模型訓(xùn)練
samples\csharp\end-to-end-apps\Recommendation-MovieRecommender\MovieRecommender 這個是 asp.net core程序
27 訓(xùn)練數(shù)據(jù)
userId,movieId,rating,timestamp
1,1,4,964982703
1,3,4,964981247
1,6,4,964982224
1,47,5,964983815
原始數(shù)據(jù)使用如下函數(shù)切分進(jìn)行評分歸一化，并按照9:1的比例進(jìn)行切分
/*
* FieldAwareFactorizationMachine the learner used in this example requires the problem to setup as a binary classification problem.
* The DataPrep method performs two tasks:
* 1. It goes through all the ratings and replaces the ratings > 3 as 1, suggesting a movie is recommended and ratings < 3 as 0, suggesting
a movie is not recommended
2. This piece of code also splits the ratings.csv into rating-train.csv and ratings-test.csv used for model training and testing respectively.
*/
public static void DataPrep()
28 數(shù)據(jù)結(jié)構(gòu)
public class MovieRating
{
[LoadColumn(0)]
public string userId;
[LoadColumn(1)]
public string movieId;
[LoadColumn(2)]
public bool Label;
}
這個和矩陣分解算法是一樣的
29 訓(xùn)練
// ML.NET doesn't cache data set by default. Therefore, if one reads a data set from a file and accesses it many times, it can be slow due to
// expensive featurization and disk operations. When the considered data can fit into memory, a solution is to cache the data in memory. Caching is especially
// helpful when working with iterative algorithms which needs many data passes. Since SDCA is the case, we cache. Inserting a
// cache step in a pipeline is also possible, please see the construction of pipeline below.
trainingDataView = mlContext.Data.Cache(trainingDataView);
Console.WriteLine("=============== Transform Data And Preview ===============", color);
Console.WriteLine();
//STEP 4: Transform your data by encoding the two features userId and movieID.
// These encoded features will be provided as input to FieldAwareFactorizationMachine learner
var dataProcessPipeline = mlContext.Transforms.Text.FeaturizeText(outputColumnName: "userIdFeaturized", inputColumnName: nameof(MovieRating.userId))
.Append(mlContext.Transforms.Text.FeaturizeText(outputColumnName: "movieIdFeaturized", inputColumnName: nameof(MovieRating.movieId))
.Append(mlContext.Transforms.Concatenate("Features", "userIdFeaturized", "movieIdFeaturized")));
Common.ConsoleHelper.PeekDataViewInConsole(mlContext, trainingDataView, dataProcessPipeline, 10);
// STEP 5: Train the model fitting to the DataSet
Console.WriteLine("=============== Training the model ===============", color);
Console.WriteLine();
var trainingPipeLine = dataProcessPipeline.Append(mlContext.BinaryClassification.Trainers.FieldAwareFactorizationMachine(new string[] { "Features" }));
var model = trainingPipeLine.Fit(trainingDataView);
運(yùn)行這個工程就把模型訓(xùn)練出來并保存成文件了
30 模型消費(fèi)和使用
運(yùn)行這個工程
初始界面都是靜態(tài)數(shù)據(jù)的展示，選擇如Ankit查看如下：
點(diǎn)擊推薦，顯示如下：
MovieRatingPrediction prediction = null;
foreach (var movie in _movieService.GetTrendingMovies)
{
// Call the Rating Prediction for each movie prediction
prediction = _model.Predict(new MovieRating
{
userId = id.ToString(),
movieId = movie.MovieID.ToString()
});
// Normalize the prediction scores for the "ratings" b/w 0 - 100
float normalizedscore = Sigmoid(prediction.Score);
// Add the score for recommendation of each movie in the trending movie list
ratings.Add((movie.MovieID, normalizedscore));
}
對當(dāng)前的票房電影每個算個推薦的評分
