如何计算keras中的接收操作特性（ROC）和AUC？

5 回答

• 10

  由于您无法通过小批量计算ROC和AUC，因此您只能在一个时期结束时计算它 . 有一个来自jamartinh的解决方案，为方便起见，我修补了下面的代码：

  from sklearn.metrics import roc_auc_score
  from keras.callbacks import Callback
  class roc_callback(Callback):
      def __init__(self,training_data,validation_data):
          self.x = training_data[0]
          self.y = training_data[1]
          self.x_val = validation_data[0]
          self.y_val = validation_data[1]
  
  
      def on_train_begin(self, logs={}):
          return
  
      def on_train_end(self, logs={}):
          return
  
      def on_epoch_begin(self, epoch, logs={}):
          return
  
      def on_epoch_end(self, epoch, logs={}):
          y_pred = self.model.predict(self.x)
          roc = roc_auc_score(self.y, y_pred)
          y_pred_val = self.model.predict(self.x_val)
          roc_val = roc_auc_score(self.y_val, y_pred_val)
          print('\rroc-auc: %s - roc-auc_val: %s' % (str(round(roc,4)),str(round(roc_val,4))),end=100*' '+'\n')
          return
  
      def on_batch_begin(self, batch, logs={}):
          return
  
      def on_batch_end(self, batch, logs={}):
          return
  
  model.fit(X_train, y_train, validation_data=(X_test, y_test), callbacks=[roc_callback(training_data=(X_train, y_train),validation_data=(X_test, y_test))])
  

  A more hackable way using tf.contrib.metrics.streaming_auc:

  import numpy as np
  import tensorflow as tf
  from sklearn.metrics import roc_auc_score
  from sklearn.datasets import make_classification
  from keras.models import Sequential
  from keras.layers import Dense
  from keras.utils import np_utils
  from keras.callbacks import Callback, EarlyStopping
  
  
  # define roc_callback, inspired by https://github.com/keras-team/keras/issues/6050#issuecomment-329996505
  def auc_roc(y_true, y_pred):
      # any tensorflow metric
      value, update_op = tf.contrib.metrics.streaming_auc(y_pred, y_true)
  
      # find all variables created for this metric
      metric_vars = [i for i in tf.local_variables() if 'auc_roc' in i.name.split('/')[1]]
  
      # Add metric variables to GLOBAL_VARIABLES collection.
      # They will be initialized for new session.
      for v in metric_vars:
          tf.add_to_collection(tf.GraphKeys.GLOBAL_VARIABLES, v)
  
      # force to update metric values
      with tf.control_dependencies([update_op]):
          value = tf.identity(value)
          return value
  
  # generation a small dataset
  N_all = 10000
  N_tr = int(0.7 * N_all)
  N_te = N_all - N_tr
  X, y = make_classification(n_samples=N_all, n_features=20, n_classes=2)
  y = np_utils.to_categorical(y, num_classes=2)
  
  X_train, X_valid = X[:N_tr, :], X[N_tr:, :]
  y_train, y_valid = y[:N_tr, :], y[N_tr:, :]
  
  # model & train
  model = Sequential()
  model.add(Dense(2, activation="softmax", input_shape=(X.shape[1],)))
  
  model.compile(loss='categorical_crossentropy',
                optimizer='adam',
                metrics=['accuracy', auc_roc])
  
  my_callbacks = [EarlyStopping(monitor='auc_roc', patience=300, verbose=1, mode='max')]
  
  model.fit(X, y,
            validation_split=0.3,
            shuffle=True,
            batch_size=32, nb_epoch=5, verbose=1,
            callbacks=my_callbacks)
  
  # # or use independent valid set
  # model.fit(X_train, y_train,
  #           validation_data=(X_valid, y_valid),
  #           batch_size=32, nb_epoch=5, verbose=1,
  #           callbacks=my_callbacks)
  

  回复于 2024-04-29T10:31:26+08:00
• 14

  我这样解决了我的问题

  考虑您有功能的测试数据集 x_test 和相应目标的 y_test .

  首先，我们使用训练模型预测特征中的目标

  y_pred = model.predict_proba(x_test)
  

  然后从sklearn我们导入roc_auc_score函数然后简单地将原始目标和预测目标传递给函数 .

  roc_auc_score(y_test, y_pred)
  

  回复于 2024-04-29T10:31:26+08:00
• 6

  以下解决方案适合我：

  import tensorflow as tf
  from keras import backend as K
  
  def auc(y_true, y_pred):
      auc = tf.metrics.auc(y_true, y_pred)[1]
      K.get_session().run(tf.local_variables_initializer())
      return auc
  
  model.compile(loss="binary_crossentropy", optimizer='adam', metrics=[auc])
  

  回复于 2024-04-29T10:31:26+08:00
• 42

  和你一样，我更喜欢使用scikit-learn的内置方法来评估AUROC . 我发现在keras中执行此操作的最佳和最简单的方法是创建自定义指标 . 如果tensorflow是你的后端，那么实现这一点可以在很少的代码行中完成：

  import tensorflow as tf
  from sklearn.metrics import roc_auc_score
  
  def auroc(y_true, y_pred):
      return tf.py_func(roc_auc_score, (y_true, y_pred), tf.double)
  
  # Build Model...
  
  model.compile(loss='categorical_crossentropy', optimizer='adam',metrics=['accuracy', auroc])
  

  如其他答案中所述，创建自定义回调将不适用于您的情况，因为您的模型具有多个输出，但这将起作用 . 另外，该方法允许在训练和验证数据上评估度量，而keras回调不能访问训练数据，因此仅可用于评估训练数据的性能 .

  回复于 2024-04-29T10:31:26+08:00
• 14

  'roc_curve'，'auc'不是标准指标，你不能将它们传递给指标变量，这是不允许的 . 你可以传递像'fmeasure'这样的标准指标 .

  在此处查看可用的指标：https://keras.io/metrics/您可能还想查看制作自己的自定义指标：https://keras.io/metrics/#custom-metrics

  另请查看本博客中为ROC，AUC提及的generate_results方法... https://vkolachalama.blogspot.in/2016/05/keras-implementation-of-mlp-neural.html

  回复于 2024-04-29T10:31:26+08:00