如何使用Keras在TensorBoard中显示自定义图像？

5 回答

• 0

  因此，以下解决方案适合我：

  import tensorflow as tf
  
  def make_image(tensor):
      """
      Convert an numpy representation image to Image protobuf.
      Copied from https://github.com/lanpa/tensorboard-pytorch/
      """
      from PIL import Image
      height, width, channel = tensor.shape
      image = Image.fromarray(tensor)
      import io
      output = io.BytesIO()
      image.save(output, format='PNG')
      image_string = output.getvalue()
      output.close()
      return tf.Summary.Image(height=height,
                           width=width,
                           colorspace=channel,
                           encoded_image_string=image_string)
  
  class TensorBoardImage(keras.callbacks.Callback):
      def __init__(self, tag):
          super().__init__() 
          self.tag = tag
  
      def on_epoch_end(self, epoch, logs={}):
          # Load image
          img = data.astronaut()
          # Do something to the image
          img = (255 * skimage.util.random_noise(img)).astype('uint8')
  
          image = make_image(img)
          summary = tf.Summary(value=[tf.Summary.Value(tag=self.tag, image=image)])
          writer = tf.summary.FileWriter('./logs')
          writer.add_summary(summary, epoch)
          writer.close()
  
          return
  
  tbi_callback = TensorBoardImage('Image Example')
  

  只需将回调传递给 fit 或 fit_generator .

  请注意，您还可以使用回调内的 model 运行某些操作 . 例如，您可以在某些图像上运行模型以检查其性能 .

  

  回复于 2024-05-04T08:19:27+08:00
• 1

  同样地，您可能想尝试tf-matplotlib . 这是散点图

  import tensorflow as tf
  import numpy as np
  
  import tfmpl
  
  @tfmpl.figure_tensor
  def draw_scatter(scaled, colors): 
      '''Draw scatter plots. One for each color.'''  
      figs = tfmpl.create_figures(len(colors), figsize=(4,4))
      for idx, f in enumerate(figs):
          ax = f.add_subplot(111)
          ax.axis('off')
          ax.scatter(scaled[:, 0], scaled[:, 1], c=colors[idx])
          f.tight_layout()
  
      return figs
  
  with tf.Session(graph=tf.Graph()) as sess:
  
      # A point cloud that can be scaled by the user
      points = tf.constant(
          np.random.normal(loc=0.0, scale=1.0, size=(100, 2)).astype(np.float32)
      )
      scale = tf.placeholder(tf.float32)        
      scaled = points*scale
  
      # Note, `scaled` above is a tensor. Its being passed `draw_scatter` below. 
      # However, when `draw_scatter` is invoked, the tensor will be evaluated and a
      # numpy array representing its content is provided.   
      image_tensor = draw_scatter(scaled, ['r', 'g'])
      image_summary = tf.summary.image('scatter', image_tensor)      
      all_summaries = tf.summary.merge_all() 
  
      writer = tf.summary.FileWriter('log', sess.graph)
      summary = sess.run(all_summaries, feed_dict={scale: 2.})
      writer.add_summary(summary, global_step=0)
  

  执行时，这将导致Tensorboard中的以下图表
  

  请注意 tf-matplotlib 负责评估任何张量输入，避免 pyplot 线程问题并支持运行时关键绘图的blitting .

  回复于 2024-05-04T08:19:27+08:00
• 0

  我相信我找到了一种更好的方法来将这些自定义图像记录到使用tf-matplotlib的张量板上 . 这是怎么...

  class TensorBoardDTW(tf.keras.callbacks.TensorBoard):
      def __init__(self, **kwargs):
          super(TensorBoardDTW, self).__init__(**kwargs)
          self.dtw_image_summary = None
  
      def _make_histogram_ops(self, model):
          super(TensorBoardDTW, self)._make_histogram_ops(model)
          tf.summary.image('dtw-cost', create_dtw_image(model.output))
  

  只需要从TensorBoard回调类覆盖_make_histogram_ops方法来添加自定义摘要 . 在我的例子中， create_dtw_image 是一个使用tf-matplotlib创建图像的函数 .

  问候， .

  回复于 2024-05-04T08:19:27+08:00
• 16

  以下是如何在图像上绘制地标的示例：

  class CustomCallback(keras.callbacks.Callback):
      def __init__(self, model, generator):
          self.generator = generator
          self.model = model
  
      def tf_summary_image(self, tensor):
          import io
          from PIL import Image
  
          tensor = tensor.astype(np.uint8)
  
          height, width, channel = tensor.shape
          image = Image.fromarray(tensor)
          output = io.BytesIO()
          image.save(output, format='PNG')
          image_string = output.getvalue()
          output.close()
          return tf.Summary.Image(height=height,
                               width=width,
                               colorspace=channel,
                               encoded_image_string=image_string)
  
      def on_epoch_end(self, epoch, logs={}):
          frames_arr, landmarks = next(self.generator)
  
          # Take just 1st sample from batch
          frames_arr = frames_arr[0:1,...]
  
          y_pred = self.model.predict(frames_arr)
  
          # Get last frame for which we have done predictions
          img = frames_arr[0,-1,:,:]
  
          img = img * 255
          img = img[:, :, ::-1]
          img = np.copy(img)
  
          landmarks_gt = landmarks[-1].reshape(-1,2)
          landmarks_pred = y_pred.reshape(-1,2)
  
          img = draw_landmarks(img, landmarks_gt, (0,255,0))
          img = draw_landmarks(img, landmarks_pred, (0,0,255))
  
          image = self.tf_summary_image(img)
          summary = tf.Summary(value=[tf.Summary.Value(image=image)])
          writer = tf.summary.FileWriter('./logs')
          writer.add_summary(summary, epoch)
          writer.close()
          return
  

  回复于 2024-05-04T08:19:27+08:00
• 0

  根据以上答案和我自己的搜索，我提供以下代码，使用Keras中的TensorBoard完成以下操作：

  ---

  • 问题设置：预测双目立体匹配中的视差图;

  • 用输入左图像 x 和地面实况视差图 gt 来提供模型;

  • 在某个迭代时间显示输入 x 和地面实况'gt';

  • 在某个迭代时间显示模型的输出 y .

  ---

  • 首先，你必须使用 Callback 制作你的服装回调类 . Note 回调可以通过类属性 self.model 访问其关联模型 . 另外 Note ： you have to feed the input to the model with feed_dict, if you want to get and display the output of your model.

  from keras.callbacks import Callback
  import numpy as np
  from keras import backend as K
  import tensorflow as tf
  
  # make the 1 channel input image or disparity map look good within this color map. This function is not necessary for this Tensorboard problem shown as above. Just a function used in my own research project.
  def colormap_jet(img):
      return cv2.cvtColor(cv2.applyColorMap(np.uint8(img), 2), cv2.COLOR_BGR2RGB)
  
  class customModelCheckpoint(Callback):
      def __init__(self, log_dir = './logs/tmp/', feed_inputd_display = None):
            super(customModelCheckpoint, self).__init__()
            self.seen = 0
            self.feed_inputs_display = feed_inputs_display
            self.writer = tf.summary.FileWriter(log_dir)
  
      # this function will return the feeding data for TensorBoard visualization;
      # arguments:
      #  * feed_input_display : [(input_yourModelNeed, left_image, disparity_gt ), ..., (input_yourModelNeed, left_image, disparity_gt), ...], i.e., the list of tuples of Numpy Arrays what your model needs as input and what you want to display using TensorBoard. Note: you have to feed the input to the model with feed_dict, if you want to get and display the output of your model. 
      def custom_set_feed_input_to_display(self, feed_inputs_display):
            self.feed_inputs_display = feed_inputs_display
  
      # copied from the above answers;
      def make_image(self, numpy_img):
            from PIL import Image
            height, width, channel = numpy_img.shape
            image = Image.fromarray(numpy_img)
            import io
            output = io.BytesIO()
            image.save(output, format='PNG')
            image_string = output.getvalue()
            output.close()
            return tf.Summary.Image(height=height, width=width, colorspace= channel, encoded_image_string=image_string)
  
  
      # A callback has access to its associated model through the class property self.model.
      def on_batch_end(self, batch, logs = None):
            logs = logs or {} 
            self.seen += 1
            if self.seen % 200 == 0: # every 200 iterations or batches, plot the costumed images using TensorBorad;
                summary_str = []
                for i in range(len(self.feed_inputs_display)):
                    feature, disp_gt, imgl = self.feed_inputs_display[i]
                    disp_pred = np.squeeze(K.get_session().run(self.model.output, feed_dict = {self.model.input : feature}), axis = 0)
                    #disp_pred = np.squeeze(self.model.predict_on_batch(feature), axis = 0)
                    summary_str.append(tf.Summary.Value(tag= 'plot/img0/{}'.format(i), image= self.make_image( colormap_jet(imgl)))) # function colormap_jet(), defined above;
                    summary_str.append(tf.Summary.Value(tag= 'plot/disp_gt/{}'.format(i), image= self.make_image( colormap_jet(disp_gt))))
                    summary_str.append(tf.Summary.Value(tag= 'plot/disp/{}'.format(i), image= self.make_image( colormap_jet(disp_pred))))
  
                self.writer.add_summary(tf.Summary(value = summary_str), global_step =self.seen)
  

  • 接下来，将此回调对象传递给模型的 fit_generator() ，如：

  feed_inputs_4_display = some_function_you_wrote()
     callback_mc = customModelCheckpoint( log_dir = log_save_path, feed_inputd_display = feed_inputs_4_display)
     # or 
     callback_mc.custom_set_feed_input_to_display(feed_inputs_4_display)
     yourModel.fit_generator(... callbacks = callback_mc)
     ...
  

  • 现在你可以运行代码，然后去TensorBoard主机看看打扮的图像显示 . 例如，这是我使用上述代码得到的：
    

  ---

  完成！请享用！

  回复于 2024-05-04T08:19:27+08:00