首页 文章

错误! coreML模型预测图像是错误的,视频是正确的

提问于
浏览
1

我在一个示例iOS应用程序中使用CoreML和我的自定义训练对象检测模型 . 在视频帧上使用时,该模型能够很好地执行并显示正确的类检测和边界框 .

在图像上使用时, bounding box detections are all wrong and all predictions are classified to 1 class.

两种情况下的模型设置是相同的 .

the model prediction call is handled as

func processClassifications(for request: VNRequest, error: Error?) -> [Prediction]? {

    let results = request.results

    let results1 = results as! [VNCoreMLFeatureValueObservation]

    let results2 = try? postprocess().prediction( output: results1[0].featureValue.multiArrayValue! )

    // Some processing from results2 -> predictions

    return predictions
}

For the video:

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)

    // self.visionModel is same as the "  MODEL_TF2keras_OutConv12().model " below...

    guard let visionModel = self.visionModel

    var requestOptions:[VNImageOption : Any] = [:]
    if let cameraIntrinsicData = CMGetAttachment(sampleBuffer, kCMSampleBufferAttachmentKey_CameraIntrinsicMatrix, nil) {
        requestOptions = [.cameraIntrinsics:cameraIntrinsicData]
    }
    let orientation = CGImagePropertyOrientation(rawValue: UInt32(EXIFOrientation.rightTop.rawValue))

    let trackingRequest = VNCoreMLRequest(model: visionModel) { (request, error) in
    guard let predictions = self.processClassifications(for: request, error: error) else { return }. // This function performs the coreML model on the frame and return the predictions.
    }
    trackingRequest.imageCropAndScaleOption = VNImageCropAndScaleOption.centerCrop

    let imageRequestHandler = VNImageRequestHandler(cvPixelBuffer: pixelBuffer, orientation: orientation!, options: requestOptions)
    try imageRequestHandler.perform([trackingRequest])

    }
}

Where as for the Single image the prediction is handled as:

lazy var classificationRequest: VNCoreMLRequest = {

        let model = try VNCoreMLModel(for: MODEL_TF2keras_OutConv12().model)

        let request = VNCoreMLRequest(model: model, completionHandler: { [weak self] request, error in
            let predictions =  self?.processClassifications(for: request, error: error)
        })
        request.imageCropAndScaleOption = VNImageCropAndScaleOption.centerCrop
        return request
    }
}()

func updateClassifications(for image: UIImage) {

    let orientation = CGImagePropertyOrientation(image.imageOrientation)
    guard let ciImage = CIImage(image: image)

    let handler = VNImageRequestHandler(ciImage: ciImage, orientation: orientation)
    handler.perform([self.classificationRequest])
    }

在我的理解中,问题在于在视频案例中使用CVPixelbuffer,在单个图像案例中使用CIImage .

问题是: why is such a difference happening when the function and model call is the same.

我怎么解决这个问题 ?

感谢您的帮助 .

1 回答

  • 0

    包括关键错误

    • 图像的方向在转换中混合 . 视频序列保留了方向 .

    • 转换期间模型输出已损坏 . 从第一步开始的重新转换解决了这个问题 . 因此,我自己的错误,然而一个重要的观察 . 由于从一个平台转换到另一个平台的步骤如此之多,可能的错误来源是多方面的,这有助于从第一步开始 .

    • 图像调整大小和应用程序ImageView调整大小正在影响边界框的可视化 . 因此,检查这些很重要 . 我做了天真的推断,预测是不正确的,而可视化是不正确的 .

    希望这可以帮助 .

相关问题