Loading…
Back To Schedule
Monday, May 20 • 2:50pm - 3:10pm
MPP: Model Performance Predictor

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

Operations is a key challenge in the domain of machine learning pipeline deployments involving monitoring and management of real-time prediction quality. Typically, metrics like accuracy, RMSE etc., are used to track the performance of models in deployment. However, these metrics cannot be calculated in production due to the absence of labels. We propose using an ML algorithm - Model Performance Predictor, to track the performance of the models in deployment. We argue that an ensemble of such metrics can be used to create a score representing the prediction quality in production. This in turn facilitates formulation and customization of ML alerts, that can be escalated by an operations team to the data science team. Such a score automates monitoring and enables ML deployments at scale.

Speakers

Monday May 20, 2019 2:50pm - 3:10pm PDT
Stevens Creek Room

Attendees (8)