MLOPs has been gaining traction and a lot of fortune 500 companies are creating team specifically for MLOPs.Can anyone help me understand? Why MLOPs is gaining so much traction? How it is different from DevOps? What are the tools used for MLOPs? How can i create a strategy for MLOPs? How to get started?

Suppose I have a binary classifier $f$ which acts on an input $x$. Given a threshold $t$, the predicted binary output is defined as: $$ \widehat{y} = \begin{cases} 1, & f(x) \geq t \\ 0, & f(x) < t \end{cases} $$ I then compute the $TPR$ (true positive rate) and $FPR$ (false positive rate) metrics on the hold-out test set (call it $S_1$): $TPR_{S_1} = \Pr(\widehat{y} = 1 | y = 1, S_1)$ $FPR_{S_1} = \Pr(\widehat{y} = 1 | y …

Suppose I deployed a model by manual labeling the ground truth labels with my training data, as the use case is such that there's no way to get the ground truth labels without humans. Once the model is deployed, if I wanted to evaluate how the model is doing on live data, how can I evaluate it without sampling some of that live data, that doesn't come with ground truth labels, and manually giving it the ground truth labels? And …

I just finished some courses on machine learning in production that teaches the concepts and techniques used in training, serving and maintaining ml models. Now I want to do my own project the apply the things I learn. I need a dynamic structured dataset that changes over time to make the need for large scale deployment and induce model decay. I am also thinking about simulating data and concept drift with some noise so if anyone has something that helps …

I am currently working on some python machine learning projects that are soon to be deployed to production. As such, in our team we are interested in doing this the most "correct" way, following MLOps principles. Specifically, I am currently researching the step of data preprocessing and how to implement it in a robust way against training-serving skew. I've considered Tensorflow Transform, that after a single run of some defined preprocessing steps, generates a graph artifact that can be reused …

I have an an ml ops pipeline setup to automatically update a machine learning model in production. The general pipeline steps are: Run unit tests Preprocess training data Train model Evaluate model Deploy model to beta endpoint Run Integration tests Deploy model to production endpoint (manually approved step) My question is: Is it best practise to automatically perform some kind of load testing of the endpoint after step 5? By load testing I mean making sure that the endpoint will …

One key component of MLOPS is continuous training. Which means the end to end training is put in a pipeline which can be triggered, versioned and metadata of the pipeline can be tracked. Thus enabling retraining of the model without lots of manual efforts. Which tool/package do you use for creating such a training pipeline? I am looking for simple tool with following criteria Using only python (i.e. docker is not mandatory like kubeflow) Dev have lot of flexibility and …

There are inconsistant wrong labels and consistant errors in training data. For the former I tried MC-dropout and data Shapley. For the later I wonder if manual data curation is a requisite?

As a software engineers we familiar with a concept of testing (unit, integration, e2e) Tests give us a level of confidence about the code and changes in our code. Looks like for ML the "code" is the data that was used for the model. And unfortunately data not so deterministic as source code. If I consider that data is kind of code for ML: What technics and tools cane be used for verifying / testing the data? My expectation is …

Cross-posted on Reddit ML. Should a Feature Store be part of an enterprise data catalog? To me, a feature store seems to be a highly niche data catalog but missing a lot of the benefits of having an enterprise data catalog / data discovery tool. My need is to have generated features discoverable when searching for data. For example, if I have dataset A and B used to generate a feature set AB', I would want to know about that …

According to MLOPs principle, it is recommended to have a feature store. The question is in the context of doing image classification using deep learning models like convolutional neural networks which does automatic feature engineering(using convolution layers) as part of the training process. Questions Does it make sense to have a feature store for pure image classification/segmentation models? What features should be stored in the feature store? Output of convolution layers? but then they cannot be reused during the training …

Can someone provide a summary of the real world deployment experience of MLflow? We have a few ML models (e.g., LightGBM, tensorflow v2, etc.) and want to avoid framework like SageMaker (due to customer requirement). So we are looking into various ways of hosting ML models for inferencing. Latency is one key performance metric that is very important to us. MLflow looks like a good choice for us. It would be greatly appreciated if the users of MLflow can share …