Custom Metrics Sample

Requirements

• Authenticated to gcloud (gcloud auth application-default login)

!pip install --upgrade -r requirements.txt > /dev/null

%env GOOGLE_CLOUD_PROJECT=your-gcp-project

import merlin
import warnings
import os
from merlin.model import ModelType, PyFuncModel
warnings.filterwarnings('ignore')

1. Initialize

1.1 Set Server

merlin.set_url("http://localhost:3000")

1.2 Set Active Project

project represent a project in real life. You may have multiple model within a project.

merlin.set_project(<project_name>) will set the active project into the name matched by argument. You can only set it to an existing project. If you would like to create a new project, please do so from the MLP console at http://localhost:3000/projects/create.

merlin.set_project("sample")

1.3 Set Active Model

model represents an abstract ML model. Conceptually, model in Merlin is similar to a class in programming language. To instantiate a model you’ll have to create a model_version.

Each model has a type, currently model type supported by Merlin are: sklearn, xgboost, tensorflow, pytorch, and user defined model (i.e. pyfunc model).

model_version represents a snapshot of particular model iteration. You’ll be able to attach information such as metrics and tag to a given model_version as well as deploy it as a model service.

merlin.set_model(<model_name>, <model_type>) will set the active model to the name given by parameter, if the model with given name is not found, a new model will be created.

merlin.set_model("pyfunc-metric", ModelType.PYFUNC)

2. Create Model

In this step we are going to create an echo model which count the number of incoming request as a metrics called my_counter and return the incoming request as response.

2.1 Define PyFunc Model Class

To create a PyFunc model you’ll have to extend merlin.PyFuncModel class and implement its initialize and infer method.

initialize will be called once during model initialization. The argument to initialize is a dictionary containing a key value pair of artifact name and its URL. The artifact’s keys are the same value as received by log_pyfunc_model.

infer method is the prediction method that is need to be implemented. It accept a dictionary type argument which represent incoming request body. infer should return a dictionary object which correspond to response body of prediction result.

In following example we are creating PyFunc model called EchoModel. In its initialize method we create a prometheus counter with my_counter as metrics name. The infer method itself is just increment the counter and simply return incoming request.

from prometheus_client import Counter

class EchoModel(PyFuncModel):
    def initialize(self, artifacts):
        self.counter = Counter("my_counter", 'My custom counter')

    def infer(self, request):
        self.counter.inc()
        return request

Let’s test it locally

m = EchoModel()
m.initialize({})
m.infer({"instances": [[1,2,3,4], [2,1,2,4]] })

3. Deploy Model

To deploy the model, we will have to create an iteration of the model (by create a model_version), upload the serialized model to Merlin, and then deploy.

3.1 Create Model Version and Upload

merlin.new_model_version() is a convenient method to create a model version and start its development process. It is equal to following codes:

v = model.new_model_version()
v.start()
v.log_pyfunc_model(model_instance=EchoModel(),
                conda_env="env.yaml")
v.finish()

To upload PyFunc model you have to provide following arguments: 1. model_instance is the instance of PyFunc model, the model has to extend merlin.PyFuncModel 2. conda_env is path to conda environment yaml file. The environment yaml file must contain all dependency required by the PyFunc model. 3. (Optional) artifacts is additional artifact that you want to include in the model 4. (Optional) code_path is a list of directory containing python code that will be loaded during model initialization, this is required when model_instance depend on local python package

with merlin.new_model_version() as v:
    merlin.log_pyfunc_model(model_instance=EchoModel(),
                conda_env="env.yaml")

3.2 Deploy Model

Each of a deployed model version will have its own generated url

endpoint = merlin.deploy(v)

3.3 Send Test Request

%%bash -s "$endpoint.url"
curl -v -X POST $1 -d '{
  "instances": [
    [2.8,  1.0,  6.8,  0.4],
    [3.1,  1.4,  4.5,  1.6]
  ]
}'

3.4 Delete Deployment

merlin.undeploy(v)