MLOps Recipes

A library of minimalistic & modular recipes. Easily compose your custom end to end CI/CD pipelines for Machine Learning

Intro

MLOps
MLOps helps you to manage the lifecycle & operationalize your ML models by extending devops principles. It brings in discipline, automation & best practices
Sample MLOps Flow
View image
Technologies
We use Azure ML and Azure Devops in this repository

Principles for the recipes

Minimalistic
Focus on core functionality while keeping the code/config clean & understandable
Modular
Atomic recipes that could be reused (e.g recipe: Deploy to production after approval)
Best of breed
Use best of breed frameworks for individual tasks e.g. postman/newman for functional testing
GitOps/IaC
Entire system is described declaratively: Following principles of GitOps/IaC

Devops approach

Templatized pipelines
Recipes are nothing but templatized devops steps (i.e. pipelines). Each recipe has set of input parameters and defined behaviour and can be mixed and matched independently. Simple CLI based pipelines are autogenerated from cli spec to minimize error
YAML pipelines
Fully CI/CD YAML based multistage pipeline (does not use classic release pipelines in Azure devops)
CLI based MLOps
Use Azure ML CLI from within Devops pipelines as a mechanism for interacting with the ML platform. Simple and clean
YAML based variables
Uses YAML based variables template (UI configuration of variable groups needed only in exceptional cases)



Getting Started

Step 1: Setup the environment
Follow the steps to setup the prerequsites, devops project & optionally a local dev environment
Step 2: Run an end to end pipeline
Follow the steps to run an end to end MLOPs pipeline
Step 3: Contribute
Contribute if you can




Continuous Integration Recipes


Recipe Register a model
Description Register a pretrained model stored in the repo (supports git lfs i.e. large file system) or from model metadata (generated from training run)
Key parameters
  • model name
  • lfs (defaults to true)
  • model path (optional)
  • run metadata artifact: name of metadata file artifact (defaults to run-metadata) (optional)

Parameter details from CLI spec

Sample pipeline output image
Sample devops pipelines
Recipe Run functional tests on service endpoint
Description Run functional tests using newman (postman's cli version) and publishes the test results to Devops pipeline
Key parameters
  • service name: name of registered service in Azure ML
  • postman colection file path
Sample pipeline output image (test results)
image (devops output)
Sample devops pipelines
Recipe Publish a batch inference pipeline
Description Publishes a batch inference pipeline to the workspace & optionally runs it
Key parameters
  • pipeline yaml path: path of batch inference pipeline yml file
  • run pipeline: whether to run the pipeline after publishing it
Sample pipeline output image
Sample devops pipelines
Recipe Run training in the cloud
Description Run the training in the specified cluster and publish the run metadata for optional downstream consumption in devops pipeline
Key parameters
  • source dir
  • entry script
  • conda dependencies
  • model name
  • dataset name
  • aml compute cluster
Sample pipeline output image
Sample devops pipelines



Continuous Delivery Recipes


Recipe Deploy to ACI or AKS
Description This recipe deploys a registered model to ACI or AKS from both registered model or from metadata
Key parameters
  • service name
  • model name
  • model version
  • inference config
  • deploy config
Parameter details from CLI spec
Sample pipeline output image
Sample devops pipelines
Recipe Runs pipeline
Description Runs a specified published pipeline
Key parameters
  • pipeline id
  • experiment name
Sample pipeline output Batch Inference recipe uses this under the hood. Refer to it
Sample devops pipelines Batch Inference recipe uses this under the hood. Refer to it



Infrastructure as Code Recipes


Recipe Provision Azure ML workspace
Description Provisions a new ML workspace along with a Azure ML compute cluster and AKS cluster. Also provisions Storage account, ACR, KeyVault & App insights
Key parameters It sources the parameters through recipes/common/Variables.yml and recipes/IaC/arm/IaCAzureResourceManager.json
Sample pipeline output image
Sample devops pipelines
Recipe Deploy to ACI from model metadata file
Description Provisions a AKS cluster from Azure ML workspace
Key parameters Parameter details from CLI spec
Sample pipeline output image
Sample devops pipelines
Recipe Provisions Azure ML compute
Description Provisions a Azure ML compute cluster
Key parameters Parameter details from CLI spec
Sample pipeline output image
Sample devops pipelines
Recipe Teardown a resource group
Description Tear down a resource group along with all its resources (workspace, clusters etc)
Key parameters It sources the parameters through recipes/common/Variables.yml. It primarily needs service connection name, resource group name and location.
Sample pipeline output
Sample devops pipelines