source,text
introduction/index.html,"   Documentation  What is MLflow?       What is MLflow?  Stepping into the world of Machine Learning (ML) is an exciting journey, but it often comes with complexities that can hinder innovation and experimentation. MLflow is a solution to many of these issues in this dynamic landscape, offering tools and simplifying processes to streamline the ML lifecycle and foster collaboration among ML practitioners. Whether you're an individual researcher, a member of a large team, or somewhere in between, MLflow provides a unified platform to navigate the intricate maze of model development, deployment, and management. MLflow aims to enable innovation in ML solution development by streamlining otherwise cumbersome logging, organization, and lineage concerns that are unique to model development. This focus allows you to ensure that your ML projects are robust, transparent, and ready for real-world challenges. Read on to discover the core components of MLflow and understand the unique advantages it brings to the complex workflows associated with model development and management.  Core Components of MLflow  MLflow, at its core, provides a suite of tools aimed at simplifying the ML workflow. It is tailored to assist ML practitioners throughout the various stages of ML development and deployment. Despite its expansive offerings, MLflow's functionalities are rooted in several foundational components:  Tracking: MLflow Tracking provides both an API and UI dedicated to the logging of parameters, code versions, metrics, and artifacts during the ML process. This centralized repository captures details such as parameters, metrics, artifacts, data, and environment configurations, giving teams insight into their models' evolution over time. Whether working in standalone scripts, notebooks, or other environments, Tracking facilitates the logging of results either to local files or a server, making it easier to compare multiple runs across different users. Model Registry: A systematic approach to model management, the Model Registry assists in handling different versions of models, discerning their current state, and ensuring a smooth transition from development to production. It offers a centralized model store, APIs, and UI to collaboratively manage an MLflow Model's full lifecycle, including model lineage, versioning, stage transitions, and annotations. AI Gateway: This server, equipped with a set of standardized APIs, streamlines access to both SaaS and OSS LLM models. It serves as a unified interface, bolstering security through authenticated access, and offers a common set of APIs for prominent LLMs. Evaluate: Designed for in-depth model analysis, this set of tools facilitates objective model comparison, be it traditional ML algorithms or cutting-edge LLMs. Prompt Engineering UI: A dedicated environment for prompt engineering, this UI-centric component provides a space for prompt experimentation, refinement, evaluation, testing, and deployment. Recipes: Serving as a guide for structuring ML projects, Recipes, while offering recommendations, are focused on ensuring functional end results optimized for real-world deployment scenarios. Projects: MLflow Projects standardize the packaging of ML code, workflows, and artifacts, akin to an executable. Each project, be it a directory with code or a Git repository, employs a descriptor or convention to define its dependencies and execution method.  By integrating these core components, MLflow offers an end-to-end platform, ensuring efficiency, consistency, and traceability throughout the ML lifecycle.   Why Use MLflow?  The machine learning (ML) process is intricate, comprising various stages, from data preprocessing to model deployment and monitoring. Ensuring productivity and efficiency throughout this lifecycle poses several challenges:  Experiment Management: It's tough to keep track of the myriad experiments, especially when working with files or interactive notebooks. Determining which combination of data, code, and parameters led to a particular result can become a daunting task. Reproducibility: Ensuring consistent results across runs is not trivial. Beyond just tracking code versions and parameters, capturing the entire environment, including library dependencies, is critical. This becomes even more challenging when collaborating with other data scientists or when scaling the code to different platforms. Deployment Consistency: With the plethora of ML libraries available, there's often no standardized way to package and deploy models. Custom solutions can lead to inconsistencies, and the crucial link between a model and the code and parameters that produced it might be lost. Model Management: As data science teams produce numerous models, managing these models, their versions, and stage transitions becomes a significant hurdle. Without a centralized platform, managing model lifecycles, from development to staging to production, becomes unwieldy. Library Agnosticism: While individual ML libraries might offer solutions to some of the challenges, achieving the best results often involves experimenting across multiple libraries. A platform that offers compatibility with various libraries while ensuring models are usable as reproducible "black boxes" is essential.  MLflow addresses these challenges by offering a unified platform tailored for the entire ML lifecycle. Its benefits include:  Traceability: With tools like the Tracking Server, every experiment is logged, ensuring that teams can trace back and understand the evolution of models. Consistency: Be it accessing models through the AI Gateway or structuring projects with MLflow Recipes, MLflow promotes a consistent approach, reducing both the learning curve and potential errors. Flexibility: MLflow's library-agnostic design ensures compatibility with a wide range of machine learning libraries. It offers comprehensive support across different programming languages, backed by a robust REST API, CLI, and APIs for Python API, R API, and Java API.  By simplifying the complex landscape of ML workflows, MLflow empowers data scientists and developers to focus on building and refining models, ensuring a streamlined path from experimentation to production.   Who Uses MLflow?  Throughout the lifecycle of a particular project, there are components within MLflow that are designed to cater to different needs.    MLflow's versatility enhances workflows across various roles, from data scientists to prompt engineers, extending its impact beyond just the confines of a Data Science team.  Data Scientists leverage MLflow for:  Experiment tracking and hypothesis testing persistence. Code structuring for better reproducibility. Model packaging and dependency management. Evaluating hyperparameter tuning selection boundaries. Comparing the results of model retraining over time. Reviewing and selecting optimal models for deployment.  MLOps Professionals utilize MLflow to:  Manage the lifecycles of trained models, both pre and post deployment. Deploy models securely to production environments. Audit and review candidate models prior to deployment. Manage deployment dependencies.  Data Science Managers interact with MLflow by:  Reviewing the outcomes of experimentation and modeling activities. Collaborating with teams to ensure that modeling objectives align with business goals.  Prompt Engineering Users use MLflow for:  Evaluating and experimenting with large language models. Crafting custom prompts and persisting their candidate creations. Deciding on the best base model suitable for their specific project requirements.     Use Cases of MLflow  MLflow is versatile, catering to diverse machine learning scenarios. Here are some typical use cases:  Experiment Tracking: A data science team leverages MLflow Tracking to log parameters and metrics for experiments within a particular domain. Using the MLflow UI, they can compare results and fine-tune their solution approach. The outcomes of these experiments are preserved as MLflow models. Model Selection and Deployment: MLOps engineers employ the MLflow UI to assess and pick the top-performing models. The chosen model is registered in the MLflow Registry, allowing for monitoring its real-world performance. Model Performance Monitoring: Post deployment, MLOps engineers utilize the MLflow Registry to gauge the model's efficacy, juxtaposing it against other models in a live environment. Collaborative Projects: Data scientists embarking on new ventures organize their work as an MLflow Project. This structure facilitates easy sharing and parameter modifications, promoting collaboration.    Scalability in MLflow  MLflow is architected to seamlessly integrate with diverse data environments, from small datasets to Big Data applications. It's built with the understanding that quality machine learning outcomes often hinge on robust data sources, and as such, scales adeptly to accommodate varying data needs. Here's how MLflow addresses scalability across different dimensions:  Distributed Execution: MLflow runs can operate on distributed clusters. For instance, integration with Apache Spark allows for distributed processing. Furthermore, runs can be initiated on the distributed infrastructure of your preference, with results relayed to a centralized Tracking Server for analysis. Notably, MLflow offers an integrated API to initiate runs on Databricks. Parallel Runs: For use cases like hyperparameter tuning, MLflow can orchestrate multiple runs simultaneously, each with distinct parameters. Interoperability with Distributed Storage: MLflow Projects can interface with distributed storage solutions, including Azure ADLS, Azure Blob Storage, AWS S3, Cloudflare R2 and DBFS. Whether it's automatically fetching files to a local environment or interfacing with a distributed storage URI directly, MLflow ensures that projects can handle extensive datasets – even scenarios like processing a 100 TB file. Centralized Model Management with Model Registry: Large-scale organizations can benefit from the MLflow Model Registry, a unified platform tailored for collaborative model lifecycle management. In environments where multiple data science teams might be concurrently developing numerous models, the Model Registry proves invaluable. It streamlines model discovery, tracks experiments, manages versions, and facilitates understanding a model's intent across different teams.  By addressing these scalability dimensions, MLflow ensures that users can capitalize on its capabilities regardless of their data environment's size or complexity.        Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
getting-started/index.html,"   Documentation  Getting Started with MLflow       Getting Started with MLflow  For those new to MLflow or seeking a refresher on its core functionalities, the quickstart tutorials here are the perfect starting point. They will guide you step-by-step through fundamental concepts, focusing purely on a task that will maximize your understanding of how to use MLflow to solve a particular task.  5-minute Quickstart - MLflow Tracking  In this brief introductory quickstart on MLflow Tracking, you will learn how to leverage MLflow to:  Log training statistics (loss, accuracy, etc.) and hyperparameters for a model Log (save) a model for later retrieval Register a model to enable state transitions for deployment Load the model and use it for inference  In the process of learning these key concepts, you will be exposed to the MLflow fluent API, the MLflow Tracking UI, and learn how to add metadata associated with a model training event to an MLflow run.   If you would like to get started immediately by interactively running the notebook, you can:  Download the Notebook Quickstart elements  You can read through the quickstart as a guide, or navigate directly to the notebook example to get started.                          MLflow Tracking Quickstart Guide                                   Learn the basics of MLflow Tracking in a fast-paced guide with a focus on seeing your first model in the MLflow UI                                           MLflow Tracking Quickstart Notebook                                       See an example of using the MLflow fluent API to log, load, and register a model for inference. Great for code-focused learners!                        Logging your first MLflow Model  In this lengthy tutorial, you will walk through the basics of MLflow in a sequential and guided manner. With each subsequent step, you will increase your familiarity with the primary functionality around MLflow Tracking and how to navigate the MLflow UI. If you would like to get started immediately by interactively running the notebook, you can:  Download the Notebook Guide sections  Interested in navigating directly to the content that you're curious about? Select the section from each tutorial below!                          Setting up the MLflow Tracking Server                                       Learn how to start an MLflow Tracking Server and the MLflow UI Server locally                                           Using the MLflow Client                                       Connect to the Tracking Server with the MLflow Client and learn how to search for experiments                                           Create your first Experiment                                       Explore the MLflow UI and create your first MLflow experiment with a unique name and identifying tags                                           Search Experiments by tags                                       Learn how to use tags for MLflow experiments and how to leverage them for searching                                           Creating a Dataset for Testing                                       Build a synthetic dataset to use while exploring the features of MLflow                                           Logging your first MLflow run                                       Train a model using the synthetic dataset and log the trained model, metrics, and parameters                                           View the full Notebook                                       See the tutorial notebook in its entirety. If you prefer just reading code, this is the best place to look.                          15 minute Quickstart - Autologging in MLflow  In this rapid-pace quickstart, you will be exposed to the autologging feature in MLflow to simplify the logging of models, metrics, and parameters. After training and viewing the logged run data, we'll load the logged model to perform inference, showing core features of MLflow Tracking in the most time-efficient manner possible.                          Introduction to autologging                                       Train a model and use MLflow autologging to automatically record model artifacts, metrics, and parameters                                           View autologged data in the MLflow UI                                       See what autologging will autonomously log for you during model training with only a single line of code                                           Loading a model for inference                                       Load the autologged model in its native format and use it to generate predictions                         15 minute Quickstart - Comparing Runs and Deploying your Best Model  This quickstart tutorial focuses on the MLflow UI's run comparison feature, provides a brief overview of MLflow Projects, and shows how to register a model. After locally serving the registered model, a brief example of preparing a model for remote deployment via containerizing the model via Docker is covered.                          Generate runs                                       Run an MLflow Project that will perform hyperparameter tuning to generate a large volume of runs                                           Run comparison                                       Use the MLflow UI Runs Compare functionality to evaluate the hyperparameter tuning run and select the best model                                           Register the best model                                       Learn to register models with the MLflow UI and perform stage transitions from within the UI                                           Start a local ML inference server                                       Use the integrated inference server in MLflow to serve your registered model locally                                           Build a deployable container for your model                                       Learn how to generate a docker container that houses your model for deployment to external services                         5 Minute Tracking Server Overview  This quickstart tutorial walks through different types of MLflow Tracking Servers and how to use them to log your MLflow experiments.                          5 Minute Tracking Server Overview                                    Learn how to log MLflow experiments with different tracking servers                         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
new-features/index.html,"   Documentation  New Features       New Features  Looking to learn about new significant releases in MLflow? Find out about the details of major features, changes, and deprecations below.                          MLflow Docs Overhaul                                        The MLflow docs are getting a facelift with added content, tutorials, and guides. Stay tuned for further improvements to the site!                   released in 2.8.0                           New Features for LLM Evaluation                                        The functionality provided for LLM evaluation in MLflow is getting greatly expanded. Check out all of the new features in the                     guide and the tutorials.                  Learn more  released in 2.8.0                           Updated Model Registry UI                                        A new opt-in Model Registry UI has been built that uses Aliases and Tags for managing model development. See                    more about the new UI workflow in the docs.                  Learn more  released in 2.8.0                           Spark Connect support                                        You can now log, save, and load models trained using Spark Connect. Try out Spark 3.5 and the MLflow integration today!                   released in 2.8.0                           AI21 Labs added as an MLflow Gateway provider                                        You can now use the MLflow AI Gateway to connect to LLMs hosted by AI21 Labs.                  Learn more  released in 2.8.0                           AWS Bedrock added as an MLflow Gateway provider                                        You can now use the MLflow AI Gateway to connect to LLMs hosted by AWS's Bedrock service.                  Learn more  released in 2.8.0                           PaLM 2 added as an MLflow Gateway provider                                        You can now use the MLflow AI Gateway to connect to LLMs hosted by Google's PaLM 2 service.                  Learn more  released in 2.8.0                           Hugging Face TGI added as an MLflow Gateway provider                                        You can self-host your own transformers-based models from the Hugging Face Hub and directly connect to the models with the AI Gateway                     with TGI.                  Learn more  released in 2.8.0                           LLM evaluation viewer added to MLflow UI                                        You can view your LLM evaluation results directly from the MLflow UI.                  Learn more  released in 2.7.0                           Introducting the Prompt Engineering UI                                        Link your MLflow Tracking Server with your MLflow AI Gateway Server to experiment, evaluate, and construct                     prompts that can be compared amongst different providers without writing a single line of code.                  Learn more  released in 2.7.0                           MosaicML Support in AI Gateway                                        MosaicML has now been added to the supported providers in MLflow AI Gateway.                     You can now seamlessly interface with managed popular models like                     MPT-30B and other models in the MPT family.                                       Try it out today with our example.                  Learn more  released in 2.7.0                           Cloudflare R2 now supported as an artifact store                                        Cloudflare's R2 storage backend is now supported for use as an artifact store. To learn more about                     R2, read the Cloudflare docs to get more information and to explore what is possible.                   released in 2.7.0                           Params support for PyFunc Models                                        PyFunc models now support passing parameters at inference time. With this new feature,                     you can define the allowable keys, with default values, for any parameters that you would like                     consumers of your model to be able to override. This is particularly useful for LLMs, where you                     might want to let users adjust commonly modified parameters for a model, such as token counts and temperature.                  Learn more  released in 2.6.0                           MLflow Serving support added to MLflow AI Gateway                                        The MLflow AI Gateway now supports defining an MLflow serving endpoint as provider. With this                     new feature, you can serve any OSS transformers model that conforms to the                     completions or embeddings route type                     definitions.                                       Try it out today with our end-to-end example.                  Learn more  released in 2.6.0                           Introducing the MLflow AI Gateway                                        We're excited to announce the newest top-level component in the MLflow ecosystem: The AI Gateway.                                       With this new feature, you can create a single access point to many of the most popular LLM SaaS services available now,                     simplifying interfaces, managing credentials, and providing a unified standard set of APIs to reduce the complexity of                     building products and services around LLMs.                  Learn more  released in 2.5.0                           MLflow Evaluate now supports LLMs                                        You can now use MLflow evaluate to compare results from your favorite LLMs on a fixed prompt.                                       With support for many of the standard evaluation metrics for LLMs built in directly to the API, the featured                     LLM modeling tasks of text summarization, text classification, question answering, and text generation allows you                     to view the results of submitted text to multiple models in a single UI element.                  Learn more  released in 2.4.0                           Chart View added to the MLflow UI                                        You can now visualize parameters and metrics across multiple runs as a chart on the runs table.                  Learn more  released in 2.2.0           Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
llms/index.html,"   Documentation  LLMs       LLMs  LLMs, or Large Language Models, have rapidly become a cornerstone in the machine learning domain, offering immense capabilities ranging from natural language understanding to code generation and more. However, harnessing the full potential of LLMs often involves intricate processes, from interfacing with multiple providers to fine-tuning specific models to achieve desired outcomes. Such complexities can easily become a bottleneck for developers and data scientists aiming to integrate LLM capabilities into their applications. MLflow's Support for LLMs aims to alleviate these challenges by introducing a suite of features and tools designed with the end-user in mind:  MLflow AI Gateway    Serving as a unified interface, the MLflow AI Gateway simplifies interactions with multiple LLM providers, such as OpenAI, MosaicML, Cohere, Anthropic, PaLM 2, AWS Bedrock, and AI21 Labs. In addition to supporting the most popular SaaS LLM providers, the AI Gateway provides an integration to MLflow model serving, allowing you to serve your own LLM or a fine-tuned foundation model within your own serving infrastructure.  Note The MLflow AI Gateway is in active development and has been marked as Experimental. APIs may change as this new feature is refined and its functionality is expanded based on feedback.   Benefits of the MLflow AI Gateway   Unified Endpoint: No more juggling between multiple provider APIs. Simplified Integrations: One-time setup, no repeated complex integrations. Secure Credential Management:  Centralized storage prevents scattered API keys. No hardcoding or user-handled keys.   Consistent API Experience:  Uniform API across all providers. Easy-to-use REST endpoints and Client API.   Seamless Provider Swapping:  Swap providers without touching your code. Zero downtime provider, model, or route swapping.      Explore the Native Provider integrations  The MLflow AI Gateway supports a large range of foundational models from popular SaaS model vendors, as well as providing a means of self-hosting your own open source model via an integration with MLflow model serving. To learn more about how to get started using the MLflow AI Gateway to simplify the configuration and management of your LLM serving needs, select the provider that you're interested in below:                                                   Getting Started Examples for each Provider  If you're interested in learning about how to set up the MLflow AI Gateway for a specific provider, follow the links below for our up-to-date documentation on GitHub. Each link will take you to a README file that will explain how to set up a route for the provider. In the same directory as the README, you will find a runnable example of how to query the routes that the example creates, providing you with a quick reference for getting started with your favorite provider!  OpenAI quickstart MosaicML quickstart Anthropic quickstart Cohere quickstart MLflow quickstart AWS Bedrock quickstart AI21 Labs quickstart PaLM 2 quickstart Azure OpenAI quickstart Hugging Face Text Generation Interface (TGI) quickstart   Note The MLflow and Hugging Face TGI providers are for self-hosted LLM serving of either foundation open-source LLM models, fine-tuned open-source LLM models, or your own custom LLM. The example documentation for these providers will show you how to get started with these, using free-to-use open-source models from the Hugging Face Hub.     LLM Evaluation    Navigating the vast landscape of Large Language Models (LLMs) can be daunting. Determining the right model, prompt, or service that aligns with a project's needs is no small feat. Traditional machine learning evaluation metrics often fall short when it comes to assessing the nuanced performance of generative models. Enter MLflow LLM Evaluation. This feature is designed to simplify the evaluation process, offering a streamlined approach to compare foundational models, providers, and prompts.  Benefits of MLflow's LLM Evaluation   Simplified Evaluation: Navigate the LLM space with ease, ensuring the best fit for your project with standard metrics that can be used to compare generated text. Use-Case Specific Metrics: Leverage MLflow's mlflow.evaluate() API for a high-level, frictionless evaluation experience. Customizable Metrics: Beyond the provided metrics, MLflow supports a plugin-style for custom scoring, enhancing the evaluation's flexibility. Comparative Analysis: Effortlessly compare foundational models, providers, and prompts to make informed decisions. Deep Insights: Dive into the intricacies of generative models with a comprehensive suite of LLM-relevant metrics.  MLflow's LLM Evaluation is designed to bridge the gap between traditional machine learning evaluation and the unique challenges posed by LLMs.    Prompt Engineering UI    Effective utilization of LLMs often hinges on crafting the right prompts. The development of a high-quality prompt is an iterative process of trial and error, where subsequent experimentation is not guaranteed to result in cumulative quality improvements. With the volume and speed of iteration through prompt experimentation, it can quickly become very overwhelming to remember or keep a history of the state of different prompts that were tried. Serving as a powerful tool for prompt engineering, the MLflow Prompt Engineering UI revolutionizes the way developers interact with and refine LLM prompts.  Benefits of the MLflow Prompt Engineering UI   Iterative Development: Streamlined process for trial and error without the overwhelming complexity. UI-Based Prototyping: Prototype, iterate, and refine prompts without diving deep into code. Accessible Engineering: Makes prompt engineering more user-friendly, speeding up experimentation. Optimized Configurations: Quickly hone in on the best model configurations for tasks like question answering or document summarization. Transparent Tracking:  Every model iteration and configuration is meticulously tracked. Ensures reproducibility and transparency in your development process.     Note The MLflow Prompt Engineering UI is in active development and has been marked as Experimental. Features and interfaces may evolve as feedback is gathered and the tool is refined.     Native MLflow Flavors for LLMs  Harnessing the power of LLMs becomes effortless with flavors designed specifically for working with LLM libraries and frameworks.  Benefits of MLflow's Native Flavors for LLMs   Support for Popular Packages:  Native integration with packages like transformers, sentence-transformers, open-ai , and langchain. Standardized interfaces for tasks like saving, logging, and managing inference configurations.   PyFunc Compatibility:  Load models as PyFuncs for broad compatibility across serving infrastructures. Strengthens the MLOps process for LLMs, ensuring smooth deployments.   Cohesive Ecosystem:  All essential tools and functionalities consolidated under MLflow. Focus on deriving value from LLMs without getting bogged down by interfacing and optimization intricacies.      Explore the Native LLM Flavors  Select the integration below to read the documentation on how to leverage MLflow's native integration with these popular libraries:                          Native Integration Examples  If you'd like to directly explore code examples for how to get started with using our official library integrations, you can navigate directly to our up-to-date examples on GitHub below:  transformers   Simple Text Generation Example A Conversational Model Example Component Logging with Transformers Audio Transcription with Whisper Fine-tuning a Text Classification Model    sentence-transformers   Text Encoding Example    langchain   Logging and using a Chain Logging and using an Agent Logging and using a Retriever Chain 1 Logging and using a Retrieval QA Chain 1     1 Demonstrates the use of Retrieval Augmented Generation (RAG) using a Vector Store  openai   Using a Completions endpoint Using a Chat endpoint Performing Embeddings Generation Using OpenAI on a Spark DataFrame for Batch Processing Using Azure OpenAI        LLM Tracking in MLflow    Empowering developers with advanced tracking capabilities, the MLflow LLM Tracking System stands out as the premier solution for managing and analyzing interactions with Large Language Models (LLMs).  Benefits of the MLflow LLM Tracking System   Robust Interaction Management: Comprehensive tracking of every LLM interaction for maximum insight. Tailor-Made for LLMs:  Unique features specifically designed for LLMs. From logging prompts to tracking dynamic data, MLflow has it covered.   Deep Model Insight:  Introduces 'predictions' as a core entity, alongside the existing artifacts, parameters, and metrics. Gain unparalleled understanding of text-generating model behavior and performance.   Clarity and Repeatability:  Ensures consistent and transparent tracking across all LLM interactions. Facilitates informed decision-making and optimization in LLM deployment and utilization.       Tutorials and Use Case Guides for LLMs in MLflow  Interested in learning how to leverage MLflow for your LLM projects? Look in the tutorials and guides below to learn more about interesting use cases that could help to make your journey into leveraging LLMs a bit easier!                            Evaluating LLMs                                       Learn how to evaluate LLMs with MLflow.                                           Using Custom PyFunc with LLMs                                       Explore the nuances of packaging, customizing, and deploying advanced LLMs in MLflow using custom PyFuncs.                                           Question Generation for RAG                                       Learn how to leverage LLMs to generate a question dataset for use in Retrieval Augmented Generation applications.                            Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
model-evaluation/index.html,"   Documentation  Model Evaluation       Model Evaluation   Harnessing the Power of Automation  In the evolving landscape of machine learning, the evaluation phase of model development is just as important as ever. Ensuring the accuracy, reliability, and efficiency of models is paramount to ensure that the model that has been trained has been as thoroughly validated as it can be prior to promoting it beyond the development phase. However, manual evaluation can be tedious, error-prone, and time-consuming. MLflow addresses these challenges head-on, offering a suite of automated tools that streamline the evaluation process, saving time and enhancing accuracy, helping you to have confidence that the solution that you've spent so much time working on will meet the needs of the problem you're trying to solve.   LLM Model Evaluation  The rise of Large Language Models (LLMs) like ChatGPT has transformed the landscape of text generation, finding applications in question answering, translation, and text summarization. However, evaluating LLMs introduces unique challenges, primarily because there's often no single ground truth to compare against. MLflow's evaluation tools are tailored for LLMs, ensuring a streamlined and accurate evaluation process. Key Features:  Versatile Model Evaluation: MLflow supports evaluating various types of LLMs, whether it's an MLflow pyfunc model, a URI pointing to a registered MLflow model, or any python callable representing your model. Comprehensive Metrics: MLflow offers a range of metrics for LLM evaluation. From metrics that rely on SaaS models like OpenAI for scoring (e.g., mlflow.metrics.genai.answer_relevance()) to function-based per-row metrics such as Rouge (mlflow.metrics.rougeL()) or Flesch Kincaid (mlflow.metrics.flesch_kincaid_grade_level()). Predefined Metric Collections: Depending on your LLM use case, MLflow provides predefined metric collections, such as "question-answering" or "text-summarization", simplifying the evaluation process. Custom Metric Creation: Beyond the predefined metrics, MLflow allows users to create custom LLM evaluation metrics. Whether you're looking to evaluate the professionalism of a response or any other custom criteria, MLflow provides the tools to define and implement these metrics. Evaluation with Static Datasets: As of MLflow 2.8.0, you can evaluate a static dataset without specifying a model. This is especially useful when you've saved model outputs in a dataset and want a swift evaluation without rerunning the model. Integrated Results View: MLflow's mlflow.evaluate() returns comprehensive evaluation results, which can be viewed directly in the code or through the MLflow UI for a more visual representation.  Harnessing these features, MLflow's LLM evaluation tools eliminate the complexities and ambiguities associated with evaluating large language models. By automating these critical evaluation tasks, MLflow ensures that users can confidently assess the performance of their LLMs, leading to more informed decisions in the deployment and application of these models.  Guides and Tutorials for LLM Model Evaluation  To learn more about how you can leverage MLflow's evaluation features for your LLM-powered project work, see the tutorials below:                          RAG Evaluation                                       Learn how to evaluate a Retrieval Augmented Generation setup with MLflow Evaluate                                           Question-Answering Evaluation                                       See a working example of how to evaluate the quality of an LLM Question-Answering solution                                           RAG Question Generation Evaluation                                       See how to generate Questions for RAG generation and how to evaluate a RAG solution using MLflow                        Traditional ML Evaluation  Traditional machine learning techniques, from classification to regression, have been the bedrock of many industries. MLflow recognizes their significance and offers automated evaluation tools tailored for these classic techniques. Key Features:  Evaluating a Function: To get immediate results, you can evaluate a python function directly without logging the model. This is especially useful when you want a quick evaluation without the overhead of logging. Evaluating a Dataset: MLflow also supports evaluating a static dataset without specifying a model. This is invaluable when you've saved model outputs in a dataset and want a swift evaluation without having to rerun model inference. Evaluating a Model: With MLflow, you can set validation thresholds for your metrics. If a model doesn't meet these thresholds compared to a baseline, MLflow will alert you. This automated validation ensures that only high-quality models progress to the next stages. Common Metrics and Visualizations: MLflow automatically logs common metrics like accuracy, precision, recall, and more. Additionally, visual graphs such as the confusion matrix, lift_curve_plot, and others are auto-logged, providing a comprehensive view of your model's performance. SHAP Integration: MLflow is integrated with SHAP, allowing for the auto-logging of SHAP's summarization importances validation visualizations when using the evaluate APIs.         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
deep-learning/index.html,"   Documentation  Deep Learning       Deep Learning  The realm of deep learning has witnessed an unprecedented surge, revolutionizing numerous sectors with its ability to process vast amounts of data and capture intricate patterns. From the real-time object detection in autonomous vehicles to the generation of art through Generative Adversarial Networks, and from natural language processing applications in chatbots to predictive analytics in e-commerce, deep learning models are at the forefront of today's AI-driven innovations. MLflow acknowledges the profound impact and complexity of deep learning. With a keen focus on the unique challenges posed by deep learning workflows, such as iterative model training and hyperparameter tuning, MLflow introduces a robust suite of tools specifically designed for these advanced models. MLflow helps to facilitate seamless model development, ensuring reproducibility, and provides enhanced monitoring capabilities with the concept of 'steps' for recording metrics at various training iterations, with integrated UI features that enable you to easily visualize the iterative improvements of key metrics during training epochs.  Key Benefits:   Iterative Model Training: With the concept of 'steps', MLflow allows users to log metrics at various training iterations, offering a granular view of the model's progress. Reproducibility: Ensure that every model training run can be replicated with the exact same conditions. Scalability: Handle projects ranging from small-scale models to enterprise-level deployments with ease. Traceability: Keep track of every detail, from hyperparameters to the final model output.    Deep Autologging Integrations  One of the standout features of MLflow's deep learning support is its deep autologging integrations. These integrations automatically capture and log intricate details during the training of deep learning models, ensuring that every nuance, from model parameters to evaluation metrics, is meticulously recorded. This is especially prominent in frameworks like TensorFlow, PyTorch Lightning, base PyTorch, and Keras, making the iterative training process more insightful and manageable.   Native Library Support  Deep learning in MLflow is enriched by its native support for a number of the most popular libraries. The native integration with each of these libraries within MLflow help to streamline and simplify the training process, as well as saving, logging, loading, and representing models as generic Python functions for inference use anywhere. Opting for these native integrations brings forth a myriad of advantages:  Auto-logging Capabilities: Automatically capture details without manual intervention. Custom Serialization: Streamline the model saving and loading process with custom methods tailored for each library. Unified Interface: Regardless of the underlying library, interact with a consistent MLflow interface.  The officially supported integrations for deep learning libraries in MLflow encompass:                             Harness the power of these integrations and elevate your deep learning projects with MLflow's comprehensive support.   MLflow Tracking for Deep Learning  Tracking remains a cornerstone of the MLflow ecosystem, especially vital for the iterative nature of deep learning:  Experiments and Runs: Organize your deep learning projects into experiments, with each experiment containing multiple runs. Each run captures essential data like metrics at various training steps, hyperparameters, and the code state. Artifacts: Store vital outputs such as deep learning models, visualizations, or even tensorboard logs. This artifact repository ensures traceability and easy access. Metrics at Steps: With deep learning's iterative nature, MLflow allows logging metrics at various training steps, offering a granular view of the model's progress. Dependencies and Environment: Capture the computational environment, including deep learning frameworks' versions, ensuring reproducibility. Input Examples and Model Signatures: Define the expected format of the model's inputs, crucial for complex data like images or sequences. UI Integration: The enhanced UI provides a visual overview of deep learning runs, facilitating comparison and insights into training progress. Search Functionality: Efficiently navigate through your deep learning experiments using robust search capabilities. APIs: Interact with the tracking system programmatically, integrating deep learning workflows seamlessly.    Model Registry  A centralized repository for your deep learning models:  Versioning: Handle multiple iterations and versions of deep learning models, facilitating comparison or reversion. Annotations: Attach notes, training datasets, or other relevant metadata to models. Lifecycle Stages: Clearly define the stage of each model version, ensuring clarity in deployment and further fine-tuning.    Deployment for Deep Learning Models  Transition deep learning models from training to real-world applications:  Consistency: Ensure models, especially those with GPU dependencies, behave consistently across different deployment environments. Docker and GPU Support: Deploy in containerized environments, ensuring all dependencies, including GPU support, are encapsulated. Scalability: From deploying a single model to serving multiple distributed deep learning models, MLflow scales as per your requirements.         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
traditional-ml/index.html,"   Documentation  Traditional ML       Traditional ML  In the dynamic landscape of machine learning, traditional techniques remain foundational, playing pivotal roles across various industries and research institutions. From the precision of classification algorithms in healthcare diagnostics to the predictive prowess of regression models in finance, and from the forecasting capabilities of time-series analyses in supply chain management to the insights drawn from statistical modeling in social sciences, these core methodologies underscore many of the technological advancements we witness today. MLflow recognizes the enduring significance of traditional machine learning. Designed with precision and a deep understanding of the challenges and intricacies faced by data scientists and ML practitioners, MLflow offers a comprehensive suite of tools tailor-made for these classic techniques. This platform not only streamlines the model development and deployment processes but also ensures reproducibility, scalability, and traceability. As we delve further, we'll explore the multifaceted functionalities MLflow offers, showcasing how it enhances the efficacy, reliability, and insights derived from traditional ML models. Whether you're a seasoned expert looking to optimize workflows or a newcomer eager to make a mark, MLflow stands as an invaluable ally in your machine learning journey.  Native Library Support  There are a number of natively supported traditional ML libraries within MLflow. Throughout the documentation, you may see these referred to as "flavors", as they are specific implementations of native support for saving, logging, loading, and generic python function representation for the models that are produced from these libraries. There are distinct benefits to using the native versions of these implementations, as many have auto-logging functionality built in, as well as specific custom handling with serialization and deserialization that can greatly simplify your MLOps experiences when using these libraries. The officially supported integrations for traditional ML libraries include:                                         Tutorials and Guides                           Hyperparameter Tuning with MLflow and Optuna                                       Explore the integration of MLflow Tracking with Optuna for hyperparameter tuning. Dive into the capabilities of MLflow,                     understand parent-child run relationships, and compare different tuning runs to optimize model performance.                                           Custom Pyfunc Models with MLflow                                       Dive deep into the world of MLflow's Custom Pyfunc. Starting with basic model definitions, embark on a journey that                     showcases the versatility and power of Pyfunc. From simple mathematical curves to complex machine learning integrations,                     discover how Pyfunc offers standardized, reproducible, and efficient workflows for a variety of use cases.                         MLflow Tracking  Tracking is central to the MLflow ecosystem, facilitating the systematic organization of experiments and runs:  Experiments and Runs: Each experiment encapsulates a specific aspect of your research, and each experiment can house multiple runs. Runs document critical data like metrics, parameters, and the code state. Artifacts: Store crucial output from runs, be it models, visualizations, datasets, or other metadata. This repository of artifacts ensures traceability and easy access. Metrics and Parameters: By allowing users to log parameters and metrics, MLflow makes it straightforward to compare different runs, facilitating model optimization. Dependencies and Environment: The platform automatically captures the computational environment, ensuring that experiments are reproducible across different setups. Input Examples and Model Signatures: These features allow developers to define the expected format of the model's inputs, making validation and debugging more straightforward. UI Integration: The integrated UI provides a visual overview of all runs, enabling easy comparison and deeper insights. Search Functionality: Efficiently sift through your experiments using MLflow's robust search functionality. APIs: Comprehensive APIs are available, allowing users to interact with the tracking system programmatically, integrating it into existing workflows.    MLflow Recipes  Recipes in MLflow are predefined templates tailored for specific tasks:  Reduced Boilerplate: These templates help eliminate repetitive setup or initialization code, speeding up development. Best Practices: MLflow's recipes are crafted keeping best practices in mind, ensuring that users are aligned with industry standards right from the get-go. Customizability: While recipes provide a structured starting point, they're designed to be flexible, accommodating tweaks and modifications as needed.    MLflow Evaluate  Ensuring model quality is paramount:  Auto-generated Metrics: MLflow automatically evaluates models, providing key metrics for regression (like RMSE, MAE) and classification (such as F1-score, AUC-ROC). Visualization: Understand your model better with automatically generated plots. For instance, MLflow can produce confusion matrices, precision-recall curves, and more for classification tasks. Extensibility: While MLflow provides a rich set of evaluation tools out of the box, it's also designed to accommodate custom metrics and visualizations.    Model Registry  This feature acts as a catalog for models:  Versioning: As models evolve, keeping track of versions becomes crucial. The Model Registry handles versioning, ensuring that users can revert to older versions or compare different iterations. Annotations: Models in the registry can be annotated with descriptions, use-cases, or other relevant metadata. Lifecycle Stages: Track the stage of each model version, be it 'staging', 'production', or 'archived'. This ensures clarity in deployment and maintenance processes.    Deployment  MLflow simplifies the transition from development to production:  Consistency: By meticulously recording dependencies and the computational environment, MLflow ensures that models behave consistently across different deployment setups. Docker Support: Facilitate deployment in containerized environments using Docker, encapsulating all dependencies and ensuring a uniform runtime environment. Scalability: MLflow is designed to accommodate both small-scale deployments and large, distributed setups, ensuring that it scales with your needs.         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
deployment/index.html,"   Documentation  Deployment       Deployment  In the modern age of machine learning, deploying models effectively and consistently plays a pivotal role. The capability to serve predictions at scale, manage dependencies, and ensure reproducibility is paramount for businesses to derive actionable insights from their ML models. Whether it's for real-time predictions, batch processing, or interactive analyses, a robust model serving framework is essential. MLflow offers a comprehensive suite tailored for seamless model deployment. With its focus on ease of use, consistency, and adaptability, MLflow simplifies the model serving process, ensuring models are not just artifacts but actionable tools for decision-making.  The Power of MLflow in Model Serving   Dependency and Environment Management: MLflow ensures that the deployment environment mirrors the training environment, capturing all dependencies. This guarantees that models run consistently, regardless of where they're deployed. Packaging Models and Code: With MLflow, not just the model, but any supplementary code and configurations are packaged along with the deployment container. This ensures that the model can be executed seamlessly without any missing components.    Deployment Avenues  MLflow offers multiple ways to deploy your models based on your needs:  Local Flask Server: Quickly deploy your model in a containerized local environment. This server runs a model container with the dependencies defined during model saving. Local Flask Server with MLServer: Easily deploy a containerized model along with MLServer and KServe. This powerful alternative to a base Flask server leverages all of the benefits of Seldon's framework to enhance your inference capabilities. Remote Container Serving: Once a model container has been defined and built, it can be deployed to a remote serving environment. This is especially useful for cloud deployments where providers offer elastic container execution capabilities.   AzureML AWS Sagemaker    Kubernetes: For those invested in the Kubernetes ecosystem, MLflow supports model deployment to Kubernetes clusters, ensuring scalability and resilience. Databricks Model Serving: Directly deploy models in a Databricks environment, taking advantage of Databricks' performance optimizations and integrations.    Tutorials and Guides                           Deploying a Model to Kubernetes with MLflow                                       Explore an end-to-end guide on using MLflow to train a linear regression model, package it, and deploy it to a Kubernetes cluster.                     Understand how MLflow simplifies the entire process, from training to serving.                         Conclusion  Model serving is an intricate process, and MLflow is designed to make it as intuitive and reliable as possible. With its myriad deployment options and focus on consistency, MLflow ensures that models are ready for action, be it in a local environment, the cloud, or on a large-scale Kubernetes cluster. Dive into the provided tutorials, explore the functionalities, and streamline your model deployment journey with MLflow.        Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
tracking.html,"   Documentation  MLflow Tracking       MLflow Tracking  The MLflow Tracking component is an API and UI for logging parameters, code versions, metrics, and output files when running your machine learning code and for later visualizing the results. MLflow Tracking lets you log and query experiments using Python, REST, R API, and Java API APIs.  Table of Contents  Concepts Where Runs Are Recorded How runs and artifacts are recorded  Scenario 1: MLflow on localhost Scenario 2: MLflow on localhost with SQLite Scenario 3: MLflow on localhost with Tracking Server Scenario 4: MLflow with remote Tracking Server, backend and artifact stores Scenario 5: MLflow Tracking Server enabled with proxied artifact storage access Scenario 6: MLflow Tracking Server used exclusively as proxied access host for artifact storage access   Logging Data to Runs  Logging functions Launching Multiple Runs in One Program Performance Tracking with Metrics Visualizing Metrics   Automatic Logging  Scikit-learn Keras Gluon XGBoost LightGBM Statsmodels Spark Fastai Pytorch   Organizing Runs in Experiments  Managing Experiments and Runs with the Tracking Service API   Tracking UI Querying Runs Programmatically MLflow Tracking Servers  Storage Networking Using the Tracking Server for proxied artifact access Logging to a Tracking Server   System Tags    Concepts  MLflow Tracking is organized around the concept of runs, which are executions of some piece of data science code. Each run records the following information:  Code VersionGit commit hash used for the run, if it was run from an MLflow Project.  Start & End TimeStart and end time of the run  SourceName of the file to launch the run, or the project name and entry point for the run if run from an MLflow Project.  ParametersKey-value input parameters of your choice. Both keys and values are strings.  MetricsKey-value metrics, where the value is numeric. Each metric can be updated throughout the course of the run (for example, to track how your model's loss function is converging), and MLflow records and lets you visualize the metric's full history.  ArtifactsOutput files in any format. For example, you can record images (for example, PNGs), models (for example, a pickled scikit-learn model), and data files (for example, a Parquet file) as artifacts.   You can record runs using MLflow Python, R, Java, and REST APIs from anywhere you run your code. For example, you can record them in a standalone program, on a remote cloud machine, or in an interactive notebook. If you record runs in an MLflow Project, MLflow remembers the project URI and source version. You can optionally organize runs into experiments, which group together runs for a specific task. You can create an experiment using the mlflow experiments CLI, with mlflow.create_experiment(), or using the corresponding REST parameters. The MLflow API and UI let you create and search for experiments. Once your runs have been recorded, you can query them using the Tracking UI or the MLflow API.   Where Runs Are Recorded  MLflow runs can be recorded to local files, to a SQLAlchemy-compatible database, or remotely to a tracking server. By default, the MLflow Python API logs runs locally to files in an mlruns directory wherever you ran your program. You can then run mlflow ui to see the logged runs. To log runs remotely, set the MLFLOW_TRACKING_URI environment variable to a tracking server's URI or call mlflow.set_tracking_uri(). There are different kinds of remote tracking URIs:  Local file path (specified as file:/my/local/dir), where data is just directly stored locally. Database encoded as <dialect>+<driver>://<username>:<password>@<host>:<port>/<database>. MLflow supports the dialects mysql, mssql, sqlite, and postgresql. For more details, see SQLAlchemy database uri. HTTP server (specified as https://my-server:5000), which is a server hosting an MLflow tracking server. Databricks workspace (specified as databricks or as databricks://<profileName>, a Databricks CLI profile. Refer to Access the MLflow tracking server from outside Databricks [AWS] [Azure], or the quickstart to easily get started with hosted MLflow on Databricks Community Edition.    How runs and artifacts are recorded  As mentioned above, MLflow runs can be recorded to local files, to a SQLAlchemy-compatible database, or remotely to a tracking server. MLflow artifacts can be persisted to local files and a variety of remote file storage solutions. For storing runs and artifacts, MLflow uses two components for storage: backend store and artifact store. While the backend store persists MLflow entities (runs, parameters, metrics, tags, notes, metadata, etc), the artifact store persists artifacts (files, models, images, in-memory objects, or model summary, etc). The MLflow server can be configured with an artifacts HTTP proxy, passing artifact requests through the tracking server to store and retrieve artifacts without having to interact with underlying object store services. Usage of the proxied artifact access feature is described in Scenarios 5 and 6 below. The MLflow client can interface with a variety of backend and artifact storage configurations. Here are four common configuration scenarios:  Scenario 1: MLflow on localhost  Many developers run MLflow on their local machine, where both the backend and artifact store share a directory on the local filesystem—./mlruns—as shown in the diagram. The MLflow client directly interfaces with an instance of a FileStore and LocalArtifactRepository.    In this simple scenario, the MLflow client uses the following interfaces to record MLflow entities and artifacts:   An instance of a LocalArtifactRepository (to store artifacts) An instance of a FileStore (to save MLflow entities)     Scenario 2: MLflow on localhost with SQLite  Many users also run MLflow on their local machines with a SQLAlchemy-compatible database: SQLite. In this case, artifacts are stored under the local ./mlruns directory, and MLflow entities are inserted in a SQLite database file mlruns.db.    In this scenario, the MLflow client uses the following interfaces to record MLflow entities and artifacts:   An instance of a LocalArtifactRepository (to save artifacts) An instance of an SQLAlchemyStore (to store MLflow entities to a SQLite file mlruns.db)     Scenario 3: MLflow on localhost with Tracking Server  Similar to scenario 1 but a tracking server is launched, listening for REST request calls at the default port 5000. The arguments supplied to the mlflow server <args> dictate what backend and artifact stores are used. The default is local FileStore. For example, if a user launched a tracking server as mlflow server --backend-store-uri sqlite:///mydb.sqlite, then SQLite would be used for backend storage instead. As in scenario 1, MLflow uses a local mlruns filesystem directory as a backend store and artifact store. With a tracking server running, the MLflow client interacts with the tracking server via REST requests, as shown in the diagram.     Command to run the tracking server in this configuration  mlflow server --backend-store-uri file:///path/to/mlruns --no-serve-artifacts    To store all runs' MLflow entities, the MLflow client interacts with the tracking server via a series of REST requests:   Part 1a and b:    The MLflow client creates an instance of a RestStore and sends REST API requests to log MLflow entities The Tracking Server creates an instance of a FileStore to save MLflow entities and writes directly to the local mlruns directory    For the artifacts, the MLflow client interacts with the tracking server via a REST request:   Part 2a, b, and c:  The MLflow client uses RestStore to send a REST request to fetch the artifact store URI location The Tracking Server responds with an artifact store URI location The MLflow client creates an instance of a LocalArtifactRepository and saves artifacts to the local filesystem location specified by the artifact store URI (a subdirectory of mlruns)       Scenario 4: MLflow with remote Tracking Server, backend and artifact stores  MLflow also supports distributed architectures, where the tracking server, backend store, and artifact store reside on remote hosts. This example scenario depicts an architecture with a remote MLflow Tracking Server, a Postgres database for backend entity storage, and an S3 bucket for artifact storage.     Command to run the tracking server in this configuration  mlflow server --backend-store-uri postgresql://user:password@postgres:5432/mlflowdb --default-artifact-root s3://bucket_name --host remote_host --no-serve-artifacts    To record all runs' MLflow entities, the MLflow client interacts with the tracking server via a series of REST requests:   Part 1a and b:    The MLflow client creates an instance of a RestStore and sends REST API requests to log MLflow entities The Tracking Server creates an instance of an SQLAlchemyStore and connects to the remote host to insert MLflow entities in the database    For artifact logging, the MLflow client interacts with the remote Tracking Server and artifact storage host:   Part 2a, b, and c:    The MLflow client uses RestStore to send a REST request to fetch the artifact store URI location from the Tracking Server The Tracking Server responds with an artifact store URI location (an S3 storage URI in this case) The MLflow client creates an instance of an S3ArtifactRepository, connects to the remote AWS host using the boto client libraries, and uploads the artifacts to the S3 bucket URI location    The FileStore, RestStore, and SQLAlchemyStore are concrete implementations of the abstract class AbstractStore, and the LocalArtifactRepository and S3ArtifactRepository are concrete implementations of the abstract class ArtifactRepository.   Scenario 5: MLflow Tracking Server enabled with proxied artifact storage access  MLflow's Tracking Server supports utilizing the host as a proxy server for operations involving artifacts. Once configured with the appropriate access requirements, an administrator can start the tracking server to enable assumed-role operations involving the saving, loading, or listing of model artifacts, images, documents, and files. This eliminates the need to allow end users to have direct path access to a remote object store (e.g., s3, adls, gcs, hdfs) for artifact handling and eliminates the need for an end-user to provide access credentials to interact with an underlying object store.     Command to run the tracking server in this configuration  mlflow server \   --backend-store-uri postgresql://user:password@postgres:5432/mlflowdb \   # Artifact access is enabled through the proxy URI 'mlflow-artifacts:/',   # giving users access to this location without having to manage credentials   # or permissions.   --artifacts-destination s3://bucket_name \   --host remote_host    Enabling the Tracking Server to perform proxied artifact access in order to route client artifact requests to an object store location:   Part 1a and b:    The MLflow client creates an instance of a RestStore and sends REST API requests to log MLflow entities The Tracking Server creates an instance of an SQLAlchemyStore and connects to the remote host for inserting tracking information in the database (i.e., metrics, parameters, tags, etc.)    Part 1c and d:    Retrieval requests by the client return information from the configured SQLAlchemyStore table    Part 2a and b:    Logging events for artifacts are made by the client using the HttpArtifactRepository to write files to MLflow Tracking Server The Tracking Server then writes these files to the configured object store location with assumed role authentication    Part 2c and d:    Retrieving artifacts from the configured backend store for a user request is done with the same authorized authentication that was configured at server start Artifacts are passed to the end user through the Tracking Server through the interface of the HttpArtifactRepository     Note When an experiment is created, the artifact storage location from the configuration of the tracking server is logged in the experiment's metadata. When enabling proxied artifact storage, any existing experiments that were created while operating a tracking server in non-proxied mode will continue to use a non-proxied artifact location. In order to use proxied artifact logging, a new experiment must be created. If the intention of enabling a tracking server in -serve-artifacts mode is to eliminate the need for a client to have authentication to the underlying storage, new experiments should be created for use by clients so that the tracking server can handle authentication after this migration.   Warning The MLflow artifact proxied access service enables users to have an assumed role of access to all artifacts that are accessible to the Tracking Server. Administrators who are enabling this feature should ensure that the access level granted to the Tracking Server for artifact operations meets all security requirements prior to enabling the Tracking Server to operate in a proxied file handling role.    Scenario 6: MLflow Tracking Server used exclusively as proxied access host for artifact storage access  MLflow's Tracking Server can be used in an exclusive artifact proxied artifact handling role. Specifying the --artifacts-only flag restricts an MLflow server instance to only serve artifact-related API requests by proxying to an underlying object store.  Note Starting a Tracking Server with the --artifacts-only parameter will disable all Tracking Server functionality apart from API calls related to saving, loading, or listing artifacts. Creating runs, logging metrics or parameters, and accessing other attributes about experiments are all not permitted in this mode.      Command to run the tracking server in this configuration  mlflow server --artifacts-destination s3://bucket_name --artifacts-only --host remote_host    Running an MLflow server in --artifacts-only mode:   Part 1a and b:    The MLflow client will interact with the Tracking Server using the HttpArtifactRepository interface. Listing artifacts associated with a run will be conducted from the Tracking Server using the access credentials set at server startup Saving of artifacts will transmit the files to the Tracking Server which will then write the files to the file store using credentials set at server start.    Part 1c and d:    Listing of artifact responses will pass from the file store through the Tracking Server to the client Loading of artifacts will utilize the access credentials of the MLflow Tracking Server to acquire the files which are then passed on to the client     Note  If migrating from Scenario 5 to Scenario 6 due to request volumes, it is important to perform two validations:  Ensure that the new tracking server that is operating in --artifacts-only mode has access permissions to the location set by --artifacts-destination that the former multi-role tracking server had. The former multi-role tracking server that was serving artifacts must have the -serve-artifacts argument disabled.      Warning Operating the Tracking Server in proxied artifact access mode by setting the parameter --serve-artifacts during server start, even in --artifacts-only mode, will give access to artifacts residing on the object store to any user that has authentication to access the Tracking Server. Ensure that any per-user security posture that you are required to maintain is applied accordingly to the proxied access that the Tracking Server will have in this mode of operation.     Logging Data to Runs  You can log data to runs using the MLflow Python, R, Java, or REST API. This section shows the Python API.  In this section:  Logging functions Launching Multiple Runs in One Program Performance Tracking with Metrics Visualizing Metrics    Logging functions  mlflow.set_tracking_uri() connects to a tracking URI. You can also set the MLFLOW_TRACKING_URI environment variable to have MLflow find a URI from there. In both cases, the URI can either be a HTTP/HTTPS URI for a remote server, a database connection string, or a local path to log data to a directory. The URI defaults to mlruns. mlflow.get_tracking_uri() returns the current tracking URI. mlflow.create_experiment() creates a new experiment and returns its ID. Runs can be launched under the experiment by passing the experiment ID to mlflow.start_run. mlflow.set_experiment() sets an experiment as active. If the experiment does not exist, creates a new experiment. If you do not specify an experiment in mlflow.start_run(), new runs are launched under this experiment. mlflow.start_run() returns the currently active run (if one exists), or starts a new run and returns a mlflow.ActiveRun object usable as a context manager for the current run. You do not need to call start_run explicitly: calling one of the logging functions with no active run automatically starts a new one.  Note  If the argument run_name is not set within mlflow.start_run(), a unique run name will be generated for each run.   mlflow.end_run() ends the currently active run, if any, taking an optional run status. mlflow.active_run() returns a mlflow.entities.Run object corresponding to the currently active run, if any. Note: You cannot access currently-active run attributes (parameters, metrics, etc.) through the run returned by mlflow.active_run. In order to access such attributes, use the MlflowClient as follows: client = mlflow.MlflowClient() data = client.get_run(mlflow.active_run().info.run_id).data   mlflow.last_active_run() returns a mlflow.entities.Run object corresponding to the currently active run, if any. Otherwise, it returns a mlflow.entities.Run object corresponding the last run started from the current Python process that reached a terminal status (i.e. FINISHED, FAILED, or KILLED). mlflow.get_parent_run() returns a mlflow.entities.Run object corresponding to the parent run for the given run id, if one exists. Otherwise, it returns None. mlflow.log_param() logs a single key-value param in the currently active run. The key and value are both strings. Use mlflow.log_params() to log multiple params at once. mlflow.log_metric() logs a single key-value metric. The value must always be a number. MLflow remembers the history of values for each metric. Use mlflow.log_metrics() to log multiple metrics at once. mlflow.log_input() logs a single mlflow.data.dataset.Dataset object corresponding to the currently active run. You may also log a dataset context string and a dict of key-value tags. mlflow.set_tag() sets a single key-value tag in the currently active run. The key and value are both strings. Use mlflow.set_tags() to set multiple tags at once. mlflow.log_artifact() logs a local file or directory as an artifact, optionally taking an artifact_path to place it in within the run's artifact URI. Run artifacts can be organized into directories, so you can place the artifact in a directory this way. mlflow.log_artifacts() logs all the files in a given directory as artifacts, again taking an optional artifact_path. mlflow.get_artifact_uri() returns the URI that artifacts from the current run should be logged to.   Launching Multiple Runs in One Program  Sometimes you want to launch multiple MLflow runs in the same program: for example, maybe you are performing a hyperparameter search locally or your experiments are just very fast to run. This is easy to do because the ActiveRun object returned by mlflow.start_run() is a Python context manager. You can "scope" each run to just one block of code as follows: with mlflow.start_run():     mlflow.log_param(""x"", 1)     mlflow.log_metric(""y"", 2)     ...   The run remains open throughout the with statement, and is automatically closed when the statement exits, even if it exits due to an exception.   Performance Tracking with Metrics  You log MLflow metrics with log methods in the Tracking API. The log methods support two alternative methods for distinguishing metric values on the x-axis: timestamp and step. timestamp is an optional long value that represents the time that the metric was logged. timestamp defaults to the current time. step is an optional integer that represents any measurement of training progress (number of training iterations, number of epochs, and so on). step defaults to 0 and has the following requirements and properties:  Must be a valid 64-bit integer value. Can be negative. Can be out of order in successive write calls. For example, (1, 3, 2) is a valid sequence. Can have "gaps" in the sequence of values specified in successive write calls. For example, (1, 5, 75, -20) is a valid sequence.  If you specify both a timestamp and a step, metrics are recorded against both axes independently.  Examples   Pythonwith mlflow.start_run():     for epoch in range(0, 3):         mlflow.log_metric(key=""quality"", value=2 * epoch, step=epoch)    Java and ScalaMlflowClient client = new MlflowClient(); RunInfo run = client.createRun(); for (int epoch = 0; epoch < 3; epoch ++) {     client.logMetric(run.getRunId(), ""quality"", 2 * epoch, System.currentTimeMillis(), epoch); }        Visualizing Metrics  Here is an example plot of the quick start tutorial with the step x-axis and two timestamp axes:   X-axis step     X-axis wall time - graphs the absolute time each metric was logged     X-axis relative time - graphs the time relative to the first metric logged, for each run      Automatic Logging  Automatic logging allows you to log metrics, parameters, and models without the need for explicit log statements. There are two ways to use autologging:  Call mlflow.autolog() before your training code. This will enable autologging for each supported library you have installed as soon as you import it. Use library-specific autolog calls for each library you use in your code. See below for examples.  The following libraries support autologging:   Scikit-learn Keras Gluon XGBoost LightGBM Statsmodels Spark Fastai Pytorch   For flavors that automatically save models as an artifact, additional files for dependency management are logged. You can access the most recent autolog run through the mlflow.last_active_run() function. Here's a short sklearn autolog example that makes use of this function: import mlflow  from sklearn.model_selection import train_test_split from sklearn.datasets import load_diabetes from sklearn.ensemble import RandomForestRegressor  mlflow.autolog()  db = load_diabetes() X_train, X_test, y_train, y_test = train_test_split(db.data, db.target)  # Create and train models. rf = RandomForestRegressor(n_estimators=100, max_depth=6, max_features=3) rf.fit(X_train, y_train)  # Use the model to make predictions on the test dataset. predictions = rf.predict(X_test) autolog_run = mlflow.last_active_run()    Scikit-learn  Call mlflow.sklearn.autolog() before your training code to enable automatic logging of sklearn metrics, params, and models. See example usage here. Autologging for estimators (e.g. LinearRegression) and meta estimators (e.g. Pipeline) creates a single run and logs:         Metrics Parameters Tags Artifacts  Training score obtained by estimator.score Parameters obtained by estimator.get_params  Class name Fully qualified class name   Fitted estimator    Autologging for parameter search estimators (e.g. GridSearchCV) creates a single parent run and nested child runs - Parent run   - Child run 1   - Child run 2   - ...   containing the following data:          Run type Metrics Parameters Tags Artifacts  Parent Training score  Parameter search estimator's parameters Best parameter combination    Class name Fully qualified class name    Fitted parameter search estimator Fitted best estimator Search results csv file    Child CV test score for each parameter combination Each parameter combination  Class name Fully qualified class name   –      Keras  Call mlflow.tensorflow.autolog() before your training code to enable automatic logging of metrics and parameters. As an example, try running the Keras/Tensorflow example. Note that only tensorflow>=2.3 are supported. The respective metrics associated with tf.estimator and EarlyStopping are automatically logged. As an example, try running the Keras/TensorFlow example. Autologging captures the following information:          Framework/module Metrics Parameters Tags Artifacts  tf.keras Training loss; validation loss; user-specified metrics fit() parameters; optimizer name; learning rate; epsilon – Model summary on training start; MLflow Model (Keras model); TensorBoard logs on training end  tf.keras.callbacks.EarlyStopping Metrics from the EarlyStopping callbacks. For example, stopped_epoch, restored_epoch, restore_best_weight, etc fit() parameters from EarlyStopping. For example, min_delta, patience, baseline, restore_best_weights, etc – –    If no active run exists when autolog() captures data, MLflow will automatically create a run to log information to. Also, MLflow will then automatically end the run once training ends via calls to tf.keras.fit(). If a run already exists when autolog() captures data, MLflow will log to that run but not automatically end that run after training.   Gluon  Call mlflow.gluon.autolog() before your training code to enable automatic logging of metrics and parameters. See example usages with Gluon . Autologging captures the following information:          Framework Metrics Parameters Tags Artifacts  Gluon Training loss; validation loss; user-specified metrics Number of layers; optimizer name; learning rate; epsilon – MLflow Model (Gluon model); on training end      XGBoost  Call mlflow.xgboost.autolog() before your training code to enable automatic logging of metrics and parameters. Autologging captures the following information:          Framework Metrics Parameters Tags Artifacts  XGBoost user-specified metrics xgboost.train parameters – MLflow Model (XGBoost model) with model signature on training end; feature importance; input example    If early stopping is activated, metrics at the best iteration will be logged as an extra step/iteration.   LightGBM  Call mlflow.lightgbm.autolog() before your training code to enable automatic logging of metrics and parameters. Autologging captures the following information:          Framework Metrics Parameters Tags Artifacts  LightGBM user-specified metrics lightgbm.train parameters – MLflow Model (LightGBM model) with model signature on training end; feature importance; input example    If early stopping is activated, metrics at the best iteration will be logged as an extra step/iteration.   Statsmodels  Call mlflow.statsmodels.autolog() before your training code to enable automatic logging of metrics and parameters. Autologging captures the following information:          Framework Metrics Parameters Tags Artifacts  Statsmodels user-specified metrics statsmodels.base.model.Model.fit parameters – MLflow Model (statsmodels.base.wrapper.ResultsWrapper) on training end     Note  Each model subclass that overrides fit expects and logs its own parameters.     Spark  Initialize a SparkSession with the mlflow-spark JAR attached (e.g. SparkSession.builder.config(""spark.jars.packages"", ""org.mlflow.mlflow-spark"")) and then call mlflow.spark.autolog() to enable automatic logging of Spark datasource information at read-time, without the need for explicit log statements. Note that autologging of Spark ML (MLlib) models is not yet supported. Autologging captures the following information:          Framework Metrics Parameters Tags Artifacts  Spark – – Single tag containing source path, version, format. The tag contains one line per datasource –     Note  Moreover, Spark datasource autologging occurs asynchronously - as such, it's possible (though unlikely) to see race conditions when launching short-lived MLflow runs that result in datasource information not being logged.     Fastai  Call mlflow.fastai.autolog() before your training code to enable automatic logging of metrics and parameters. See an example usage with Fastai. Autologging captures the following information:          Framework Metrics Parameters Tags Artifacts  fastai user-specified metrics Logs optimizer data as parameters. For example, epochs, lr, opt_func, etc; Logs the parameters of the EarlyStoppingCallback and OneCycleScheduler callbacks – Model checkpoints are logged to a 'models' directory; MLflow Model (fastai Learner model) on training end; Model summary text is logged      Pytorch  Call mlflow.pytorch.autolog() before your Pytorch Lightning training code to enable automatic logging of metrics, parameters, and models. See example usages here. Note that currently, Pytorch autologging supports only models trained using Pytorch Lightning. Autologging is triggered on calls to pytorch_lightning.trainer.Trainer.fit and captures the following information:          Framework/module Metrics Parameters Tags Artifacts  pytorch_lightning.trainer.Trainer Training loss; validation loss; average_test_accuracy; user-defined-metrics. fit() parameters; optimizer name; learning rate; epsilon. – Model summary on training start, MLflow Model (Pytorch model) on training end;  pytorch_lightning.callbacks.earlystopping Training loss; validation loss; average_test_accuracy; user-defined-metrics. Metrics from the EarlyStopping callbacks. For example, stopped_epoch, restored_epoch, restore_best_weight, etc. fit() parameters; optimizer name; learning rate; epsilon Parameters from the EarlyStopping callbacks. For example, min_delta, patience, baseline,``restore_best_weights``, etc – Model summary on training start; MLflow Model (Pytorch model) on training end; Best Pytorch model checkpoint, if training stops due to early stopping callback.    If no active run exists when autolog() captures data, MLflow will automatically create a run to log information, ending the run once the call to pytorch_lightning.trainer.Trainer.fit() completes. If a run already exists when autolog() captures data, MLflow will log to that run but not automatically end that run after training.  Note  Parameters not explicitly passed by users (parameters that use default values) while using pytorch_lightning.trainer.Trainer.fit() are not currently automatically logged In case of a multi-optimizer scenario (such as usage of autoencoder), only the parameters for the first optimizer are logged      Organizing Runs in Experiments  MLflow allows you to group runs under experiments, which can be useful for comparing runs intended to tackle a particular task. You can create experiments using the Command-Line Interface (mlflow experiments) or the mlflow.create_experiment() Python API. You can pass the experiment name for an individual run using the CLI (for example, mlflow run ... --experiment-name [name]) or the MLFLOW_EXPERIMENT_NAME environment variable. Alternatively, you can use the experiment ID instead, via the --experiment-id CLI flag or the MLFLOW_EXPERIMENT_ID environment variable. # Set the experiment via environment variables export MLFLOW_EXPERIMENT_NAME=fraud-detection  mlflow experiments create --experiment-name fraud-detection   # Launch a run. The experiment is inferred from the MLFLOW_EXPERIMENT_NAME environment # variable, or from the --experiment-name parameter passed to the MLflow CLI (the latter # taking precedence) with mlflow.start_run():     mlflow.log_param(""a"", 1)     mlflow.log_metric(""b"", 2)    Managing Experiments and Runs with the Tracking Service API  MLflow provides a more detailed Tracking Service API for managing experiments and runs directly, which is available through client SDK in the mlflow.client module. This makes it possible to query data about past runs, log additional information about them, create experiments, add tags to a run, and more. Example from mlflow.tracking import MlflowClient  client = MlflowClient() experiments = (     client.search_experiments() )  # returns a list of mlflow.entities.Experiment run = client.create_run(experiments[0].experiment_id)  # returns mlflow.entities.Run client.log_param(run.info.run_id, ""hello"", ""world"") client.set_terminated(run.info.run_id)    Adding Tags to Runs  The MlflowClient.set_tag() function lets you add custom tags to runs. A tag can only have a single unique value mapped to it at a time. For example: client.set_tag(run.info.run_id, ""tag_key"", ""tag_value"")    Important Do not use the prefix mlflow. (e.g. mlflow.note) for a tag.  This prefix is reserved for use by MLflow. See System Tags for a list of reserved tag keys.      Tracking UI  The Tracking UI lets you visualize, search and compare runs, as well as download run artifacts or metadata for analysis in other tools. If you log runs to a local mlruns directory, run mlflow ui in the directory above it, and it loads the corresponding runs. Alternatively, the MLflow tracking server serves the same UI and enables remote storage of run artifacts. In that case, you can view the UI using URL http://<ip address of your MLflow tracking server>:5000 in your browser from any machine, including any remote machine that can connect to your tracking server. The UI contains the following key features:  Experiment-based run listing and comparison (including run comparison across multiple experiments) Searching for runs by parameter or metric value Visualizing run metrics Downloading run results    Querying Runs Programmatically  You can access all of the functions in the Tracking UI programmatically. This makes it easy to do several common tasks:  Query and compare runs using any data analysis tool of your choice, for example, pandas. Determine the artifact URI for a run to feed some of its artifacts into a new run when executing a workflow. For an example of querying runs and constructing a multistep workflow, see the MLflow Multistep Workflow Example project. Load artifacts from past runs as MLflow Models. For an example of training, exporting, and loading a model, and predicting using the model, see the MLflow Keras/TensorFlow example. Run automated parameter search algorithms, where you query the metrics from various runs to submit new ones. For an example of running automated parameter search algorithms, see the MLflow Hyperparameter Tuning Example project.    MLflow Tracking Servers   In this section:  Storage  Backend Stores Artifact Stores File store performance Deletion Behavior SQLAlchemy Options   Networking Using the Tracking Server for proxied artifact access  Optionally using a Tracking Server instance exclusively for artifact handling   Logging to a Tracking Server  Tracking Server versioning     You run an MLflow tracking server using mlflow server.  An example configuration for a server is:  Command to run the tracking server in this configuration  mlflow server \     --backend-store-uri /mnt/persistent-disk \     --default-artifact-root s3://my-mlflow-bucket/ \     --host 0.0.0.0     Note When started in --artifacts-only mode, the tracking server will not permit any operation other than saving, loading, and listing artifacts.   Storage  An MLflow tracking server has two components for storage: a backend store and an artifact store.  Backend Stores  The backend store is where MLflow Tracking Server stores experiment and run metadata as well as params, metrics, and tags for runs. MLflow supports two types of backend stores: file store and database-backed store.  Note In order to use model registry functionality, you must run your server using a database-backed store.  Use --backend-store-uri to configure the type of backend store. You specify:  A file store backend as ./path_to_store or file:/path_to_store A database-backed store as SQLAlchemy database URI. The database URI typically takes the format <dialect>+<driver>://<username>:<password>@<host>:<port>/<database>. MLflow supports the database dialects mysql, mssql, sqlite, and postgresql. Drivers are optional. If you do not specify a driver, SQLAlchemy uses a dialect's default driver. For example, --backend-store-uri sqlite:///mlflow.db would use a local SQLite database.   Important mlflow server will fail against a database-backed store with an out-of-date database schema. To prevent this, upgrade your database schema to the latest supported version using mlflow db upgrade [db_uri]. Schema migrations can result in database downtime, may take longer on larger databases, and are not guaranteed to be transactional. You should always take a backup of your database prior to running mlflow db upgrade - consult your database's documentation for instructions on taking a backup.   Note 2d6e25af4d3e_increase_max_param_val_length is a non-invertible migration script that increases the param value length to 8k (but we limit param value max length to 6000 internally). Please be careful if you want to upgrade and backup your database before upgrading.  By default --backend-store-uri is set to the local ./mlruns directory (the same as when running mlflow run locally), but when running a server, make sure that this points to a persistent (that is, non-ephemeral) file system location.   Artifact Stores   In this section:  Amazon S3 and S3-compatible storage Azure Blob Storage Google Cloud Storage FTP server SFTP Server NFS HDFS   The artifact store is a location suitable for large data (such as an S3 bucket or shared NFS file system) and is where clients log their artifact output (for example, models). artifact_location is a property recorded on mlflow.entities.Experiment for default location to store artifacts for all runs in this experiment. Additionally, artifact_uri is a property on mlflow.entities.RunInfo to indicate location where all artifacts for this run are stored. The MLflow client caches artifact location information on a per-run basis. It is therefore not recommended to alter a run's artifact location before it has terminated. In addition to local file paths, MLflow supports the following storage systems as artifact stores: Amazon S3, Azure Blob Storage, Google Cloud Storage, SFTP server, and NFS. Use --default-artifact-root (defaults to local ./mlruns directory) to configure default location to server's artifact store. This will be used as artifact location for newly-created experiments that do not specify one. Once you create an experiment, --default-artifact-root is no longer relevant to that experiment. By default, a server is launched with the --serve-artifacts flag to enable proxied access for artifacts. The uri mlflow-artifacts:/ replaces an otherwise explicit object store destination (e.g., "s3:/my_bucket/mlartifacts") for interfacing with artifacts. The client can access artifacts via HTTP requests to the MLflow Tracking Server. This simplifies access requirements for users of the MLflow client, eliminating the need to configure access tokens or username and password environment variables for the underlying object store when writing or retrieving artifacts. To disable proxied access for artifacts, specify --no-serve-artifacts. Provided an MLflow server configuration where the --default-artifact-root is s3://my-root-bucket, the following patterns will all resolve to the configured proxied object store location of s3://my-root-bucket/mlartifacts:   https://<host>:<port>/mlartifacts http://<host>/mlartifacts mlflow-artifacts://<host>/mlartifacts mlflow-artifacts://<host>:<port>/mlartifacts mlflow-artifacts:/mlartifacts   If the host or host:port declaration is absent in client artifact requests to the MLflow server, the client API will assume that the host is the same as the MLflow Tracking uri.  Note If an MLflow server is running with the --artifact-only flag, the client should interact with this server explicitly by including either a host or host:port definition for uri location references for artifacts. Otherwise, all artifact requests will route to the MLflow Tracking server, defeating the purpose of running a distinct artifact server.   Important Access credentials and configuration for the artifact storage location are configured once during server initialization in the place of having users handle access credentials for artifact-based operations. Note that all users who have access to the Tracking Server in this mode will have access to artifacts served through this assumed role.  To allow the server and clients to access the artifact location, you should configure your cloud provider credentials as normal. For example, for S3, you can set the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables, use an IAM role, or configure a default profile in ~/.aws/credentials. See Set up AWS Credentials and Region for Development for more info.  Important If you do not specify a --default-artifact-root or an artifact URI when creating the experiment (for example, mlflow experiments create --artifact-location s3://<my-bucket>), the artifact root is a path inside the file store. Typically this is not an appropriate location, as the client and server probably refer to different physical locations (that is, the same path on different disks).  You may set an MLflow environment variable to configure the timeout for artifact uploads and downloads:  MLFLOW_ARTIFACT_UPLOAD_DOWNLOAD_TIMEOUT - (Experimental, may be changed or removed) Sets the timeout for artifact upload/download in seconds (Default set by individual artifact stores).   Amazon S3 and S3-compatible storage  To store artifacts in S3 (whether on Amazon S3 or on an S3-compatible alternative, such as MinIO or Digital Ocean Spaces), specify a URI of the form s3://<bucket>/<path>. MLflow obtains credentials to access S3 from your machine's IAM role, a profile in ~/.aws/credentials, or the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY depending on which of these are available. For more information on how to set credentials, see Set up AWS Credentials and Region for Development. To add S3 file upload extra arguments, set MLFLOW_S3_UPLOAD_EXTRA_ARGS to a JSON object of key/value pairs. For example, if you want to upload to a KMS Encrypted bucket using the KMS Key 1234: export MLFLOW_S3_UPLOAD_EXTRA_ARGS='{""ServerSideEncryption"": ""aws:kms"", ""SSEKMSKeyId"": ""1234""}'   For a list of available extra args see Boto3 ExtraArgs Documentation. To store artifacts in a custom endpoint, set the MLFLOW_S3_ENDPOINT_URL to your endpoint's URL. For example, if you are using Digital Ocean Spaces: export MLFLOW_S3_ENDPOINT_URL=https://<region>.digitaloceanspaces.com   If you have a MinIO server at 1.2.3.4 on port 9000: export MLFLOW_S3_ENDPOINT_URL=http://1.2.3.4:9000   If the MinIO server is configured with using SSL self-signed or signed using some internal-only CA certificate, you could set MLFLOW_S3_IGNORE_TLS or AWS_CA_BUNDLE variables (not both at the same time!) to disable certificate signature check, or add a custom CA bundle to perform this check, respectively: export MLFLOW_S3_IGNORE_TLS=true #or export AWS_CA_BUNDLE=/some/ca/bundle.pem   Additionally, if MinIO server is configured with non-default region, you should set AWS_DEFAULT_REGION variable: export AWS_DEFAULT_REGION=my_region    Warning The MLflow tracking server utilizes specific reserved keywords to generate a qualified path. These environment configurations, if present in the client environment, can create path resolution issues. For example, providing --default-artifact-root $MLFLOW_S3_ENDPOINT_URL on the server side and MLFLOW_S3_ENDPOINT_URL on the client side will create a client path resolution issue for the artifact storage location. Upon resolving the artifact storage location, the MLflow client will use the value provided by --default-artifact-root and suffixes the location with the values provided in the environment variable  MLFLOW_S3_ENDPOINT_URL. Depending on the value set for the environment variable MLFLOW_S3_ENDPOINT_URL, the resulting artifact storage path for this scenario would be one of the following invalid object store paths:  https://<bucketname>.s3.<region>.amazonaws.com/<key>/<bucketname>/<key> or  s3://<bucketname>/<key>/<bucketname>/<key>. To prevent path parsing issues, ensure that reserved environment variables are removed (unset) from client environments.  Complete list of configurable values for an S3 client is available in boto3 documentation.   Azure Blob Storage  To store artifacts in Azure Blob Storage, specify a URI of the form wasbs://<container>@<storage-account>.blob.core.windows.net/<path>. MLflow expects Azure Storage access credentials in the AZURE_STORAGE_CONNECTION_STRING, AZURE_STORAGE_ACCESS_KEY environment variables or having your credentials configured such that the DefaultAzureCredential(). class can pick them up. The order of precedence is:  AZURE_STORAGE_CONNECTION_STRING AZURE_STORAGE_ACCESS_KEY DefaultAzureCredential()  You must set one of these options on both your client application and your MLflow tracking server. Also, you must run pip install azure-storage-blob separately (on both your client and the server) to access Azure Blob Storage. Finally, if you want to use DefaultAzureCredential, you must pip install azure-identity; MLflow does not declare a dependency on these packages by default. You may set an MLflow environment variable to configure the timeout for artifact uploads and downloads:  MLFLOW_ARTIFACT_UPLOAD_DOWNLOAD_TIMEOUT - (Experimental, may be changed or removed) Sets the timeout for artifact upload/download in seconds (Default: 600 for Azure blob).    Google Cloud Storage  To store artifacts in Google Cloud Storage, specify a URI of the form gs://<bucket>/<path>. You should configure credentials for accessing the GCS container on the client and server as described in the GCS documentation. Finally, you must run pip install google-cloud-storage (on both your client and the server) to access Google Cloud Storage; MLflow does not declare a dependency on this package by default. You may set some MLflow environment variables to troubleshoot GCS read-timeouts (eg. due to slow transfer speeds) using the following variables:  MLFLOW_ARTIFACT_UPLOAD_DOWNLOAD_TIMEOUT - (Experimental, may be changed or removed) Sets the standard timeout for transfer operations in seconds (Default: 60 for GCS). Use -1 for indefinite timeout. MLFLOW_GCS_DEFAULT_TIMEOUT - (Deprecated, please use MLFLOW_ARTIFACT_UPLOAD_DOWNLOAD_TIMEOUT) Sets the standard timeout for transfer operations in seconds (Default: 60). Use -1 for indefinite timeout. MLFLOW_GCS_UPLOAD_CHUNK_SIZE - Sets the standard upload chunk size for bigger files in bytes (Default: 104857600 ≙ 100MiB), must be multiple of 256 KB. MLFLOW_GCS_DOWNLOAD_CHUNK_SIZE - Sets the standard download chunk size for bigger files in bytes (Default: 104857600 ≙ 100MiB), must be multiple of 256 KB    FTP server  To store artifacts in a FTP server, specify a URI of the form ftp://user@host/path/to/directory . The URI may optionally include a password for logging into the server, e.g. ftp://user:pass@host/path/to/directory   SFTP Server  To store artifacts in an SFTP server, specify a URI of the form sftp://user@host/path/to/directory. You should configure the client to be able to log in to the SFTP server without a password over SSH (e.g. public key, identity file in ssh_config, etc.). The format sftp://user:pass@host/ is supported for logging in. However, for safety reasons this is not recommended. When using this store, pysftp must be installed on both the server and the client. Run pip install pysftp to install the required package.   NFS  To store artifacts in an NFS mount, specify a URI as a normal file system path, e.g., /mnt/nfs. This path must be the same on both the server and the client – you may need to use symlinks or remount the client in order to enforce this property.   HDFS  To store artifacts in HDFS, specify a hdfs: URI. It can contain host and port: hdfs://<host>:<port>/<path> or just the path: hdfs://<path>. There are also two ways to authenticate to HDFS:  Use current UNIX account authorization Kerberos credentials using following environment variables:  export MLFLOW_KERBEROS_TICKET_CACHE=/tmp/krb5cc_22222222 export MLFLOW_KERBEROS_USER=user_name_to_use   Most of the cluster contest settings are read from hdfs-site.xml accessed by the HDFS native driver using the CLASSPATH environment variable. The used HDFS driver is libhdfs.    File store performance  MLflow will automatically try to use LibYAML bindings if they are already installed. However if you notice any performance issues when using file store backend, it could mean LibYAML is not installed on your system. On Linux or Mac you can easily install it using your system package manager: # On Ubuntu/Debian apt-get install libyaml-cpp-dev libyaml-dev  # On macOS using Homebrew brew install yaml-cpp libyaml   After installing LibYAML, you need to reinstall PyYAML: # Reinstall PyYAML pip --no-cache-dir install --force-reinstall -I pyyaml     Deletion Behavior  In order to allow MLflow Runs to be restored, Run metadata and artifacts are not automatically removed from the backend store or artifact store when a Run is deleted. The mlflow gc CLI is provided for permanently removing Run metadata and artifacts for deleted runs.   SQLAlchemy Options  You can inject some SQLAlchemy connection pooling options using environment variables.       MLflow Environment Variable SQLAlchemy QueuePool Option  MLFLOW_SQLALCHEMYSTORE_POOL_SIZE pool_size  MLFLOW_SQLALCHEMYSTORE_POOL_RECYCLE pool_recycle  MLFLOW_SQLALCHEMYSTORE_MAX_OVERFLOW max_overflow       Networking  The --host option exposes the service on all interfaces. If running a server in production, we would recommend not exposing the built-in server broadly (as it is unauthenticated and unencrypted), and instead putting it behind a reverse proxy like NGINX or Apache httpd, or connecting over VPN. You can then pass authentication headers to MLflow using these environment variables. Additionally, you should ensure that the --backend-store-uri (which defaults to the ./mlruns directory) points to a persistent (non-ephemeral) disk or database connection.   Using the Tracking Server for proxied artifact access  To use an instance of the MLflow Tracking server for artifact operations ( Scenario 5: MLflow Tracking Server enabled with proxied artifact storage access ), start a server with the optional parameters --serve-artifacts to enable proxied artifact access and set a path to record artifacts to by providing a value for the argument --artifacts-destination. The tracking server will, in this mode, stream any artifacts that a client is logging directly through an assumed (server-side) identity, eliminating the need for access credentials to be handled by end-users.  Note Authentication access to the value set by --artifacts-destination must be configured when starting the tracking server, if required.  To start the MLflow server with proxy artifact access enabled to an HDFS location (as an example): export HADOOP_USER_NAME=mlflowserverauth  mlflow server \     --host 0.0.0.0 \     --port 8885 \     --artifacts-destination hdfs://myhost:8887/mlprojects/models \    Optionally using a Tracking Server instance exclusively for artifact handling  If the volume of tracking server requests is sufficiently large and performance issues are noticed, a tracking server can be configured to serve in --artifacts-only mode ( Scenario 6: MLflow Tracking Server used exclusively as proxied access host for artifact storage access ), operating in tandem with an instance that operates with --no-serve-artifacts specified. This configuration ensures that the processing of artifacts is isolated from all other tracking server event handling. When a tracking server is configured in --artifacts-only mode, any tasks apart from those concerned with artifact handling (i.e., model logging, loading models, logging artifacts, listing artifacts, etc.) will return an HTTPError. See the following example of a client REST call in Python attempting to list experiments from a server that is configured in --artifacts-only mode: import requests  response = requests.get(""http://0.0.0.0:8885/api/2.0/mlflow/experiments/list"")   Output >> HTTPError: Endpoint: /api/2.0/mlflow/experiments/list disabled due to the mlflow server running in `--artifacts-only` mode.   Using an additional MLflow server to handle artifacts exclusively can be useful for large-scale MLOps infrastructure. Decoupling the longer running and more compute-intensive tasks of artifact handling from the faster and higher-volume metadata functionality of the other Tracking API requests can help minimize the burden of an otherwise single MLflow server handling both types of payloads.    Logging to a Tracking Server  To log to a tracking server, set the MLFLOW_TRACKING_URI environment variable to the server's URI, along with its scheme and port (for example, http://10.0.0.1:5000) or call mlflow.set_tracking_uri(). The mlflow.start_run(), mlflow.log_param(), and mlflow.log_metric() calls then make API requests to your remote tracking server.   import mlflow  remote_server_uri = ""...""  # set to your server URI mlflow.set_tracking_uri(remote_server_uri) # Note: on Databricks, the experiment name passed to mlflow_set_experiment must be a # valid path in the workspace mlflow.set_experiment(""/my-experiment"") with mlflow.start_run():     mlflow.log_param(""a"", 1)     mlflow.log_metric(""b"", 2)   library(mlflow) install_mlflow() remote_server_uri = ""..."" # set to your server URI mlflow_set_tracking_uri(remote_server_uri) # Note: on Databricks, the experiment name passed to mlflow_set_experiment must be a # valid path in the workspace mlflow_set_experiment(""/my-experiment"") mlflow_log_param(""a"", ""1"")     In addition to the MLFLOW_TRACKING_URI environment variable, the following environment variables allow passing HTTP authentication to the tracking server:  MLFLOW_TRACKING_USERNAME and MLFLOW_TRACKING_PASSWORD - username and password to use with HTTP Basic authentication. To use Basic authentication, you must set both environment variables . MLFLOW_TRACKING_TOKEN - token to use with HTTP Bearer authentication. Basic authentication takes precedence if set. MLFLOW_TRACKING_INSECURE_TLS - If set to the literal true, MLflow does not verify the TLS connection, meaning it does not validate certificates or hostnames for https:// tracking URIs. This flag is not recommended for production environments. If this is set to true then MLFLOW_TRACKING_SERVER_CERT_PATH must not be set. MLFLOW_TRACKING_SERVER_CERT_PATH - Path to a CA bundle to use. Sets the verify param of the requests.request function (see requests main interface). When you use a self-signed server certificate you can use this to verify it on client side. If this is set MLFLOW_TRACKING_INSECURE_TLS must not be set (false). MLFLOW_TRACKING_CLIENT_CERT_PATH - Path to ssl client cert file (.pem). Sets the cert param of the requests.request function (see requests main interface). This can be used to use a (self-signed) client certificate.   Note If the MLflow server is not configured with the --serve-artifacts option, the client directly pushes artifacts to the artifact store. It does not proxy these through the tracking server by default. For this reason, the client needs direct access to the artifact store. For instructions on setting up these credentials, see Artifact Stores.   Tracking Server versioning  The version of MLflow running on the server can be found by querying the /version endpoint. This can be used to check that the client-side version of MLflow is up-to-date with a remote tracking server prior to running experiments. For example: import requests import mlflow  response = requests.get(""http://<mlflow-host>:<mlflow-port>/version"") assert response.text == mlflow.__version__  # Checking for a strict version match       System Tags  You can annotate runs with arbitrary tags. Tag keys that start with mlflow. are reserved for internal use. The following tags are set automatically by MLflow, when appropriate:       Key Description    mlflow.note.content A descriptive note about this run. This reserved tag is not set automatically and can be overridden by the user to include additional information about the run. The content is displayed on the run's page under the Notes section.  mlflow.parentRunId The ID of the parent run, if this is a nested run.  mlflow.user Identifier of the user who created the run.  mlflow.source.type Source type. Possible values: ""NOTEBOOK"", ""JOB"", ""PROJECT"", ""LOCAL"", and ""UNKNOWN""  mlflow.source.name Source identifier (e.g., GitHub URL, local Python filename, name of notebook)  mlflow.source.git.commit Commit hash of the executed code, if in a git repository.  mlflow.source.git.branch Name of the branch of the executed code, if in a git repository.  mlflow.source.git.repoURL URL that the executed code was cloned from.  mlflow.project.env The runtime context used by the MLflow project. Possible values: ""docker"" and ""conda"".  mlflow.project.entryPoint Name of the project entry point associated with the current run, if any.  mlflow.docker.image.name Name of the Docker image used to execute this run.  mlflow.docker.image.id ID of the Docker image used to execute this run.  mlflow.log-model.history Model metadata collected by log-model calls. Includes the serialized form of the MLModel model files logged to a run, although the exact format and information captured is subject to change.           Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
projects.html,"   Documentation  MLflow Projects       MLflow Projects  An MLflow Project is a format for packaging data science code in a reusable and reproducible way, based primarily on conventions. In addition, the Projects component includes an API and command-line tools for running projects, making it possible to chain together projects into workflows.  Table of Contents  Overview Specifying Projects Running Projects Iterating Quickly Building Multistep Workflows    Overview  At the core, MLflow Projects are just a convention for organizing and describing your code to let other data scientists (or automated tools) run it. Each project is simply a directory of files, or a Git repository, containing your code. MLflow can run some projects based on a convention for placing files in this directory (for example, a conda.yaml file is treated as a Conda environment), but you can describe your project in more detail by adding a MLproject file, which is a YAML formatted text file. Each project can specify several properties:  NameA human-readable name for the project.  Entry PointsCommands that can be run within the project, and information about their parameters. Most projects contain at least one entry point that you want other users to call. Some projects can also contain more than one entry point: for example, you might have a single Git repository containing multiple featurization algorithms. You can also call any .py or .sh file in the project as an entry point. If you list your entry points in a MLproject file, however, you can also specify parameters for them, including data types and default values.  EnvironmentThe software environment that should be used to execute project entry points. This includes all library dependencies required by the project code. See Project Environments for more information about the software environments supported by MLflow Projects, including Conda environments, Virtualenv environments, and Docker containers.   You can run any project from a Git URI or from a local directory using the mlflow run command-line tool, or the mlflow.projects.run() Python API. These APIs also allow submitting the project for remote execution on Databricks and Kubernetes.  Important By default, MLflow uses a new, temporary working directory for Git projects. This means that you should generally pass any file arguments to MLflow project using absolute, not relative, paths. If your project declares its parameters, MLflow automatically makes paths absolute for parameters of type path.    Specifying Projects  By default, any Git repository or local directory can be treated as an MLflow project; you can invoke any bash or Python script contained in the directory as a project entry point. The Project Directories section describes how MLflow interprets directories as projects. To provide additional control over a project's attributes, you can also include an MLproject file in your project's repository or directory. Finally, MLflow projects allow you to specify the software environment that is used to execute project entry points.  Project Environments  MLflow currently supports the following project environments: Virtualenv environment, conda environment, Docker container environment, and system environment.  Virtualenv environment (preferred)Virtualenv environments support Python packages available on PyPI. When an MLflow Project specifies a Virtualenv environment, MLflow will download the specified version of Python by using pyenv and create an isolated environment that contains the project dependencies using virtualenv, activating it as the execution environment prior to running the project code. You can specify a Virtualenv environment for your MLflow Project by including a python_env entry in your MLproject file. For details, see the Project Directories and Specifying an Environment sections.    Docker container environmentDocker containers allow you to capture non-Python dependencies such as Java libraries. When you run an MLflow project that specifies a Docker image, MLflow runs your image as is with the parameters specified in your MLproject file. In this case you'll need to pre build your images with both environment and code to run it. To run the project with a new image that's based on your image and contains the project's contents in the /mlflow/projects/code directory, use the --build-image flag when running mlflow run. Environment variables, such as MLFLOW_TRACKING_URI, are propagated inside the Docker container during project execution. Additionally, runs and experiments created by the project are saved to the tracking server specified by your tracking URI. When running against a local tracking URI, MLflow mounts the host system's tracking directory (e.g., a local mlruns directory) inside the container so that metrics, parameters, and artifacts logged during project execution are accessible afterwards. See Dockerized Model Training with MLflow for an example of an MLflow project with a Docker environment. To specify a Docker container environment, you must add an MLproject file to your project. For information about specifying a Docker container environment in an MLproject file, see Specifying an Environment.    Conda environmentConda environments support both Python packages and native libraries (e.g, CuDNN or Intel MKL). When an MLflow Project specifies a Conda environment, it is activated before project code is run.  Warning By using conda, you're responsible for adhering to Anaconda's terms of service.  By default, MLflow uses the system path to find and run the conda binary. You can use a different Conda installation by setting the MLFLOW_CONDA_HOME environment variable; in this case, MLflow attempts to run the binary at $MLFLOW_CONDA_HOME/bin/conda. You can specify a Conda environment for your MLflow project by including a conda.yaml file in the root of the project directory or by including a conda_env entry in your MLproject file. For details, see the Project Directories and Specifying an Environment sections. The mlflow run command supports running a conda environment project as a virtualenv environment project. To do this, run mlflow run with --env-manager virtualenv: mlflow run /path/to/conda/project --env-manager virtualenv    Warning When a conda environment project is executed as a virtualenv environment project, conda dependencies will be ignored and only pip dependencies will be installed.   System environmentYou can also run MLflow Projects directly in your current system environment. All of the project's dependencies must be installed on your system prior to project execution. The system environment is supplied at runtime. It is not part of the MLflow Project's directory contents or MLproject file. For information about using the system environment when running a project, see the Environment parameter description in the Running Projects section.     Project Directories  When running an MLflow Project directory or repository that does not contain an MLproject file, MLflow uses the following conventions to determine the project's attributes:  The project's name is the name of the directory. The Conda environment is specified in conda.yaml, if present. If no conda.yaml file is present, MLflow uses a Conda environment containing only Python (specifically, the latest Python available to Conda) when running the project. Any .py and .sh file in the project can be an entry point. MLflow uses Python to execute entry points with the .py extension, and it uses bash to execute entry points with the .sh extension. For more information about specifying project entrypoints at runtime, see Running Projects. By default, entry points do not have any parameters when an MLproject file is not included. Parameters can be supplied at runtime via the mlflow run CLI or the mlflow.projects.run() Python API. Runtime parameters are passed to the entry point on the command line using --key value syntax. For more information about running projects and with runtime parameters, see Running Projects.    MLproject File  You can get more control over an MLflow Project by adding an MLproject file, which is a text file in YAML syntax, to the project's root directory. The following is an example of an MLproject file: name: My Project  python_env: python_env.yaml # or # conda_env: my_env.yaml # or # docker_env: #    image:  mlflow-docker-example  entry_points:   main:     parameters:       data_file: path       regularization: {type: float, default: 0.1}     command: ""python train.py -r {regularization} {data_file}""   validate:     parameters:       data_file: path     command: ""python validate.py {data_file}""   The file can specify a name and a Conda or Docker environment, as well as more detailed information about each entry point. Specifically, each entry point defines a command to run and parameters to pass to the command (including data types).  Specifying an Environment  This section describes how to specify Conda and Docker container environments in an MLproject file. MLproject files cannot specify both a Conda environment and a Docker environment.  Virtualenv environmentInclude a top-level python_env entry in the MLproject file. The value of this entry must be a relative path to a python_env YAML file within the MLflow project's directory. The following is an example MLProject file with a python_env definition: python_env: files/config/python_env.yaml   python_env refers to an environment file located at <MLFLOW_PROJECT_DIRECTORY>/files/config/python_env.yaml, where <MLFLOW_PROJECT_DIRECTORY> is the path to the MLflow project's root directory. The following is an example of a python_env.yaml file: # Python version required to run the project. python: ""3.8.15"" # Dependencies required to build packages. This field is optional. build_dependencies:   - pip   - setuptools   - wheel==0.37.1 # Dependencies required to run the project. dependencies:   - mlflow==2.3   - scikit-learn==1.0.2    Conda environmentInclude a top-level conda_env entry in the MLproject file. The value of this entry must be a relative path to a Conda environment YAML file within the MLflow project's directory. In the following example: conda_env: files/config/conda_environment.yaml   conda_env refers to an environment file located at <MLFLOW_PROJECT_DIRECTORY>/files/config/conda_environment.yaml, where <MLFLOW_PROJECT_DIRECTORY> is the path to the MLflow project's root directory.  Docker container environmentInclude a top-level docker_env entry in the MLproject file. The value of this entry must be the name of a Docker image that is accessible on the system executing the project; this image name may include a registry path and tags. Here are a couple of examples. Example 1: Image without a registry path docker_env:   image: mlflow-docker-example-environment   In this example, docker_env refers to the Docker image with name mlflow-docker-example-environment and default tag latest. Because no registry path is specified, Docker searches for this image on the system that runs the MLflow project. If the image is not found, Docker attempts to pull it from DockerHub. Example 2: Mounting volumes and specifying environment variables You can also specify local volumes to mount in the docker image (as you normally would with Docker's -v option), and additional environment variables (as per Docker's -e option). Environment variables can either be copied from the host system's environment variables, or specified as new variables for the Docker environment. The environment field should be a list. Elements in this list can either be lists of two strings (for defining a new variable) or single strings (for copying variables from the host system). For example: docker_env:   image: mlflow-docker-example-environment   volumes: [""/local/path:/container/mount/path""]   environment: [[""NEW_ENV_VAR"", ""new_var_value""], ""VAR_TO_COPY_FROM_HOST_ENVIRONMENT""]   In this example our docker container will have one additional local volume mounted, and two additional environment variables: one newly-defined, and one copied from the host system. Example 3: Image in a remote registry docker_env:   image: 012345678910.dkr.ecr.us-west-2.amazonaws.com/mlflow-docker-example-environment:7.0   In this example, docker_env refers to the Docker image with name mlflow-docker-example-environment and tag 7.0 in the Docker registry with path 012345678910.dkr.ecr.us-west-2.amazonaws.com, which corresponds to an Amazon ECR registry. When the MLflow project is run, Docker attempts to pull the image from the specified registry. The system executing the MLflow project must have credentials to pull this image from  the specified registry. Example 4: Build a new image docker_env:   image: python:3.8   mlflow run ... --build-image   To build a new image that's based on the specified image and files contained in the project directory, use the --build-image argument. In the above example, the image python:3.8 is pulled from Docker Hub if it's not present locally, and a new image is built based on it. The project is executed in a container created from this image.     Command Syntax  When specifying an entry point in an MLproject file, the command can be any string in Python format string syntax. All of the parameters declared in the entry point's parameters field are passed into this string for substitution. If you call the project with additional parameters not listed in the parameters field, MLflow passes them using --key value syntax, so you can use the MLproject file to declare types and defaults for just a subset of your parameters. Before substituting parameters in the command, MLflow escapes them using the Python shlex.quote function, so you don't need to worry about adding quotes inside your command field.   Specifying Parameters  MLflow allows specifying a data type and default value for each parameter. You can specify just the data type by writing: parameter_name: data_type   in your YAML file, or add a default value as well using one of the following syntaxes (which are equivalent in YAML): parameter_name: {type: data_type, default: value}  # Short syntax  parameter_name:     # Long syntax   type: data_type   default: value   MLflow supports four parameter types, some of which it treats specially (for example, downloading data to local files). Any undeclared parameters are treated as string. The parameter types are:  stringA text string.  floatA real number. MLflow validates that the parameter is a number.  pathA path on the local file system. MLflow converts any relative path parameters to absolute paths. MLflow also downloads any paths passed as distributed storage URIs (s3://, dbfs://, gs://, etc.) to local files. Use this type for programs that can only read local files.  uriA URI for data either in a local or distributed storage system. MLflow converts relative paths to absolute paths, as in the path type. Use this type for programs that know how to read from distributed storage (e.g., programs that use Spark).       Running Projects  MLflow provides two ways to run projects: the mlflow run command-line tool, or the mlflow.projects.run() Python API. Both tools take the following parameters:  Project URIA directory on the local file system or a Git repository path, specified as a URI of the form https://<repo> (to use HTTPS) or user@host:path (to use Git over SSH). To run against an MLproject file located in a subdirectory of the project, add a '#' to the end of the URI argument, followed by the relative path from the project's root directory to the subdirectory containing the desired project.  Project VersionFor Git-based projects, the commit hash or branch name in the Git repository.  Entry PointThe name of the entry point, which defaults to main. You can use any entry point named in the MLproject file, or any .py or .sh file in the project, given as a path from the project root (for example, src/test.py).  ParametersKey-value parameters. Any parameters with declared types are validated and transformed if needed.  Deployment Mode Both the command-line and API let you launch projects remotely in a Databricks environment. This includes setting cluster parameters such as a VM type. Of course, you can also run projects on any other computing infrastructure of your choice using the local version of the mlflow run command (for example, submit a script that does mlflow run to a standard job queueing system). You can also launch projects remotely on Kubernetes clusters using the mlflow run CLI (see Run an MLflow Project on Kubernetes).   EnvironmentBy default, MLflow Projects are run in the environment specified by the project directory or the MLproject file (see Specifying Project Environments). You can ignore a project's specified environment and run the project in the current system environment by supplying the --env-manager=local flag, but this can lead to unexpected results if there are dependency mismatches between the project environment and the current system environment.   For example, the tutorial creates and publishes an MLflow Project that trains a linear model. The project is also published on GitHub at https://github.com/mlflow/mlflow-example. To run this project: mlflow run git@github.com:mlflow/mlflow-example.git -P alpha=0.5   There are also additional options for disabling the creation of a Conda environment, which can be useful if you quickly want to test a project in your existing shell environment.  Run an MLflow Project on Databricks  You can run MLflow Projects remotely on Databricks. To use this feature, you must have an enterprise Databricks account (Community Edition is not supported) and you must have set up the Databricks CLI. Find detailed instructions in the Databricks docs (Azure Databricks, Databricks on AWS).   Run an MLflow Project on Kubernetes  You can run MLflow Projects with Docker environments on Kubernetes. The following sections provide an overview of the feature, including a simple Project execution guide with examples. To see this feature in action, you can also refer to the Docker example, which includes the required Kubernetes backend configuration (kubernetes_backend.json) and Kubernetes Job Spec (kubernetes_job_template.yaml) files.  How it works  When you run an MLflow Project on Kubernetes, MLflow constructs a new Docker image containing the Project's contents; this image inherits from the Project's Docker environment. MLflow then pushes the new Project image to your specified Docker registry and starts a Kubernetes Job on your specified Kubernetes cluster. This Kubernetes Job downloads the Project image and starts a corresponding Docker container. Finally, the container invokes your Project's entry point, logging parameters, tags, metrics, and artifacts to your MLflow tracking server.   Execution guide  You can run your MLflow Project on Kubernetes by following these steps:  Add a Docker environment to your MLflow Project, if one does not already exist. For reference, see Specifying an Environment. Create a backend configuration JSON file with the following entries:  kube-context The Kubernetes context where MLflow will run the job. If not provided, MLflow will use the current context. If no context is available, MLflow will assume it is running in a Kubernetes cluster and it will use the Kubernetes service account running the current pod ('in-cluster' configuration). repository-uri The URI of the docker repository where the Project execution Docker image will be uploaded (pushed). Your Kubernetes cluster must have access to this repository in order to run your MLflow Project. kube-job-template-path The path to a YAML configuration file for your Kubernetes Job - a Kubernetes Job Spec. MLflow reads the Job Spec and replaces certain fields to facilitate job execution and monitoring; MLflow does not modify the original template file. For more information about writing Kubernetes Job Spec templates for use with MLflow, see the Job Templates section.     Example Kubernetes backend configuration {   ""kube-context"": ""docker-for-desktop"",   ""repository-uri"": ""username/mlflow-kubernetes-example"",   ""kube-job-template-path"": ""/Users/username/path/to/kubernetes_job_template.yaml"" }     If necessary, obtain credentials to access your Project's Docker and Kubernetes resources, including:  The Docker environment image specified in the MLproject file. The Docker repository referenced by repository-uri in your backend configuration file. The Kubernetes context referenced by kube-context in your backend configuration file.  MLflow expects these resources to be accessible via the docker and kubectl CLIs before running the Project.  Run the Project using the MLflow Projects CLI or Python API, specifying your Project URI and the path to your backend configuration file. For example: mlflow run <project_uri> --backend kubernetes --backend-config examples/docker/kubernetes_config.json   where <project_uri> is a Git repository URI or a folder.     Job Templates  MLflow executes Projects on Kubernetes by creating Kubernetes Job resources. MLflow creates a Kubernetes Job for an MLflow Project by reading a user-specified Job Spec. When MLflow reads a Job Spec, it formats the following fields:  metadata.name Replaced with a string containing the name of the MLflow Project and the time of Project execution spec.template.spec.container[0].name Replaced with the name of the MLflow Project spec.template.spec.container[0].image Replaced with the URI of the Docker image created during Project execution. This URI includes the Docker image's digest hash. spec.template.spec.container[0].command Replaced with the Project entry point command specified when executing the MLflow Project.  The following example shows a simple Kubernetes Job Spec that is compatible with MLflow Project execution. Replaced fields are indicated using bracketed text. Example Kubernetes Job Spec apiVersion: batch/v1 kind: Job metadata:   name: ""{replaced with MLflow Project name}""   namespace: mlflow spec:   ttlSecondsAfterFinished: 100   backoffLimit: 0   template:     spec:       containers:       - name: ""{replaced with MLflow Project name}""         image: ""{replaced with URI of Docker image created during Project execution}""         command: [""{replaced with MLflow Project entry point command}""]         env: [""{appended with MLFLOW_TRACKING_URI, MLFLOW_RUN_ID and MLFLOW_EXPERIMENT_ID}""]         resources:           limits:             memory: 512Mi           requests:             memory: 256Mi       restartPolicy: Never   The container.name, container.image, and container.command fields are only replaced for the first container defined in the Job Spec. Further, the MLFLOW_TRACKING_URI, MLFLOW_RUN_ID and MLFLOW_EXPERIMENT_ID are appended to container.env. Use KUBE_MLFLOW_TRACKING_URI to pass a different tracking URI to the job container from the standard MLFLOW_TRACKING_URI. All subsequent container definitions are applied without modification.     Iterating Quickly  If you want to rapidly develop a project, we recommend creating an MLproject file with your main program specified as the main entry point, and running it with mlflow run .. To avoid having to write parameters repeatedly, you can add default parameters in your MLproject file.   Building Multistep Workflows  The mlflow.projects.run() API, combined with mlflow.client, makes it possible to build multi-step workflows with separate projects (or entry points in the same project) as the individual steps. Each call to mlflow.projects.run() returns a run object, that you can use with mlflow.client to determine when the run has ended and get its output artifacts. These artifacts can then be passed into another step that takes path or uri parameters. You can coordinate all of the workflow in a single Python program that looks at the results of each step and decides what to submit next using custom code. Some example use cases for multi-step workflows include:  Modularizing Your Data Science CodeDifferent users can publish reusable steps for data featurization, training, validation, and so on, that other users or team can run in their workflows. Because MLflow supports Git versioning, another team can lock their workflow to a specific version of a project, or upgrade to a new one on their own schedule.  Hyperparameter TuningUsing mlflow.projects.run() you can launch multiple runs in parallel either on the local machine or on a cloud platform like Databricks. Your driver program can then inspect the metrics from each run in real time to cancel runs, launch new ones, or select the best performing run on a target metric.  Cross-validationSometimes you want to run the same training code on different random splits of training and validation data. With MLflow Projects, you can package the project in a way that allows this, for example, by taking a random seed for the train/validation split as a parameter, or by calling another project first that can split the input data.   For an example of how to construct such a multistep workflow, see the MLflow Multistep Workflow Example project.        Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
models.html,"   Documentation  MLflow Models       MLflow Models  An MLflow Model is a standard format for packaging machine learning models that can be used in a variety of downstream tools—for example, real-time serving through a REST API or batch inference on Apache Spark. The format defines a convention that lets you save a model in different "flavors" that can be understood by different downstream tools.  Table of Contents  Storage Format Model Signature And Input Example Model API Built-In Model Flavors Model Evaluation Model Customization Built-In Deployment Tools Deployment to Custom Targets Community Model Flavors    Storage Format  Each MLflow Model is a directory containing arbitrary files, together with an MLmodel file in the root of the directory that can define multiple flavors that the model can be viewed in. Flavors are the key concept that makes MLflow Models powerful: they are a convention that deployment tools can use to understand the model, which makes it possible to write tools that work with models from any ML library without having to integrate each tool with each library. MLflow defines several "standard" flavors that all of its built-in deployment tools support, such as a "Python function" flavor that describes how to run the model as a Python function. However, libraries can also define and use other flavors. For example, MLflow's mlflow.sklearn library allows loading models back as a scikit-learn Pipeline object for use in code that is aware of scikit-learn, or as a generic Python function for use in tools that just need to apply the model (for example, the mlflow deployments tool with the option -t sagemaker for deploying models to Amazon SageMaker). All of the flavors that a particular model supports are defined in its MLmodel file in YAML format. For example, mlflow.sklearn outputs models as follows: # Directory written by mlflow.sklearn.save_model(model, ""my_model"") my_model/ ├── MLmodel ├── model.pkl ├── conda.yaml ├── python_env.yaml └── requirements.txt   And its MLmodel file describes two flavors: time_created: 2018-05-25T17:28:53.35  flavors:   sklearn:     sklearn_version: 0.19.1     pickled_model: model.pkl   python_function:     loader_module: mlflow.sklearn   This model can then be used with any tool that supports either the sklearn or python_function model flavor. For example, the mlflow models serve command can serve a model with the python_function or the crate (R Function) flavor: mlflow models serve -m my_model    Note If you wish to serve a model from inside a docker container (or to query it from another machine), you need to change the network address to 0.0.0.0 using the -h argument. mlflow models serve -h 0.0.0.0 -m my_model    In addition, the mlflow deployments command-line tool can package and deploy models to AWS SageMaker as long as they support the python_function flavor: mlflow deployments create -t sagemaker -m my_model [other options]    Note When a model registered in the MLflow Model Registry is downloaded, a YAML file named registered_model_meta is added to the model directory on the downloader's side. This file contains the name and version of the model referenced in the MLflow Model Registry, and will be used for deployment and other purposes.   Fields in the MLmodel Format  Apart from a flavors field listing the model flavors, the MLmodel YAML format can contain the following fields:  time_createdDate and time when the model was created, in UTC ISO 8601 format.  run_idID of the run that created the model, if the model was saved using MLflow Tracking.  signaturemodel signature in JSON format.  input_examplereference to an artifact with input example.  databricks_runtimeDatabricks runtime version and type, if the model was trained in a Databricks notebook or job.  mlflow_versionThe version of MLflow that was used to log the model.     Additional Logged Files  For environment recreation, we automatically log conda.yaml, python_env.yaml, and requirements.txt files whenever a model is logged. These files can then be used to reinstall dependencies using conda or virtualenv with pip.  Note Anaconda Inc. updated their terms of service for anaconda.org channels. Based on the new terms of service you may require a commercial license if you rely on Anaconda's packaging and distribution. See Anaconda Commercial Edition FAQ for more information. Your use of any Anaconda channels is governed by their terms of service. MLflow models logged before v1.18 were by default logged with the conda defaults channel (https://repo.anaconda.com/pkgs/) as a dependency. Because of this license change, MLflow has stopped the use of the defaults channel for models logged using MLflow v1.18 and above. The default channel logged is now conda-forge, which points at the community managed https://conda-forge.org/. If you logged a model before MLflow v1.18 without excluding the defaults channel from the conda environment for the model, that model may have a dependency on the defaults channel that you may not have intended. To manually confirm whether a model has this dependency, you can examine channel value in the conda.yaml file that is packaged with the logged model. For example, a model's conda.yaml with a defaults channel dependency may look like this: name: mlflow-env channels: - defaults dependencies: - python=3.8.8 - pip - pip:     - mlflow==2.3     - scikit-learn==0.23.2     - cloudpickle==1.6.0   If you would like to change the channel used in a model's environment, you can re-register the model to the model registry with a new conda.yaml. You can do this by specifying the channel in the conda_env parameter of log_model(). For more information on the log_model() API, see the MLflow documentation for the model flavor you are working with, for example, mlflow.sklearn.log_model().   conda.yamlWhen saving a model, MLflow provides the option to pass in a conda environment parameter that can contain dependencies used by the model. If no conda environment is provided, a default environment is created based on the flavor of the model. This conda environment is then saved in conda.yaml.  python_env.yamlThis file contains the following information that's required to restore a model environment using virtualenv:  Python version Version specifiers for pip, setuptools, and wheel Pip requirements of the model (reference to requirements.txt)   requirements.txtThe requirements file is created from the pip portion of the conda.yaml environment specification. Additional pip dependencies can be added to requirements.txt by including them as a pip dependency in a conda environment and logging the model with the environment or using the pip_requirements argument of the mlflow.<flavor>.log_model API.   The following shows an example of saving a model with a manually specified conda environment and the corresponding content of the generated conda.yaml and requirements.txt files. conda_env = {     ""channels"": [""conda-forge""],     ""dependencies"": [""python=3.8.8"", ""pip""],     ""pip"": [""mlflow==2.3"", ""scikit-learn==0.23.2"", ""cloudpickle==1.6.0""],     ""name"": ""mlflow-env"", } mlflow.sklearn.log_model(..., conda_env=conda_env)   The written conda.yaml file: name: mlflow-env channels:   - conda-forge dependencies: - python=3.8.8 - pip - pip:   - mlflow==2.3   - scikit-learn==0.23.2   - cloudpickle==1.6.0   The written python_env.yaml file: python: 3.8.8 build_dependencies:   - pip==21.1.3   - setuptools==57.4.0   - wheel==0.37.0 dependencies:   - -r requirements.txt   The written requirements.txt file: mlflow==2.3 scikit-learn==0.23.2 cloudpickle==1.6.0      Model Signature And Input Example  When working with ML models you often need to know some basic functional properties of the model at hand, such as "What inputs does it expect?" and "What output does it produce?". MLflow models can include the following additional metadata about model inputs, outputs and params that can be used by downstream tooling:  Model Inference Params - description of params used for model inference. Model Signature - description of a model's inputs, outputs and parameters. Model Input Example - example of a valid model input.   Model Inference Params  Inference params are parameters that are passed to the model at inference time. These parameters do not need to be specified when training the model, but could be useful for inference. With the advances in foundational models, more often "inference configuration" is used to modify the behavior of a model. In some cases, especially popular LLMs, the same model may require different parameter configurations for different samples at inference time. With this newly introduced feature, you can now specify a dictionary of inference params during model inference, providing a broader utility and improved control over the generated inference results, particularly for LLM use cases. By passing different params such as temperature, max_length, etc. to the model at inference time, you can easily control the output of the model. In order to use params at inference time, a valid Model Signature with params must be defined. The params are passed to the model at inference time as a dictionary and each param value will be validated against the corresponding param type defined in the model signature. Valid param types are DataType or a list of DataType as listed below.  DataType.string or an array of DataType.string DataType.integer or an array of DataType.integer DataType.boolean or an array of DataType.boolean DataType.double or an array of DataType.double DataType.float or an array of DataType.float DataType.long or an array of DataType.long DataType.datetime or an array of DataType.datetime   Note When validating param values, the values will be converted to python native types. For example, np.float32(0.1) will be converted to float(0.1).  A simple example of using params for model inference: import mlflow from mlflow.models import infer_signature   class MyModel(mlflow.pyfunc.PythonModel):     def predict(self, ctx, model_input, params):         return list(params.values())   params = {""str_param"": ""string"", ""int_array"": [1, 2, 3]} # params' default values are saved with ModelSignature signature = infer_signature([""input""], params=params)  with mlflow.start_run():     model_info = mlflow.pyfunc.log_model(         python_model=MyModel(), artifact_path=""my_model"", signature=signature     )  loaded_model = mlflow.pyfunc.load_model(model_info.model_uri)  # Not passing params -- predict with default values loaded_predict = loaded_model.predict([""input""]) assert loaded_predict == [""string"", [1, 2, 3]]  # Passing some params -- add default values loaded_predict = loaded_model.predict([""input""], params={""str_param"": ""new_string""}) assert loaded_predict == [""new_string"", [1, 2, 3]]  # Passing all params -- override loaded_predict = loaded_model.predict(     [""input""], params={""str_param"": ""new_string"", ""int_array"": [4, 5, 6]} ) assert loaded_predict == [""new_string"", [4, 5, 6]]     Model Signature  Model signatures define input, output and parameters schemas for MLflow models, providing a standard interface to codify and enforce the correct use of your models. Signatures are fetched by the MLflow Tracking UI and Model Registry UI to display model inputs, outputs and params. They are also utilized by MLflow model deployment tools to validate inference inputs according to the model's assigned signature (see the Signature enforcement section for more details). To include a signature with your model, pass a model input example as an argument to the appropriate log_model or save_model call, e.g. sklearn.log_model(), and the model signature will be automatically inferred (see the How to log models with signatures section for more details). The model signature is stored in JSON format in the MLmodel file in your model artifacts, together with other model metadata. To set a signature on a logged or saved model, use the set_signature() API (see the How to set signatures on models section for more details).  Model Signature Types  A model signature consists on inputs and outputs schemas, each of which can be either column-based or tensor-based. Column-based schemas are a sequence of (optionally) named columns with type specified as one of the MLflow data types. Tensor-based schemas are a sequence of (optionally) named tensors with type specified as one of the numpy data types. Params schema is a sequence of ParamSpec, each of which contains name, type, default and shape fields. type field must be specified as one of the MLflow data types, and shape field should be None for scalar parameters, or (-1,) for list parameters. See some examples of constructing them below.  Column-based Signature Example  Each column-based input and output is represented by a type corresponding to one of MLflow data types and an optional name. Input columns can also be marked as optional, indicating whether they are required as input to the model or can be omitted. The following example displays a modified MLmodel file excerpt containing the model signature for a classification model trained on the Iris dataset. The input has 4 named, numeric columns and 1 named, optional string column. The output is an unnamed integer specifying the predicted class. signature:     inputs: '[{""name"": ""sepal length (cm)"", ""type"": ""double""}, {""name"": ""sepal width       (cm)"", ""type"": ""double""}, {""name"": ""petal length (cm)"", ""type"": ""double""}, {""name"":       ""petal width (cm)"", ""type"": ""double""}, {""name"": ""class"", ""type"": ""string"", ""optional"": ""true""}]'     outputs: '[{""type"": ""integer""}]'     params: null     Tensor-based Signature Example  Each tensor-based input and output is represented by a dtype corresponding to one of numpy data types, shape and an optional name. Tensor-based signatures do not support optional inputs. When specifying the shape, -1 is used for axes that may be variable in size. The following example displays an MLmodel file excerpt containing the model signature for a classification model trained on the MNIST dataset. The input has one named tensor where input sample is an image represented by a 28 × 28 × 1 array of float32 numbers. The output is an unnamed tensor that has 10 units specifying the likelihood corresponding to each of the 10 classes. Note that the first dimension of the input and the output is the batch size and is thus set to -1 to allow for variable batch sizes. signature:     inputs: '[{""name"": ""images"", ""dtype"": ""uint8"", ""shape"": [-1, 28, 28, 1]}]'     outputs: '[{""shape"": [-1, 10], ""dtype"": ""float32""}]'     params: null     Signature with params Example  The params field is optional and is used to specify parameters that can be used for model inference. Params accept scalar values of type MLflow data types, or a list of such values. The default value of a parameter is specified by setting the default field, and the value should be of the type specified by type field. The shape field can be used to specify the shape of the value, it should be None for scalar values and (-1,) for a list. signature:     inputs: '[{""name"": ""text"", ""type"": ""string""}]'     outputs: '[{""name"": ""output"", ""type"": ""string""}]'     params: '[{""name"": ""temperature"", ""type"": ""float"", ""default"": 0.5, ""shape"": null},               {""name"": ""top_k"", ""type"": ""integer"", ""default"": 1, ""shape"": null},               {""name"": ""suppress_tokens"", ""type"": ""integer"", ""default"": [101, 102], ""shape"": [-1]}]'      Signature Enforcement  Schema enforcement checks the provided input and params against the model's signature and raises an exception if the input is not compatible and will issue a warning or raise an exception if the params are incompatible. This enforcement is applied in MLflow before calling the underlying model implementation, and during model inference process. Note that this enforcement only applies when using MLflow model deployment tools or when loading models as python_function. In particular, it is not applied to models that are loaded in their native format (e.g. by calling mlflow.sklearn.load_model()).  Name Ordering Enforcement  The input names are checked against the model signature. If there are any missing required inputs, MLflow will raise an exception. Missing optional inputs will not raise an exception. Extra inputs that were not declared in the signature will be ignored. If the input schema in the signature defines input names, input matching is done by name and the inputs are reordered to match the signature. If the input schema does not have input names, matching is done by position (i.e. MLflow will only check the number of inputs).   Input Type Enforcement  The input types are checked against the signature. For models with column-based signatures (i.e DataFrame inputs), MLflow will perform safe type conversions if necessary. Generally, only conversions that are guaranteed to be lossless are allowed. For example, int -> long or int -> double conversions are ok, long -> double is not. If the types cannot be made compatible, MLflow will raise an error. For models with tensor-based signatures, type checking is strict (i.e an exception will be thrown if the input type does not match the type specified by the schema).   Params Type and Shape Enforcement  The params types and shapes are checked against the signature. MLflow verifies the compatibility of each parameter provided during inference by comparing its type and shape with those specified in the signature. Scalar values should have a shape of None, while list values should have a shape of (-1,). If the parameter's type or shape is incompatible, an exception will be raised. Additionally, the value of the parameter is validated against the specified type in the signature. We attempt to convert the value to the specified type, and if this conversion fails, an MlflowException will be raised. A valid list of params is documented in Model Inference Params section. Models that have signatures and are used for inference with declared params not part of the logged signature will have a warning issued with each request and the invalid params ignored during inference.   Handling Integers With Missing Values  Integer data with missing values is typically represented as floats in Python. Therefore, data types of integer columns in Python can vary depending on the data sample. This type of variance can cause schema enforcement errors at runtime since integer and float are not compatible types. For example, if your training data did not have any missing values for integer column c, its type will be integer. However, when you attempt to score a sample of the data that does include a missing value in column c, its type will be float. If your model signature specified c to have integer type, MLflow will raise an error since it can not convert float to int. Note that MLflow uses python to serve models and to deploy models to Spark, so this can affect most model deployments. The best way to avoid this problem is to declare integer columns as doubles (float64) whenever there are missing values.   Handling Date and Timestamp  For datetime values, Python has precision built into the type. For example, datetime values with day precision have numpy type datetime64[D], while values with nanosecond precision have type datetime64[ns]. Datetime precision is ignored for column-based model signature but is enforced for tensor-based signatures.   Handling Ragged Arrays  Ragged arrays can be created in numpy and are produced with a shape of (-1,) and a dytpe of object. This will be handled by default when using infer_signature, resulting in a signature containing Tensor('object', (-1,)). A similar signature can be manually created containing a more detailed representation of a ragged array, for a more expressive signature, such as Tensor('float64', (-1, -1, -1, 3)). Enforcement will then be done on as much detail as possible given the signature provided, and will support ragged input arrays as well.    How To Log Models With Signatures  To include a signature with your model, pass a model input example to the appropriate log_model or save_model call, e.g. sklearn.log_model(), and the model signature will be automatically inferred from the input example and the model's predicted output of the input example. You may also include a signature object with your model by passing a signature object as an argument to your log_model or save_model call. The model signature object can be created by hand or inferred from datasets with valid model inputs (e.g. the training dataset with target column omitted), valid model outputs (e.g. model predictions generated on the training dataset), and valid model parameters (a dictionary of parameters passed to model for inference; e.g. Generation Configs for transformers).  Note Model signatures are utilized in MLflow model deployment tools, which commonly serve the Python Function (PyFunc) flavor of MLflow models. Hence, when passing a signature object to your log_model or save_model call, it is recommended that the signature represent the inputs and outputs of the model's PyFunc flavor. This is especially important when the model loaded as a PyFunc model has an input schema that is different from the test dataset schema (as is the case with the pmdarima model flavor).   Column-based Signature Example  The following example demonstrates how to store a model signature for a simple classifier trained on the Iris dataset: import pandas as pd from sklearn import datasets from sklearn.ensemble import RandomForestClassifier import mlflow from mlflow.models import infer_signature  iris = datasets.load_iris() iris_train = pd.DataFrame(iris.data, columns=iris.feature_names) clf = RandomForestClassifier(max_depth=7, random_state=0)  with mlflow.start_run():     clf.fit(iris_train, iris.target)     # Take the first row of the training dataset as the model input example.     input_example = iris_train.iloc[[0]]     # The signature is automatically inferred from the input example and its predicted output.     mlflow.sklearn.log_model(clf, ""iris_rf"", input_example=input_example)   The same signature can be explicitly created and logged as follows: from mlflow.models import ModelSignature, infer_signature from mlflow.types.schema import Schema, ColSpec  # Option 1: Manually construct the signature object input_schema = Schema(     [         ColSpec(""double"", ""sepal length (cm)""),         ColSpec(""double"", ""sepal width (cm)""),         ColSpec(""double"", ""petal length (cm)""),         ColSpec(""double"", ""petal width (cm)""),     ] ) output_schema = Schema([ColSpec(""long"")]) signature = ModelSignature(inputs=input_schema, outputs=output_schema)  # Option 2: Infer the signature signature = infer_signature(iris_train, clf.predict(iris_train))  with mlflow.start_run():     mlflow.sklearn.log_model(clf, ""iris_rf"", signature=signature)     Tensor-based Signature Example  The following example demonstrates how to store a model signature for a simple classifier trained on the MNIST dataset: import tensorflow as tf import mlflow  mnist = tf.keras.datasets.mnist (x_train, y_train), (x_test, y_test) = mnist.load_data() x_train, x_test = x_train / 255.0, x_test / 255.0  model = tf.keras.models.Sequential(     [         tf.keras.layers.Flatten(input_shape=(28, 28)),         tf.keras.layers.Dense(128, activation=""relu""),         tf.keras.layers.Dropout(0.2),         tf.keras.layers.Dense(10),     ] ) loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True) model.compile(optimizer=""adam"", loss=loss_fn, metrics=[""accuracy""])  with mlflow.start_run():     model.fit(x_train, y_train, epochs=5)     # Take the first three training examples as the model input example.     input_example = x_train[:3, :]     mlflow.tensorflow.log_model(model, ""mnist_cnn"", input_example=input_example)   The same signature can be explicitly created and logged as follows: import numpy as np from mlflow.models import ModelSignature, infer_signature from mlflow.types.schema import Schema, TensorSpec  # Option 1: Manually construct the signature object input_schema = Schema(     [         TensorSpec(np.dtype(np.float64), (-1, 28, 28, 1)),     ] ) output_schema = Schema([TensorSpec(np.dtype(np.float32), (-1, 10))]) signature = ModelSignature(inputs=input_schema, outputs=output_schema)  # Option 2: Infer the signature signature = infer_signature(testX, model.predict(testX))  with mlflow.start_run():     mlflow.tensorflow.log_model(model, ""mnist_cnn"", signature=signature)     Signature with params Example  The following example demonstrates how to store a model signature with params for a simple transformers model: import mlflow from mlflow.models import infer_signature import transformers  architecture = ""mrm8488/t5-base-finetuned-common_gen"" model = transformers.pipeline(     task=""text2text-generation"",     tokenizer=transformers.T5TokenizerFast.from_pretrained(architecture),     model=transformers.T5ForConditionalGeneration.from_pretrained(architecture), ) data = ""pencil draw paper""  params = {     ""top_k"": 2,     ""num_beams"": 5,     ""max_length"": 30,     ""temperature"": 0.62,     ""top_p"": 0.85,     ""repetition_penalty"": 1.15,     ""begin_suppress_tokens"": [1, 2, 3], }  # infer signature with params signature = infer_signature(     data,     mlflow.transformers.generate_signature_output(model, data),     params, )  # save model with signature mlflow.transformers.save_model(     model,     ""text2text"",     signature=signature, ) pyfunc_loaded = mlflow.pyfunc.load_model(""text2text"")  # predict with params result = pyfunc_loaded.predict(data, params=params)   The same signature can be created explicitly as follows: from mlflow.models import ModelSignature from mlflow.types.schema import ColSpec, ParamSchema, ParamSpec, Schema  input_schema = Schema([ColSpec(type=""string"")]) output_schema = Schema([ColSpec(type=""string"")]) params_schema = ParamSchema(     [         ParamSpec(""top_k"", ""long"", 2),         ParamSpec(""num_beams"", ""long"", 5),         ParamSpec(""max_length"", ""long"", 30),         ParamSpec(""temperature"", ""double"", 0.62),         ParamSpec(""top_p"", ""double"", 0.85),         ParamSpec(""repetition_penalty"", ""double"", 1.15),         ParamSpec(""begin_suppress_tokens"", ""long"", [1, 2, 3], (-1,)),     ] ) signature = ModelSignature(     inputs=input_schema, outputs=output_schema, params=params_schema )      How To Set Signatures on Models  Models can be saved with without model signatures or with incorrect ones. To add a signature to an existing logged model, use the mlflow.models.set_signature() API. Here are some examples.  Set Signature on Logged Model  The following example demonstrates how to set a model signature on a logged sklearn model. Suppose you've logged a sklearn model without a signature like below: import pandas as pd from sklearn import datasets from sklearn.ensemble import RandomForestClassifier import mlflow  X, y = datasets.load_iris(return_X_y=True, as_frame=True) clf = RandomForestClassifier(max_depth=7, random_state=0) with mlflow.start_run() as run:     clf.fit(X, y)     mlflow.sklearn.log_model(clf, ""iris_rf"")   You can set a signature on the logged model as follows: import pandas as pd from sklearn import datasets import mlflow from mlflow.models.model import get_model_info from mlflow.models import infer_signature, set_signature  # load the logged model model_uri = f""runs:/{run.info.run_id}/iris_rf"" model = mlflow.pyfunc.load_model(model_uri)  # construct the model signature from test dataset X_test, _ = datasets.load_iris(return_X_y=True, as_frame=True) signature = infer_signature(X_test, model.predict(X_test))  # set the signature for the logged model set_signature(model_uri, signature)  # now when you load the model again, it will have the desired signature assert get_model_info(model_uri).signature == signature   Note that model signatures can also be set on model artifacts saved outside of MLflow Tracking. As an example, you can easily set a signature on a locally saved iris model by altering the model_uri variable in the previous code snippet to point to the model's local directory.   Set Signature on Model Version  As MLflow Model Registry artifacts are meant to be read-only, you cannot directly set a signature on a model version or model artifacts represented by models:/ URI schemes. Instead, you should first set the signature on the source model artifacts and generate a new model version using the updated model artifacts. The following example illustrates how this can be done. Supposed you have created the following model version without a signature like below: from sklearn.ensemble import RandomForestClassifier import mlflow from mlflow.client import MlflowClient  model_name = ""add_signature_model""  with mlflow.start_run() as run:     mlflow.sklearn.log_model(RandomForestClassifier(), ""sklearn-model"")  model_uri = f""runs:/{run.info.run_id}/sklearn-model"" mlflow.register_model(model_uri=model_uri, name=model_name)   To set a signature on the model version, create a duplicate model version with the new signature as follows: from sklearn.ensemble import RandomForestClassifier import mlflow from mlflow.store.artifact.models_artifact_repo import ModelsArtifactRepository  client = mlflow.client.MlflowClient() model_name = ""add_signature_model"" model_version = 1 mv = client.get_model_version(name=model_name, version=model_version)  # set a dummy signature on the model version source signature = infer_signature(np.array([1])) set_signature(mv.source, signature)  # create a new model version with the updated source client.create_model_version(name=model_name, source=mv.source, run_id=mv.run_id)   Note that this process overwrites the model artifacts from the source run of model version 1 with a new model signature.     Model Input Example  A model input example provides an instance of a valid model input. Input examples are stored with the model as separate artifacts and are referenced in the MLmodel file. To include an input example with your model, add it to the appropriate log_model call, e.g. sklearn.log_model(). Input examples are also used to infer model signatures in log_model calls when signatures aren't specified. Similar to model signatures, model inputs can be column-based (i.e DataFrames) or tensor-based (i.e numpy.ndarrays). We offer support for input_example with params by using tuple to combine model inputs and params. See examples below:  How To Log Model With Column-based Example  For models accepting column-based inputs, an example can be a single record or a batch of records. The sample input can be in the following formats:  Pandas DataFrame dict (of scalars, strings, or lists of scalar values) list str bytes  The given example will be converted to a Pandas DataFrame and then serialized to json using the Pandas split-oriented format. Bytes are base64-encoded. The following example demonstrates how you can log a column-based input example with your model: input_example = {     ""sepal length (cm)"": 5.1,     ""sepal width (cm)"": 3.5,     ""petal length (cm)"": 1.4,     ""petal width (cm)"": 0.2, } mlflow.sklearn.log_model(..., input_example=input_example)     How To Log Model With Tensor-based Example  For models accepting tensor-based inputs, an example must be a batch of inputs. By default, the axis 0 is the batch axis unless specified otherwise in the model signature. The sample input can be passed in as any of the following formats:  numpy ndarray Python dict mapping a string to a numpy array Scipy csr_matrix (sparse matrix) Scipy csc_matrix (sparse matrix).  The following example demonstrates how you can log a tensor-based input example with your model: # each input has shape (4, 4) input_example = np.array(     [         [[0, 0, 0, 0], [0, 134, 25, 56], [253, 242, 195, 6], [0, 93, 82, 82]],         [[0, 23, 46, 0], [33, 13, 36, 166], [76, 75, 0, 255], [33, 44, 11, 82]],     ],     dtype=np.uint8, ) mlflow.tensorflow.log_model(..., input_example=input_example)     How To Log Model With Example Containing Params  For models that require additional parameters during inference, you can include an input_example containing params when saving or logging the model. To achieve this, the sample input should be provided as a tuple. The first element of the tuple is the input data example, and the second element is a dict of params. A comprehensive list of valid params is documented in Model Inference Params section.  Python tuple: (input_data, params)  The following example demonstrates how to log a model with an example containing params: # input_example could be column-based or tensor-based example as shown above # params must be a valid dictionary of params input_data = ""Hello, Dolly!"" params = {""temperature"": 0.5, ""top_k"": 1} input_example = (input_data, params) mlflow.transformers.log_model(..., input_example=input_example)       Model API  You can save and load MLflow Models in multiple ways. First, MLflow includes integrations with several common libraries. For example, mlflow.sklearn contains save_model, log_model, and load_model functions for scikit-learn models. Second, you can use the mlflow.models.Model class to create and write models. This class has four key functions:  add_flavor to add a flavor to the model. Each flavor has a string name and a dictionary of key-value attributes, where the values can be any object that can be serialized to YAML. save to save the model to a local directory. log to log the model as an artifact in the current run using MLflow Tracking. load to load a model from a local directory or from an artifact in a previous run.    Built-In Model Flavors  MLflow provides several standard flavors that might be useful in your applications. Specifically, many of its deployment tools support these flavors, so you can export your own model in one of these flavors to benefit from all these tools:   Python Function (python_function) R Function (crate) H2O (h2o) Keras (keras) MLeap (mleap) PyTorch (pytorch) Scikit-learn (sklearn) Spark MLlib (spark) TensorFlow (tensorflow) ONNX (onnx) MXNet Gluon (gluon) XGBoost (xgboost) LightGBM (lightgbm) CatBoost (catboost) Spacy(spaCy) Fastai(fastai) Statsmodels (statsmodels) Prophet (prophet) Pmdarima (pmdarima) OpenAI (openai) (Experimental) LangChain (langchain) (Experimental) John Snow Labs (johnsnowlabs) (Experimental) Diviner (diviner) Transformers (transformers) (Experimental) SentenceTransformers (sentence_transformers) (Experimental)    Python Function (python_function)  The python_function model flavor serves as a default model interface for MLflow Python models. Any MLflow Python model is expected to be loadable as a python_function model. This enables other MLflow tools to work with any python model regardless of which persistence module or framework was used to produce the model. This interoperability is very powerful because it allows any Python model to be productionized in a variety of environments. In addition, the python_function model flavor defines a generic filesystem model format for Python models and provides utilities for saving and loading models to and from this format. The format is self-contained in the sense that it includes all the information necessary to load and use a model. Dependencies are stored either directly with the model or referenced via conda environment. This model format allows other tools to integrate their models with MLflow.  How To Save Model As Python Function  Most python_function models are saved as part of other model flavors - for example, all mlflow built-in flavors include the python_function flavor in the exported models. In addition, the mlflow.pyfunc module defines functions for creating python_function models explicitly. This module also includes utilities for creating custom Python models, which is a convenient way of adding custom python code to ML models. For more information, see the custom Python models documentation.   How To Load And Score Python Function Models  You can load python_function models in Python by calling the mlflow.pyfunc.load_model() function. Note that the load_model function assumes that all dependencies are already available and will not check nor install any dependencies ( see model deployment section for tools to deploy models with automatic dependency management). Once loaded, you can score the model by calling the predict method, which has the following signature: predict(data: Union[pandas.(Series | DataFrame), numpy.ndarray, csc_matrix, csr_matrix, List[Any], Dict[str, Any], str],         params: Optional[Dict[str, Any]] = None) → Union[pandas.(Series | DataFrame), numpy.ndarray, list, str]   All PyFunc models will support pandas.DataFrame as an input. In addition to pandas.DataFrame, DL PyFunc models will also support tensor inputs in the form of numpy.ndarrays. To verify whether a model flavor supports tensor inputs, please check the flavor's documentation. For models with a column-based schema, inputs are typically provided in the form of a pandas.DataFrame. If a dictionary mapping column name to values is provided as input for schemas with named columns or if a python List or a numpy.ndarray is provided as input for schemas with unnamed columns, MLflow will cast the input to a DataFrame. Schema enforcement and casting with respect to the expected data types is performed against the DataFrame. For models with a tensor-based schema, inputs are typically provided in the form of a numpy.ndarray or a dictionary mapping the tensor name to its np.ndarray value. Schema enforcement will check the provided input's shape and type against the shape and type specified in the model's schema and throw an error if they do not match. For models where no schema is defined, no changes to the model inputs and outputs are made. MLflow will propagate any errors raised by the model if the model does not accept the provided input type. The python environment that a PyFunc model is loaded into for prediction or inference may differ from the environment in which it was trained. In the case of an environment mismatch, a warning message will be printed when calling mlflow.pyfunc.load_model(). This warning statement will identify the packages that have a version mismatch between those used during training and the current environment.  In order to get the full dependencies of the environment in which the model was trained, you can call mlflow.pyfunc.get_model_dependencies(). Furthermore, if you want to run model inference in the same environment used in model training, you can call mlflow.pyfunc.spark_udf() with the env_manager argument set as "conda". This will generate the environment from the conda.yaml file, ensuring that the python UDF will execute with the exact package versions that were used during training. Some PyFunc models may accept model load configuration, which controls how the model is loaded and predictions computed. You can learn which configuration the model supports by inspecting the model's flavor metadata: model_info = mlflow.models.get_model_info(model_uri) model_info.flavors[mlflow.pyfunc.FLAVOR_NAME][mlflow.pyfunc.MODEL_CONFIG]   Alternatively, you can load the PyFunc model and inspect the model_config property: pyfunc_model = mlflow.pyfunc.load_model(model_uri) pyfunc_model.model_config   Model configuration can be changed at loading time by indicating model_config parameter in the mlflow.pyfunc.load_model() method: pyfunc_model = mlflow.pyfunc.load_model(model_uri, model_config=dict(temperature=0.93))   When a model configuration value is changed, those values the configuration the model was saved with. Indicating an invalid model configuration key for a model results in that configuration being ignored. A warning is displayed mentioning the ignored entries.  Note Model configuration vs parameters with default values in signatures: Use model configuration when you need to provide model publishers for a way to change how the model is loaded into memory and how predictions are computed for all the samples. For instance, a key like user_gpu. Model consumers are not able to change those values at predict time. Use parameters with default values in the signature to provide a users the ability to change how predictions are computed on each data sample.     R Function (crate)  The crate model flavor defines a generic model format for representing an arbitrary R prediction function as an MLflow model using the crate function from the carrier package. The prediction function is expected to take a dataframe as input and produce a dataframe, a vector or a list with the predictions as output. This flavor requires R to be installed in order to be used.  crate usage  For a minimal crate model, an example configuration for the predict function is: library(mlflow) library(carrier) # Load iris dataset data(""iris"")  # Learn simple linear regression model model <- lm(Sepal.Width~Sepal.Length, data = iris)  # Define a crate model # call package functions with an explicit :: namespace. crate_model <- crate(   function(new_obs)  stats::predict(model, data.frame(""Sepal.Length"" = new_obs)),   model = model )  # log the model model_path <- mlflow_log_model(model = crate_model, artifact_path = ""iris_prediction"")  # load the logged model and make a prediction model_uri <- paste0(mlflow_get_run()$artifact_uri, ""/iris_prediction"") mlflow_model <- mlflow_load_model(model_uri = model_uri,                                   flavor = NULL,                                   client = mlflow_client())  prediction <- mlflow_predict(model = mlflow_model, data = 5) print(prediction)      H2O (h2o)  The h2o model flavor enables logging and loading H2O models. The mlflow.h2o module defines save_model() and log_model() methods in python, and mlflow_save_model and mlflow_log_model in R for saving H2O models in MLflow Model format. These methods produce MLflow Models with the python_function flavor, allowing you to load them as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can be scored with only DataFrame input. When you load MLflow Models with the h2o flavor using mlflow.pyfunc.load_model(), the h2o.init() method is called. Therefore, the correct version of h2o(-py) must be installed in the loader's environment. You can customize the arguments given to h2o.init() by modifying the init entry of the persisted H2O model's YAML configuration file: model.h2o/h2o.yaml. Finally, you can use the mlflow.h2o.load_model() method to load MLflow Models with the h2o flavor as H2O model objects. For more information, see mlflow.h2o.  h2o pyfunc usage  For a minimal h2o model, here is an example of the pyfunc predict() method in a classification scenario : import mlflow import h2o  h2o.init() from h2o.estimators.glm import H2OGeneralizedLinearEstimator  # import the prostate data df = h2o.import_file(     ""http://s3.amazonaws.com/h2o-public-test-data/smalldata/prostate/prostate.csv.zip"" )  # convert the columns to factors df[""CAPSULE""] = df[""CAPSULE""].asfactor() df[""RACE""] = df[""RACE""].asfactor() df[""DCAPS""] = df[""DCAPS""].asfactor() df[""DPROS""] = df[""DPROS""].asfactor()  # split the data train, test, valid = df.split_frame(ratios=[0.7, 0.15])  # generate a GLM model glm_classifier = H2OGeneralizedLinearEstimator(     family=""binomial"", lambda_=0, alpha=0.5, nfolds=5, compute_p_values=True )  with mlflow.start_run():     glm_classifier.train(         y=""CAPSULE"", x=[""AGE"", ""RACE"", ""VOL"", ""GLEASON""], training_frame=train     )     metrics = glm_classifier.model_performance()     metrics_to_track = [""MSE"", ""RMSE"", ""r2"", ""logloss""]     metrics_to_log = {         key: value         for key, value in metrics._metric_json.items()         if key in metrics_to_track     }     params = glm_classifier.params     mlflow.log_params(params)     mlflow.log_metrics(metrics_to_log)     model_info = mlflow.h2o.log_model(glm_classifier, artifact_path=""h2o_model_info"")  # load h2o model and make a prediction h2o_pyfunc = mlflow.pyfunc.load_model(model_uri=model_info.model_uri) test_df = test.as_data_frame() predictions = h2o_pyfunc.predict(test_df) print(predictions)  # it is also possible to load the model and predict using h2o methods on the h2o frame  # h2o_model = mlflow.h2o.load_model(model_info.model_uri) # predictions = h2o_model.predict(test)      Keras (keras)  The keras model flavor enables logging and loading Keras models. It is available in both Python and R clients. In R, you can save or log the model using mlflow_save_model and mlflow_log_model. These functions serialize Keras models as HDF5 files using the Keras library's built-in model persistence functions. You can use mlflow_load_model function in R to load MLflow Models with the keras flavor as Keras Model objects.  Keras pyfunc usage  For a minimal Sequential model, an example configuration for the pyfunc predict() method is: import mlflow import numpy as np import pathlib import shutil from tensorflow import keras  mlflow.tensorflow.autolog()  X = np.array([-2, -1, 0, 1, 2, 1]).reshape(-1, 1) y = np.array([0, 0, 1, 1, 1, 0]) model = keras.Sequential(     [         keras.Input(shape=(1,)),         keras.layers.Dense(1, activation=""sigmoid""),     ] ) model.compile(loss=""binary_crossentropy"", optimizer=""adam"", metrics=[""accuracy""]) model.fit(X, y, batch_size=3, epochs=5, validation_split=0.2)  local_artifact_dir = ""/tmp/mlflow/keras_model"" pathlib.Path(local_artifact_dir).mkdir(parents=True, exist_ok=True)  model_uri = f""runs:/{mlflow.last_active_run().info.run_id}/model"" keras_pyfunc = mlflow.pyfunc.load_model(     model_uri=model_uri, dst_path=local_artifact_dir )  data = np.array([-4, 1, 0, 10, -2, 1]).reshape(-1, 1) predictions = keras_pyfunc.predict(data)  shutil.rmtree(local_artifact_dir)      MLeap (mleap)   Warning The mleap model flavor is deprecated as of MLflow 2.6.0 and will be removed in a future release.  The mleap model flavor supports saving Spark models in MLflow format using the MLeap persistence mechanism. MLeap is an inference-optimized format and execution engine for Spark models that does not depend on SparkContext to evaluate inputs.  Note You can save Spark models in MLflow format with the mleap flavor by specifying the sample_input argument of the mlflow.spark.save_model() or mlflow.spark.log_model() method (recommended). For more details see Spark MLlib.  The mlflow.mleap module also defines save_model() and log_model() methods for saving MLeap models in MLflow format, but these methods do not include the python_function flavor in the models they produce. Similarly, mleap models can be saved in R with mlflow_save_model and loaded with mlflow_load_model, with mlflow_save_model requiring sample_input to be specified as a sample Spark dataframe containing input data to the model is required by MLeap for data schema inference. A companion module for loading MLflow Models with the MLeap flavor is available in the mlflow/java package. For more information, see mlflow.spark, mlflow.mleap, and the MLeap documentation.   PyTorch (pytorch)  The pytorch model flavor enables logging and loading PyTorch models. The mlflow.pytorch module defines utilities for saving and loading MLflow Models with the pytorch flavor. You can use the mlflow.pytorch.save_model() and mlflow.pytorch.log_model() methods to save PyTorch models in MLflow format; both of these functions use the torch.save() method to serialize PyTorch models. Additionally, you can use the mlflow.pytorch.load_model() method to load MLflow Models with the pytorch flavor as PyTorch model objects. This loaded PyFunc model can be scored with both DataFrame input and numpy array input. Finally, models produced by mlflow.pytorch.save_model() and mlflow.pytorch.log_model() contain the python_function flavor, allowing you to load them as generic Python functions for inference via mlflow.pyfunc.load_model().  Note When using the PyTorch flavor, if a GPU is available at prediction time, the default GPU will be used to run inference. To disable this behavior, users can use the MLFLOW_DEFAULT_PREDICTION_DEVICE or pass in a device with the device parameter for the predict function.   Note In case of multi gpu training, ensure to save the model only with global rank 0 gpu. This avoids logging multiple copies of the same model.   PyTorch pyfunc usage  For a minimal PyTorch model, an example configuration for the pyfunc predict() method is: import numpy as np import mlflow from mlflow.models import infer_signature import torch from torch import nn   net = nn.Linear(6, 1) loss_function = nn.L1Loss() optimizer = torch.optim.Adam(net.parameters(), lr=1e-4)  X = torch.randn(6) y = torch.randn(1)  epochs = 5 for epoch in range(epochs):     optimizer.zero_grad()     outputs = net(X)      loss = loss_function(outputs, y)     loss.backward()      optimizer.step()  with mlflow.start_run() as run:     signature = infer_signature(X.numpy(), net(X).detach().numpy())     model_info = mlflow.pytorch.log_model(net, ""model"", signature=signature)  pytorch_pyfunc = mlflow.pyfunc.load_model(model_uri=model_info.model_uri)  predictions = pytorch_pyfunc.predict(torch.randn(6).numpy()) print(predictions)   For more information, see mlflow.pytorch.    Scikit-learn (sklearn)  The sklearn model flavor provides an easy-to-use interface for saving and loading scikit-learn models. The mlflow.sklearn module defines save_model() and log_model() functions that save scikit-learn models in MLflow format, using either Python's pickle module (Pickle) or CloudPickle for model serialization. These functions produce MLflow Models with the python_function flavor, allowing them to be loaded as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can only be scored with DataFrame input. Finally, you can use the mlflow.sklearn.load_model() method to load MLflow Models with the sklearn flavor as scikit-learn model objects.  Scikit-learn pyfunc usage  For a Scikit-learn LogisticRegression model, an example configuration for the pyfunc predict() method is: import mlflow from mlflow.models import infer_signature import numpy as np from sklearn.linear_model import LogisticRegression  with mlflow.start_run():     X = np.array([-2, -1, 0, 1, 2, 1]).reshape(-1, 1)     y = np.array([0, 0, 1, 1, 1, 0])     lr = LogisticRegression()     lr.fit(X, y)     signature = infer_signature(X, lr.predict(X))      model_info = mlflow.sklearn.log_model(         sk_model=lr, artifact_path=""model"", signature=signature     )  sklearn_pyfunc = mlflow.pyfunc.load_model(model_uri=model_info.model_uri)  data = np.array([-4, 1, 0, 10, -2, 1]).reshape(-1, 1)  predictions = sklearn_pyfunc.predict(data)   For more information, see mlflow.sklearn.    Spark MLlib (spark)  The spark model flavor enables exporting Spark MLlib models as MLflow Models. The mlflow.spark module defines  save_model() to save a Spark MLlib model to a DBFS path. log_model() to upload a Spark MLlib model to the tracking server. mlflow.spark.load_model() to load MLflow Models with the spark flavor as Spark MLlib pipelines.  MLflow Models produced by these functions contain the python_function flavor, allowing you to load them as generic Python functions via mlflow.pyfunc.load_model(). This loaded PyFunc model can only be scored with DataFrame input. When a model with the spark flavor is loaded as a Python function via mlflow.pyfunc.load_model(), a new SparkContext is created for model inference; additionally, the function converts all Pandas DataFrame inputs to Spark DataFrames before scoring. While this initialization overhead and format translation latency is not ideal for high-performance use cases, it enables you to easily deploy any MLlib PipelineModel to any production environment supported by MLflow (SageMaker, AzureML, etc).  Spark MLlib pyfunc usage  from pyspark.ml.classification import LogisticRegression from pyspark.ml.linalg import Vectors from pyspark.sql import SparkSession import mlflow  # Prepare training data from a list of (label, features) tuples. spark = SparkSession.builder.appName(""LogisticRegressionExample"").getOrCreate() training = spark.createDataFrame(     [         (1.0, Vectors.dense([0.0, 1.1, 0.1])),         (0.0, Vectors.dense([2.0, 1.0, -1.0])),         (0.0, Vectors.dense([2.0, 1.3, 1.0])),         (1.0, Vectors.dense([0.0, 1.2, -0.5])),     ],     [""label"", ""features""], )  # Create and fit a LogisticRegression instance lr = LogisticRegression(maxIter=10, regParam=0.01) lr_model = lr.fit(training)  # Serialize the Model with mlflow.start_run():     model_info = mlflow.spark.log_model(lr_model, ""spark-model"")  # Load saved model lr_model_saved = mlflow.pyfunc.load_model(model_info.model_uri)  # Make predictions on test data. # The DataFrame used in the predict method must be a Pandas DataFrame test = spark.createDataFrame(     [         (1.0, Vectors.dense([-1.0, 1.5, 1.3])),         (0.0, Vectors.dense([3.0, 2.0, -0.1])),         (1.0, Vectors.dense([0.0, 2.2, -1.5])),     ],     [""label"", ""features""], ).toPandas()  prediction = lr_model_saved.predict(test)    Note Note that when the sample_input parameter is provided to log_model() or save_model(), the Spark model is automatically saved as an mleap flavor by invoking mlflow.mleap.add_to_model(). For example, the follow code block: training_df = spark.createDataFrame([     (0, ""a b c d e spark"", 1.0),     (1, ""b d"", 0.0),     (2, ""spark f g h"", 1.0),     (3, ""hadoop mapreduce"", 0.0) ], [""id"", ""text"", ""label""])  tokenizer = Tokenizer(inputCol=""text"", outputCol=""words"") hashingTF = HashingTF(inputCol=tokenizer.getOutputCol(), outputCol=""features"") lr = LogisticRegression(maxIter=10, regParam=0.001) pipeline = Pipeline(stages=[tokenizer, hashingTF, lr]) model = pipeline.fit(training_df)  mlflow.spark.log_model(model, ""spark-model"", sample_input=training_df)   results in the following directory structure logged to the MLflow Experiment: # Directory written by with the addition of mlflow.mleap.add_to_model(model, ""spark-model"", training_df) # Note the addition of the mleap directory spark-model/ ├── mleap ├── sparkml ├── MLmodel ├── conda.yaml ├── python_env.yaml └── requirements.txt   For more information, see mlflow.mleap.  For more information, see mlflow.spark.    TensorFlow (tensorflow)  The simple example below shows how to log params and metrics in mlflow for a custom training loop using low-level TensorFlow API. See tf-keras-example. for an example of mlflow and tf.keras models. import numpy as np import tensorflow as tf  import mlflow  x = np.linspace(-4, 4, num=512) y = 3 * x + 10  # estimate w and b where y = w * x + b learning_rate = 0.1 x_train = tf.Variable(x, trainable=False, dtype=tf.float32) y_train = tf.Variable(y, trainable=False, dtype=tf.float32)  # initial values w = tf.Variable(1.0) b = tf.Variable(1.0)  with mlflow.start_run():     mlflow.log_param(""learning_rate"", learning_rate)      for i in range(1000):         with tf.GradientTape(persistent=True) as tape:             # calculate MSE = 0.5 * (y_predict - y_train)^2             y_predict = w * x_train + b             loss = 0.5 * tf.reduce_mean(tf.square(y_predict - y_train))             mlflow.log_metric(""loss"", value=loss.numpy(), step=i)          # Update the trainable variables         # w = w - learning_rate * gradient of loss function w.r.t. w         # b = b - learning_rate * gradient of loss function w.r.t. b         w.assign_sub(learning_rate * tape.gradient(loss, w))         b.assign_sub(learning_rate * tape.gradient(loss, b))  print(f""W = {w.numpy():.2f}, b = {b.numpy():.2f}"")     ONNX (onnx)  The onnx model flavor enables logging of ONNX models in MLflow format via the mlflow.onnx.save_model() and mlflow.onnx.log_model() methods. These methods also add the python_function flavor to the MLflow Models that they produce, allowing the models to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can be scored with both DataFrame input and numpy array input. The python_function representation of an MLflow ONNX model uses the ONNX Runtime execution engine for evaluation. Finally, you can use the mlflow.onnx.load_model() method to load MLflow Models with the onnx flavor in native ONNX format. For more information, see mlflow.onnx and http://onnx.ai/.  Warning The default behavior for saving ONNX files is to use the ONNX save option save_as_external_data=True in order to support model files that are in excess of 2GB. For edge deployments of small model files, this may create issues. If you need to save a small model as a single file for such deployment considerations, you can set the parameter save_as_external_data=False in either mlflow.onnx.save_model() or mlflow.onnx.log_model() to force the serialization of the model as a small file. Note that if the model is in excess of 2GB, saving as a single file will not work.   ONNX pyfunc usage example  For an ONNX model, an example configuration that uses pytorch to train a dummy model, converts it to ONNX, logs to mlflow and makes a prediction using pyfunc predict() method is: import numpy as np import mlflow from mlflow.models import infer_signature import onnx import torch from torch import nn  # define a torch model net = nn.Linear(6, 1) loss_function = nn.L1Loss() optimizer = torch.optim.Adam(net.parameters(), lr=1e-4)  X = torch.randn(6) y = torch.randn(1)  # run model training epochs = 5 for epoch in range(epochs):     optimizer.zero_grad()     outputs = net(X)      loss = loss_function(outputs, y)     loss.backward()      optimizer.step()  # convert model to ONNX and load it torch.onnx.export(net, X, ""model.onnx"") onnx_model = onnx.load_model(""model.onnx"")  # log the model into a mlflow run with mlflow.start_run():     signature = infer_signature(X.numpy(), net(X).detach().numpy())     model_info = mlflow.onnx.log_model(onnx_model, ""model"", signature=signature)  # load the logged model and make a prediction onnx_pyfunc = mlflow.pyfunc.load_model(model_info.model_uri)  predictions = onnx_pyfunc.predict(X.numpy()) print(predictions)      MXNet Gluon (gluon)   Warning The gluon model flavor is deprecated and will be removed in a future release.  The gluon model flavor enables logging of Gluon models in MLflow format via the mlflow.gluon.save_model() and mlflow.gluon.log_model() methods. These methods also add the python_function flavor to the MLflow Models that they produce, allowing the models to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can be scored with both DataFrame input and numpy array input. You can also use the mlflow.gluon.load_model() method to load MLflow Models with the gluon flavor in native Gluon format.  Gluon pyfunc usage  For a minimal gluon model, here is an example of the pyfunc predict() method with a logistic regression model : import mlflow import mxnet as mx from mxnet import nd, autograd, gluon from mxnet.gluon import nn, Trainer from mxnet.gluon.data import DataLoader, ArrayDataset import numpy as np  # this example requires a compatible version of numpy : numpy == 1.23.1 # `pip uninstall numpy`  `python -m pip install numpy==1.23.1`   def get_random_data(size, ctx):     x = nd.normal(0, 1, shape=(size, 10), ctx=ctx)     y = x.sum(axis=1) > 3     return x, y   # use cpu for this example, gpu could be used with ctx=gpu() ctx = mx.cpu() train_data_size = 1000 val_data_size = 100 batch_size = 10  train_x, train_ground_truth_class = get_random_data(train_data_size, ctx) train_dataset = ArrayDataset(train_x, train_ground_truth_class) train_dataloader = DataLoader(     train_dataset,     batch_size=batch_size,     shuffle=True, )  val_x, val_ground_truth_class = get_random_data(val_data_size, ctx) val_dataset = ArrayDataset(val_x, val_ground_truth_class) val_dataloader = DataLoader(val_dataset, batch_size=batch_size, shuffle=True)  net = nn.HybridSequential()  with net.name_scope():     net.add(nn.Dense(units=10, activation=""relu""))  # input layer     net.add(nn.Dense(units=10, activation=""relu""))  # inner layer 1     net.add(nn.Dense(units=10, activation=""relu""))  # inner layer 2     net.add(nn.Dense(units=1))  # output layer: must have only 1 neuron  net.initialize(mx.init.Xavier())  loss = gluon.loss.SigmoidBinaryCrossEntropyLoss() trainer = Trainer(     params=net.collect_params(),     optimizer=""sgd"",     optimizer_params={""learning_rate"": 0.1}, )  accuracy = mx.metric.Accuracy() f1 = mx.metric.F1() threshold = 0.5   def train_model():     cumulative_train_loss = 0      for i, (data, label) in enumerate(train_dataloader):         with autograd.record():             # do forward pass on a batch of training data             output = net(data)             # calculate loss for the training data batch             loss_result = loss(output, label)         # calculate gradients         loss_result.backward()         # update parameters of the network         trainer.step(batch_size)         # sum losses of every batch         cumulative_train_loss += nd.sum(loss_result).asscalar()      return cumulative_train_loss   def validate_model(threshold):     cumulative_val_loss = 0      for i, (val_data, val_ground_truth_class) in enumerate(val_dataloader):         # do forward pass on a batch of validation data         output = net(val_data)         # calculate cumulative validation loss         cumulative_val_loss += nd.sum(loss(output, val_ground_truth_class)).asscalar()         # prediction as a sigmoid         prediction = net(val_data).sigmoid()         # converting neuron outputs to classes         predicted_classes = mx.nd.ceil(prediction - threshold)         # update validation accuracy         accuracy.update(val_ground_truth_class, predicted_classes.reshape(-1))         # calculate probabilities of belonging to different classes         prediction = prediction.reshape(-1)         probabilities = mx.nd.stack(1 - prediction, prediction, axis=1)          f1.update(val_ground_truth_class, probabilities)      return cumulative_val_loss   # train model and get metrics cumulative_train_loss = train_model() cumulative_val_loss = validate_model(threshold) net.collect_params().initialize() metrics_to_log = {     ""training_loss"": cumulative_train_loss,     ""val_loss"": cumulative_val_loss,     ""f1"": f1.get()[1],     ""accuracy"": accuracy.get()[1], } params_to_log = {""learning_rate"": trainer.learning_rate, ""threshold"": threshold}  # the model needs to be hybridized and run forward at least once before export is called net.hybridize() net.forward(train_x)  with mlflow.start_run():     mlflow.log_params(params_to_log)     mlflow.log_metrics(metrics_to_log)     model_info = mlflow.gluon.log_model(net, ""model"")  # load the model pytorch_pyfunc = mlflow.pyfunc.load_model(model_uri=model_info.model_uri)  # make a prediction X = np.random.randn(10, 10) predictions = pytorch_pyfunc.predict(X) print(predictions)   For more information, see mlflow.gluon.    XGBoost (xgboost)  The xgboost model flavor enables logging of XGBoost models in MLflow format via the mlflow.xgboost.save_model() and mlflow.xgboost.log_model() methods in python and mlflow_save_model and mlflow_log_model in R respectively. These methods also add the python_function flavor to the MLflow Models that they produce, allowing the models to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can only be scored with DataFrame input. You can also use the mlflow.xgboost.load_model() method to load MLflow Models with the xgboost model flavor in native XGBoost format. Note that the xgboost model flavor only supports an instance of xgboost.Booster, not models that implement the scikit-learn API.  XGBoost pyfunc usage  The example below  Loads the IRIS dataset from scikit-learn Trains an XGBoost Classifier Logs the model and params using mlflow Loads the logged model and makes predictions  from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split from xgboost import XGBClassifier import mlflow from mlflow.models import infer_signature  data = load_iris() X_train, X_test, y_train, y_test = train_test_split(     data[""data""], data[""target""], test_size=0.2 )  xgb_classifier = XGBClassifier(     n_estimators=10,     max_depth=3,     learning_rate=1,     objective=""binary:logistic"",     random_state=123, )  # log fitted model and XGBClassifier parameters with mlflow.start_run():     xgb_classifier.fit(X_train, y_train)     clf_params = xgb_classifier.get_xgb_params()     mlflow.log_params(clf_params)     signature = infer_signature(X_train, xgb_classifier.predict(X_train))     model_info = mlflow.xgboost.log_model(         xgb_classifier, ""iris-classifier"", signature=signature     )  # Load saved model and make predictions xgb_classifier_saved = mlflow.pyfunc.load_model(model_info.model_uri) y_pred = xgb_classifier_saved.predict(X_test)   For more information, see mlflow.xgboost.    LightGBM (lightgbm)  The lightgbm model flavor enables logging of LightGBM models in MLflow format via the mlflow.lightgbm.save_model() and mlflow.lightgbm.log_model() methods. These methods also add the python_function flavor to the MLflow Models that they produce, allowing the models to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). You can also use the mlflow.lightgbm.load_model() method to load MLflow Models with the lightgbm model flavor in native LightGBM format. Note that the scikit-learn API for LightGBM is now supported. For more information, see mlflow.lightgbm.  LightGBM pyfunc usage  The example below  Loads the IRIS dataset from scikit-learn Trains a LightGBM LGBMClassifier Logs the model and feature importance's using mlflow Loads the logged model and makes predictions  from lightgbm import LGBMClassifier from sklearn.datasets import load_iris from sklearn.model_selection import train_test_split import mlflow from mlflow.models import infer_signature  data = load_iris()  # Remove special characters from feature names to be able to use them as keys for mlflow metrics feature_names = [     name.replace("" "", ""_"").replace(""("", """").replace("")"", """")     for name in data[""feature_names""] ] X_train, X_test, y_train, y_test = train_test_split(     data[""data""], data[""target""], test_size=0.2 ) # create model instance lgb_classifier = LGBMClassifier(     n_estimators=10,     max_depth=3,     learning_rate=1,     objective=""binary:logistic"",     random_state=123, )  # Fit and save model and LGBMClassifier feature importances as mlflow metrics with mlflow.start_run():     lgb_classifier.fit(X_train, y_train)     feature_importances = dict(zip(feature_names, lgb_classifier.feature_importances_))     feature_importance_metrics = {         f""feature_importance_{feature_name}"": imp_value         for feature_name, imp_value in feature_importances.items()     }     mlflow.log_metrics(feature_importance_metrics)     signature = infer_signature(X_train, lgb_classifier.predict(X_train))     model_info = mlflow.lightgbm.log_model(         lgb_classifier, ""iris-classifier"", signature=signature     )  # Load saved model and make predictions lgb_classifier_saved = mlflow.pyfunc.load_model(model_info.model_uri) y_pred = lgb_classifier_saved.predict(X_test) print(y_pred)      CatBoost (catboost)  The catboost model flavor enables logging of CatBoost models in MLflow format via the mlflow.catboost.save_model() and mlflow.catboost.log_model() methods. These methods also add the python_function flavor to the MLflow Models that they produce, allowing the models to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). You can also use the mlflow.catboost.load_model() method to load MLflow Models with the catboost model flavor in native CatBoost format. For more information, see mlflow.catboost.  CatBoost pyfunc usage  For a CatBoost Classifier model, an example configuration for the pyfunc predict() method is: import mlflow from mlflow.models import infer_signature from catboost import CatBoostClassifier from sklearn import datasets  # prepare data X, y = datasets.load_wine(as_frame=False, return_X_y=True)  # train the model model = CatBoostClassifier(     iterations=5,     loss_function=""MultiClass"",     allow_writing_files=False, ) model.fit(X, y)  # create model signature predictions = model.predict(X) signature = infer_signature(X, predictions)  # log the model into a mlflow run with mlflow.start_run():     model_info = mlflow.catboost.log_model(model, ""model"", signature=signature)  # load the logged model and make a prediction catboost_pyfunc = mlflow.pyfunc.load_model(model_uri=model_info.model_uri) print(catboost_pyfunc.predict(X[:5]))      Spacy(spaCy)  The spaCy model flavor enables logging of spaCy models in MLflow format via the mlflow.spacy.save_model() and mlflow.spacy.log_model() methods. Additionally, these methods add the python_function flavor to the MLflow Models that they produce, allowing the models to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can only be scored with DataFrame input. You can also use the mlflow.spacy.load_model() method to load MLflow Models with the spacy model flavor in native spaCy format. For more information, see mlflow.spacy.  Spacy pyfunc usage  The example below shows how to train a Spacy TextCategorizer model, log the model artifact and metrics to the mlflow tracking server and then load the saved model to make predictions. For this example, we will be using the Polarity 2.0 dataset available in the nltk package. This dataset consists of 10000 positive and 10000 negative short movie reviews. First we convert the texts and sentiment labels ("pos" or "neg") from NLTK native format to Spacy's DocBin format: import pandas as pd import spacy from nltk.corpus import movie_reviews from spacy import Language from spacy.tokens import DocBin  nltk.download(""movie_reviews"")   def get_sentences(sentiment_type: str) -> pd.DataFrame:     """"""Reconstruct the sentences from the word lists for each review record for a specific ``sentiment_type``     as a pandas DataFrame with two columns: 'sentence' and 'sentiment'.     """"""     file_ids = movie_reviews.fileids(sentiment_type)     sent_df = []     for file_id in file_ids:         sentence = "" "".join(movie_reviews.words(file_id))         sent_df.append({""sentence"": sentence, ""sentiment"": sentiment_type})     return pd.DataFrame(sent_df)   def convert(data_df: pd.DataFrame, target_file: str):     """"""Convert a DataFrame with 'sentence' and 'sentiment' columns to a     spacy DocBin object and save it to 'target_file'.     """"""     nlp = spacy.blank(""en"")     sentiment_labels = data_df.sentiment.unique()     spacy_doc = DocBin()      for _, row in data_df.iterrows():         sent_tokens = nlp.make_doc(row[""sentence""])         # To train a Spacy TextCategorizer model, the label must be attached to the ""cats"" dictionary of the ""Doc""         # object, e.g. {""pos"": 1.0, ""neg"": 0.0} for a ""pos"" label.         for label in sentiment_labels:             sent_tokens.cats[label] = 1.0 if label == row[""sentiment""] else 0.0         spacy_doc.add(sent_tokens)      spacy_doc.to_disk(target_file)   # Build a single DataFrame with both positive and negative reviews, one row per review review_data = [get_sentences(sentiment_type) for sentiment_type in (""pos"", ""neg"")] review_data = pd.concat(review_data, axis=0)  # Split the DataFrame into a train and a dev set train_df = review_data.groupby(""sentiment"", group_keys=False).apply(     lambda x: x.sample(frac=0.7, random_state=100) ) dev_df = review_data.loc[review_data.index.difference(train_df.index), :]  # Save the train and dev data files to the current directory as ""corpora.train"" and ""corpora.dev"", respectively convert(train_df, ""corpora.train"") convert(dev_df, ""corpora.dev"")   To set up the training job, we first need to generate a configuration file as described in the Spacy Documentation For simplicity, we will only use a TextCategorizer in the pipeline. python -m spacy init config --pipeline textcat --lang en mlflow-textcat.cfg   Change the default train and dev paths in the config file to the current directory:   [paths] - train = null - dev = null + train = ""."" + dev = "".""   In Spacy, the training loop is defined internally in Spacy's code. Spacy provides a "logging" extension point where we can use mlflow. To do this,  We have to define a function to write metrics / model input to mlfow Register it as a logger in Spacy's component registry Change the default console logger in the Spacy's configuration file (mlflow-textcat.cfg)  from typing import IO, Callable, Tuple, Dict, Any, Optional import spacy from spacy import Language import mlflow   @spacy.registry.loggers(""mlflow_logger.v1"") def mlflow_logger():     """"""Returns a function, ``setup_logger`` that returns two functions:      * ``log_step`` is called internally by Spacy for every evaluation step. We can log the intermediate train and     validation scores to the mlflow tracking server here.     * ``finalize``: is called internally by Spacy after training is complete. We can log the model artifact to the     mlflow tracking server here.     """"""      def setup_logger(         nlp: Language,         stdout: IO = sys.stdout,         stderr: IO = sys.stderr,     ) -> Tuple[Callable, Callable]:         def log_step(info: Optional[Dict[str, Any]]):             if info:                 step = info[""step""]                 score = info[""score""]                 metrics = {}                  for pipe_name in nlp.pipe_names:                     loss = info[""losses""][pipe_name]                     metrics[f""{pipe_name}_loss""] = loss                     metrics[f""{pipe_name}_score""] = score                 mlflow.log_metrics(metrics, step=step)          def finalize():             uri = mlflow.spacy.log_model(nlp, ""mlflow_textcat_example"")             mlflow.end_run()          return log_step, finalize      return setup_logger   Check the spacy-loggers library <https://pypi.org/project/spacy-loggers/> _ for a more complete implementation. Point to our mlflow logger in Spacy configuration file. For this example, we will lower the number of training steps and eval frequency:   [training.logger] - @loggers = ""spacy.ConsoleLogger.v1"" - dev = null + @loggers = ""mlflow_logger.v1""    [training] - max_steps = 20000 - eval_frequency = 100 + max_steps = 100 + eval_frequency = 10   Train our model: from spacy.cli.train import train as spacy_train  spacy_train(""mlflow-textcat.cfg"")   To make predictions, we load the saved model from the last run: from mlflow import MlflowClient  # look up the last run info from mlflow client = MlflowClient() last_run = client.search_runs(experiment_ids=[""0""], max_results=1)[0]  # We need to append the spacy model directory name to the artifact uri spacy_model = mlflow.pyfunc.load_model(     f""{last_run.info.artifact_uri}/mlflow_textcat_example"" ) predictions_in = dev_df.loc[:, [""sentence""]] predictions_out = spacy_model.predict(predictions_in).squeeze().tolist() predicted_labels = [     ""pos"" if row[""pos""] > row[""neg""] else ""neg"" for row in predictions_out ] print(dev_df.assign(predicted_sentiment=predicted_labels))      Fastai(fastai)  The fastai model flavor enables logging of fastai Learner models in MLflow format via the mlflow.fastai.save_model() and mlflow.fastai.log_model() methods. Additionally, these methods add the python_function flavor to the MLflow Models that they produce, allowing the models to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can only be scored with DataFrame input. You can also use the mlflow.fastai.load_model() method to load MLflow Models with the fastai model flavor in native fastai format. The interface for utilizing a fastai model loaded as a pyfunc type for generating predictions uses a Pandas DataFrame argument. This example runs the fastai tabular tutorial, logs the experiments, saves the model in fastai format and loads the model to get predictions using a fastai data loader: from fastai.data.external import URLs, untar_data from fastai.tabular.core import Categorify, FillMissing, Normalize, TabularPandas from fastai.tabular.data import TabularDataLoaders from fastai.tabular.learner import tabular_learner from fastai.data.transforms import RandomSplitter from fastai.metrics import accuracy from fastcore.basics import range_of import pandas as pd import mlflow import mlflow.fastai   def print_auto_logged_info(r):     tags = {k: v for k, v in r.data.tags.items() if not k.startswith(""mlflow."")}     artifacts = [         f.path for f in mlflow.MlflowClient().list_artifacts(r.info.run_id, ""model"")     ]     print(f""run_id: {r.info.run_id}"")     print(f""artifacts: {artifacts}"")     print(f""params: {r.data.params}"")     print(f""metrics: {r.data.metrics}"")     print(f""tags: {tags}"")   def main(epochs=5, learning_rate=0.01):     path = untar_data(URLs.ADULT_SAMPLE)     path.ls()      df = pd.read_csv(path / ""adult.csv"")      dls = TabularDataLoaders.from_csv(         path / ""adult.csv"",         path=path,         y_names=""salary"",         cat_names=[             ""workclass"",             ""education"",             ""marital-status"",             ""occupation"",             ""relationship"",             ""race"",         ],         cont_names=[""age"", ""fnlwgt"", ""education-num""],         procs=[Categorify, FillMissing, Normalize],     )      splits = RandomSplitter(valid_pct=0.2)(range_of(df))      to = TabularPandas(         df,         procs=[Categorify, FillMissing, Normalize],         cat_names=[             ""workclass"",             ""education"",             ""marital-status"",             ""occupation"",             ""relationship"",             ""race"",         ],         cont_names=[""age"", ""fnlwgt"", ""education-num""],         y_names=""salary"",         splits=splits,     )      dls = to.dataloaders(bs=64)      model = tabular_learner(dls, metrics=accuracy)      mlflow.fastai.autolog()      with mlflow.start_run() as run:         model.fit(5, 0.01)         mlflow.fastai.log_model(model, ""model"")      print_auto_logged_info(mlflow.get_run(run_id=run.info.run_id))      model_uri = f""runs:/{run.info.run_id}/model""     loaded_model = mlflow.fastai.load_model(model_uri)      test_df = df.copy()     test_df.drop([""salary""], axis=1, inplace=True)     dl = learn.dls.test_dl(test_df)      predictions, _ = loaded_model.get_preds(dl=dl)     px = pd.DataFrame(predictions).astype(""float"")     px.head(5)   main()   Output (Pandas DataFrame):        Index Probability of first class Probability of second class    0 0.545088 0.454912  1 0.503172 0.496828  2 0.962663 0.037337  3 0.206107 0.793893  4 0.807599 0.192401    Alternatively, when using the python_function flavor, get predictions from a DataFrame. from fastai.data.external import URLs, untar_data from fastai.tabular.core import Categorify, FillMissing, Normalize, TabularPandas from fastai.tabular.data import TabularDataLoaders from fastai.tabular.learner import tabular_learner from fastai.data.transforms import RandomSplitter from fastai.metrics import accuracy from fastcore.basics import range_of import pandas as pd import mlflow import mlflow.fastai  model_uri = ...  path = untar_data(URLs.ADULT_SAMPLE) df = pd.read_csv(path / ""adult.csv"") test_df = df.copy() test_df.drop([""salary""], axis=1, inplace=True)  loaded_model = mlflow.pyfunc.load_model(model_uri) loaded_model.predict(test_df)   Output (Pandas DataFrame):       Index Probability of first class, Probability of second class    0 [0.5450878, 0.45491222]  1 [0.50317234, 0.49682766]  2 [0.9626626, 0.037337445]  3 [0.20610662, 0.7938934]  4 [0.8075987, 0.19240129]    For more information, see mlflow.fastai.   Statsmodels (statsmodels)  The statsmodels model flavor enables logging of Statsmodels models in MLflow format via the mlflow.statsmodels.save_model() and mlflow.statsmodels.log_model() methods. These methods also add the python_function flavor to the MLflow Models that they produce, allowing the models to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can only be scored with DataFrame input. You can also use the mlflow.statsmodels.load_model() method to load MLflow Models with the statsmodels model flavor in native statsmodels format. As for now, automatic logging is restricted to parameters, metrics and models generated by a call to fit on a statsmodels model.  Statsmodels pyfunc usage  The following 2 examples illustrate usage of a basic regression model (OLS) and an ARIMA time series model from the following statsmodels apis : statsmodels.formula.api and statsmodels.tsa.api For a minimal statsmodels regression model, here is an example of the pyfunc predict() method : import mlflow import pandas as pd from sklearn.datasets import load_diabetes import statsmodels.formula.api as smf  # load the diabetes dataset from sklearn diabetes = load_diabetes()  # create X and y dataframes for the features and target X = pd.DataFrame(data=diabetes.data, columns=diabetes.feature_names) y = pd.DataFrame(data=diabetes.target, columns=[""target""])  # concatenate X and y dataframes df = pd.concat([X, y], axis=1)  # create the linear regression model (ordinary least squares) model = smf.ols(     formula=""target ~ age + sex + bmi + bp + s1 + s2 + s3 + s4 + s5 + s6"", data=df )  mlflow.statsmodels.autolog(     log_models=True,     disable=False,     exclusive=False,     disable_for_unsupported_versions=False,     silent=False,     registered_model_name=None, )  with mlflow.start_run():     res = model.fit(method=""pinv"", use_t=True)     model_info = mlflow.statsmodels.log_model(res, artifact_path=""OLS_model"")  # load the pyfunc model statsmodels_pyfunc = mlflow.pyfunc.load_model(model_uri=model_info.model_uri)  # generate predictions predictions = statsmodels_pyfunc.predict(X) print(predictions)   For a minimal time series ARIMA model, here is an example of the pyfunc predict() method : import mlflow import numpy as np import pandas as pd from statsmodels.tsa.arima.model import ARIMA  # create a time series dataset with seasonality np.random.seed(0)  # generate a time index with a daily frequency dates = pd.date_range(start=""2022-12-01"", end=""2023-12-01"", freq=""D"")  # generate the seasonal component (weekly) seasonality = np.sin(np.arange(len(dates)) * (2 * np.pi / 365.25) * 7)  # generate the trend component trend = np.linspace(-5, 5, len(dates)) + 2 * np.sin(     np.arange(len(dates)) * (2 * np.pi / 365.25) * 0.1 )  # generate the residual component residuals = np.random.normal(0, 1, len(dates))  # generate the final time series by adding the components time_series = seasonality + trend + residuals  # create a dataframe from the time series data = pd.DataFrame({""date"": dates, ""value"": time_series}) data.set_index(""date"", inplace=True)  order = (1, 0, 0) # create the ARIMA model model = ARIMA(data, order=order)  mlflow.statsmodels.autolog(     log_models=True,     disable=False,     exclusive=False,     disable_for_unsupported_versions=False,     silent=False,     registered_model_name=None, )  with mlflow.start_run():     res = model.fit()     mlflow.log_params(         {             ""order"": order,             ""trend"": model.trend,             ""seasonal_order"": model.seasonal_order,         }     )     mlflow.log_params(res.params)     mlflow.log_metric(""aic"", res.aic)     mlflow.log_metric(""bic"", res.bic)     model_info = mlflow.statsmodels.log_model(res, artifact_path=""ARIMA_model"")  # load the pyfunc model statsmodels_pyfunc = mlflow.pyfunc.load_model(model_uri=model_info.model_uri)  # prediction dataframes for a TimeSeriesModel must have exactly one row and include columns called start and end start = pd.to_datetime(""2024-01-01"") end = pd.to_datetime(""2024-01-07"")  # generate predictions prediction_data = pd.DataFrame({""start"": start, ""end"": end}, index=[0]) predictions = statsmodels_pyfunc.predict(prediction_data) print(predictions)   For more information, see mlflow.statsmodels.    Prophet (prophet)  The prophet model flavor enables logging of Prophet models in MLflow format via the mlflow.prophet.save_model() and mlflow.prophet.log_model() methods. These methods also add the python_function flavor to the MLflow Models that they produce, allowing the models to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can only be scored with DataFrame input. You can also use the mlflow.prophet.load_model() method to load MLflow Models with the prophet model flavor in native prophet format.  Prophet pyfunc usage  This example uses a time series dataset from Prophet's GitHub repository, containing log number of daily views to Peyton Manning's Wikipedia page for several years. A sample of the dataset is as follows:       ds y    2007-12-10 9.59076113897809  2007-12-11 8.51959031601596  2007-12-12 8.18367658262066  2007-12-13 8.07246736935477    import numpy as np import pandas as pd from prophet import Prophet from prophet.diagnostics import cross_validation, performance_metrics  import mlflow from mlflow.models import infer_signature  # starts on 2007-12-10, ends on 2016-01-20 train_df = pd.read_csv(     ""https://raw.githubusercontent.com/facebook/prophet/main/examples/example_wp_log_peyton_manning.csv"" )  # Create a ""test"" DataFrame with the ""ds"" column containing 10 days after the end date in train_df test_dates = pd.date_range(start=""2016-01-21"", end=""2016-01-31"", freq=""D"") test_df = pd.Series(data=test_dates.values, name=""ds"").to_frame()  prophet_model = Prophet(changepoint_prior_scale=0.5, uncertainty_samples=7)  with mlflow.start_run():     prophet_model.fit(train_df)      # extract and log parameters such as changepoint_prior_scale in the mlflow run     model_params = {         name: value for name, value in vars(prophet_model).items() if np.isscalar(value)     }     mlflow.log_params(model_params)      # cross validate with 900 days of data initially, predictions for next 30 days     # walk forward by 30 days     cv_results = cross_validation(         prophet_model, initial=""900 days"", period=""30 days"", horizon=""30 days""     )      # Calculate metrics from cv_results, then average each metric across all backtesting windows and log to mlflow     cv_metrics = [""mse"", ""rmse"", ""mape""]     metrics_results = performance_metrics(cv_results, metrics=cv_metrics)     average_metrics = metrics_results.loc[:, cv_metrics].mean(axis=0).to_dict()     mlflow.log_metrics(average_metrics)      # Calculate model signature     train = prophet_model.history     predictions = prophet_model.predict(prophet_model.make_future_dataframe(30))     signature = infer_signature(train, predictions)      model_info = mlflow.prophet.log_model(         prophet_model, ""prophet-model"", signature=signature     )  # Load saved model prophet_model_saved = mlflow.pyfunc.load_model(model_info.model_uri) predictions = prophet_model_saved.predict(test_df)   Output (Pandas DataFrame):          Index ds yhat yhat_upper yhat_lower    0 2016-01-21 8.526513 8.827397 8.328563  1 2016-01-22 8.541355 9.434994 8.112758  2 2016-01-23 8.308332 8.633746 8.201323  3 2016-01-24 8.676326 9.534593 8.020874  4 2016-01-25 8.983457 9.430136 8.121798    For more information, see mlflow.prophet.    Pmdarima (pmdarima)  The pmdarima model flavor enables logging of pmdarima models in MLflow format via the mlflow.pmdarima.save_model() and mlflow.pmdarima.log_model() methods. These methods also add the python_function flavor to the MLflow Models that they produce, allowing the model to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can only be scored with a DataFrame input. You can also use the mlflow.pmdarima.load_model() method to load MLflow Models with the pmdarima model flavor in native pmdarima formats. The interface for utilizing a pmdarima model loaded as a pyfunc type for generating forecast predictions uses a single-row Pandas DataFrame configuration argument. The following columns in this configuration Pandas DataFrame are supported:   n_periods (required) - specifies the number of future periods to generate starting from the last datetime valueof the training dataset, utilizing the frequency of the input training series when the model was trained. (for example, if the training data series elements represent one value per hour, in order to forecast 3 days of future data, set the column n_periods to 72.     X (optional) - exogenous regressor values (only supported in pmdarima version >= 1.8.0) a 2D array of values forfuture time period events. For more information, read the underlying library explanation.     return_conf_int (optional) - a boolean (Default: False) for whether to return confidence interval values.See above note.    alpha (optional) - the significance value for calculating confidence intervals. (Default: 0.05)  An example configuration for the pyfunc predict of a pmdarima model is shown below, with a future period prediction count of 100, a confidence interval calculation generation, no exogenous regressor elements, and a default alpha of 0.05:        Index n_periods return_conf_int    0 100 True     Warning The Pandas DataFrame passed to a pmdarima pyfunc flavor must only contain 1 row.   Note When predicting a pmdarima flavor, the predict method's DataFrame configuration column return_conf_int's value controls the output format. When the column's value is set to False or None (which is the default if this column is not supplied in the configuration DataFrame), the schema of the returned Pandas DataFrame is a single column: [""yhat""]. When set to True, the schema of the returned DataFrame is: [""yhat"", ""yhat_lower"", ""yhat_upper""] with the respective lower (yhat_lower) and upper (yhat_upper) confidence intervals added to the forecast predictions (yhat).  Example usage of pmdarima artifact loaded as a pyfunc with confidence intervals calculated: import pmdarima import mlflow import pandas as pd  data = pmdarima.datasets.load_airpassengers()  with mlflow.start_run():     model = pmdarima.auto_arima(data, seasonal=True)     mlflow.pmdarima.save_model(model, ""/tmp/model.pmd"")  loaded_pyfunc = mlflow.pyfunc.load_model(""/tmp/model.pmd"")  prediction_conf = pd.DataFrame(     [{""n_periods"": 4, ""return_conf_int"": True, ""alpha"": 0.1}] )  predictions = loaded_pyfunc.predict(prediction_conf)   Output (Pandas DataFrame):         Index yhat yhat_lower yhat_upper    0 467.573731 423.30995 511.83751  1 490.494467 416.17449 564.81444  2 509.138684 420.56255 597.71117  3 492.554714 397.30634 587.80309     Warning Signature logging for pmdarima will not function correctly if return_conf_int is set to True from a non-pyfunc artifact. The output of the native ARIMA.predict() when returning confidence intervals is not a recognized signature type.    OpenAI (openai) (Experimental)   Attention The openai flavor is in active development and is marked as Experimental. Public APIs may change and new features are subject to be added as additional functionality is brought to the flavor.  The openi model flavor enables logging of OpenAI models in MLflow format via the mlflow.openai.save_model() and mlflow.openai.log_model() functions. Use of these functions also adds the python_function flavor to the MLflow Models that they produce, allowing the model to be interpreted as a generic Python function for inference via mlflow.pyfunc.load_model(). You can also use the mlflow.openai.load_model() function to load a saved or logged MLflow Model with the openai flavor as a dictionary of the model's attributes. Example: import logging import os  import openai import pandas as pd  import mlflow from mlflow.models.signature import ModelSignature from mlflow.types.schema import ColSpec, ParamSchema, ParamSpec, Schema  logging.getLogger(""mlflow"").setLevel(logging.ERROR)  # Uncomment the following lines to run this script without using a real OpenAI API key. # os.environ[""MLFLOW_TESTING""] = ""true"" # os.environ[""OPENAI_API_KEY""] = ""test""  assert ""OPENAI_API_KEY"" in os.environ, ""Please set the OPENAI_API_KEY environment variable.""   print(     """""" # ****************************************************************************** # Single variable # ****************************************************************************** """""" ) with mlflow.start_run():     model_info = mlflow.openai.log_model(         model=""gpt-3.5-turbo"",         task=openai.ChatCompletion,         artifact_path=""model"",         messages=[{""role"": ""user"", ""content"": ""Tell me a joke about {animal}.""}],     )   model = mlflow.pyfunc.load_model(model_info.model_uri) df = pd.DataFrame(     {         ""animal"": [             ""cats"",             ""dogs"",         ]     } ) print(model.predict(df))  list_of_dicts = [     {""animal"": ""cats""},     {""animal"": ""dogs""}, ] print(model.predict(list_of_dicts))  list_of_strings = [     ""cats"",     ""dogs"", ] print(model.predict(list_of_strings)) print(     """""" # ****************************************************************************** # Multiple variables # ****************************************************************************** """""" ) with mlflow.start_run():     model_info = mlflow.openai.log_model(         model=""gpt-3.5-turbo"",         task=openai.ChatCompletion,         artifact_path=""model"",         messages=[{""role"": ""user"", ""content"": ""Tell me a {adjective} joke about {animal}.""}],     )   model = mlflow.pyfunc.load_model(model_info.model_uri) df = pd.DataFrame(     {         ""adjective"": [""funny"", ""scary""],         ""animal"": [""cats"", ""dogs""],     } ) print(model.predict(df))   list_of_dicts = [     {""adjective"": ""funny"", ""animal"": ""cats""},     {""adjective"": ""scary"", ""animal"": ""dogs""}, ] print(model.predict(list_of_dicts))  print(     """""" # ****************************************************************************** # Multiple prompts # ****************************************************************************** """""" ) with mlflow.start_run():     model_info = mlflow.openai.log_model(         model=""gpt-3.5-turbo"",         task=openai.ChatCompletion,         artifact_path=""model"",         messages=[             {""role"": ""system"", ""content"": ""You are {person}""},             {""role"": ""user"", ""content"": ""Let me hear your thoughts on {topic}""},         ],     )   model = mlflow.pyfunc.load_model(model_info.model_uri) df = pd.DataFrame(     {         ""person"": [""Elon Musk"", ""Jeff Bezos""],         ""topic"": [""AI"", ""ML""],     } ) print(model.predict(df))  list_of_dicts = [     {""person"": ""Elon Musk"", ""topic"": ""AI""},     {""person"": ""Jeff Bezos"", ""topic"": ""ML""}, ] print(model.predict(list_of_dicts))   print(     """""" # ****************************************************************************** # No input variables # ****************************************************************************** """""" ) with mlflow.start_run():     model_info = mlflow.openai.log_model(         model=""gpt-3.5-turbo"",         task=openai.ChatCompletion,         artifact_path=""model"",         messages=[{""role"": ""system"", ""content"": ""You are Elon Musk""}],     )  model = mlflow.pyfunc.load_model(model_info.model_uri) df = pd.DataFrame(     {         ""question"": [             ""Let me hear your thoughts on AI"",             ""Let me hear your thoughts on ML"",         ],     } ) print(model.predict(df))  list_of_dicts = [     {""question"": ""Let me hear your thoughts on AI""},     {""question"": ""Let me hear your thoughts on ML""}, ] model = mlflow.pyfunc.load_model(model_info.model_uri) print(model.predict(list_of_dicts))  list_of_strings = [     ""Let me hear your thoughts on AI"",     ""Let me hear your thoughts on ML"", ] model = mlflow.pyfunc.load_model(model_info.model_uri) print(model.predict(list_of_strings))   print(     """""" # ****************************************************************************** # Inference parameters with chat completions # ****************************************************************************** """""" ) with mlflow.start_run():     model_info = mlflow.openai.log_model(         model=""gpt-3.5-turbo"",         task=openai.ChatCompletion,         artifact_path=""model"",         messages=[{""role"": ""user"", ""content"": ""Tell me a joke about {animal}.""}],         signature=ModelSignature(             inputs=Schema([ColSpec(type=""string"", name=None)]),             outputs=Schema([ColSpec(type=""string"", name=None)]),             params=ParamSchema(                 [                     ParamSpec(name=""temperature"", default=0, dtype=""float""),                 ]             ),         ),     )   model = mlflow.pyfunc.load_model(model_info.model_uri) df = pd.DataFrame(     {         ""animal"": [             ""cats"",             ""dogs"",         ]     } ) print(model.predict(df, params={""temperature"": 1}))     LangChain (langchain) (Experimental)   Attention The langchain flavor is in active development and is marked as Experimental. Public APIs may change and new features are subject to be added as additional functionality is brought to the flavor.  The langchain model flavor enables logging of LangChain models in MLflow format via the mlflow.langchain.save_model() and mlflow.langchain.log_model() functions. Use of these functions also adds the python_function flavor to the MLflow Models that they produce, allowing the model to be interpreted as a generic Python function for inference via mlflow.pyfunc.load_model(). You can also use the mlflow.langchain.load_model() function to load a saved or logged MLflow Model with the langchain flavor as a dictionary of the model's attributes. Example: Log a LangChain LLMChain import os  from langchain.chains import LLMChain from langchain.llms import OpenAI from langchain.prompts import PromptTemplate  import mlflow  assert ""OPENAI_API_KEY"" in os.environ, ""Please set the OPENAI_API_KEY environment variable.""  llm = OpenAI(temperature=0.9) prompt = PromptTemplate(     input_variables=[""product""],     template=""What is a good name for a company that makes {product}?"", ) chain = LLMChain(llm=llm, prompt=prompt)  with mlflow.start_run():     logged_model = mlflow.langchain.log_model(chain, ""langchain_model"")  loaded_model = mlflow.pyfunc.load_model(logged_model.model_uri) print(loaded_model.predict([{""product"": ""colorful socks""}]))    Output  [""\n\nColorful Cozy Creations.""]    Example: Log a LangChain Agent import os  from langchain.agents import AgentType, initialize_agent, load_tools from langchain.llms import OpenAI  import mlflow  assert ""OPENAI_API_KEY"" in os.environ, ""Please set the OPENAI_API_KEY environment variable."" assert ""SERPAPI_API_KEY"" in os.environ, ""Please set the SERPAPI_API_KEY environment variable.""  # First, let's load the language model we're going to use to control the agent. llm = OpenAI(temperature=0)  # Next, let's load some tools to use. Note that the `llm-math` tool uses an LLM, so we need to pass that in. tools = load_tools([""serpapi"", ""llm-math""], llm=llm)  # Finally, let's initialize an agent with the tools, the language model, and the type of agent we want to use. agent = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)  with mlflow.start_run():     logged_model = mlflow.langchain.log_model(agent, ""langchain_model"")  loaded_model = mlflow.pyfunc.load_model(logged_model.model_uri) print(     loaded_model.predict(         [             {                 ""input"": ""What was the high temperature in SF yesterday in Fahrenheit? What is that number raised to the .023 power?""             }         ]     ) )    Output  [""1.1044000282035853""]     Logging RetrievalQA Chains  In MLflow, you can use the langchain flavor to save a RetrievalQA chain, including the retriever object. Native LangChain requires the user to handle the serialization and deserialization of the retriever object, but MLflow's langchain flavor handles that for you. Here are the two things you need to tell MLflow:  Where the retriever object is stored (persist_dir). How to load the retriever object from that location (loader_fn).  After you define these, MLflow takes care of the rest, saving both the content in the persist_dir and pickling the loader_fn function. Example: Log a LangChain RetrievalQA Chain import os import tempfile  from langchain.chains import RetrievalQA from langchain.document_loaders import TextLoader from langchain.embeddings.openai import OpenAIEmbeddings from langchain.llms import OpenAI from langchain.text_splitter import CharacterTextSplitter from langchain.vectorstores import FAISS  import mlflow  assert ""OPENAI_API_KEY"" in os.environ, ""Please set the OPENAI_API_KEY environment variable.""  with tempfile.TemporaryDirectory() as temp_dir:     persist_dir = os.path.join(temp_dir, ""faiss_index"")      # Create the vector db, persist the db to a local fs folder     loader = TextLoader(""tests/langchain/state_of_the_union.txt"")     documents = loader.load()     text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)     docs = text_splitter.split_documents(documents)     embeddings = OpenAIEmbeddings()     db = FAISS.from_documents(docs, embeddings)     db.save_local(persist_dir)      # Create the RetrievalQA chain     retrievalQA = RetrievalQA.from_llm(llm=OpenAI(), retriever=db.as_retriever())      # Log the retrievalQA chain     def load_retriever(persist_directory):         embeddings = OpenAIEmbeddings()         vectorstore = FAISS.load_local(persist_directory, embeddings)         return vectorstore.as_retriever()      with mlflow.start_run() as run:         logged_model = mlflow.langchain.log_model(             retrievalQA,             artifact_path=""retrieval_qa"",             loader_fn=load_retriever,             persist_dir=persist_dir,         )  # Load the retrievalQA chain loaded_model = mlflow.pyfunc.load_model(logged_model.model_uri) print(loaded_model.predict([{""query"": ""What did the president say about Ketanji Brown Jackson""}]))    Output (truncated)  ["" The president said...""]      Logging a retriever and evaluate it individually  The langchain flavor provides the functionality to log a retriever object and evaluate it individually. This is useful if you want to evaluate the quality of the relevant documents returned by a retriever object without directing these documents through a large language model (LLM) to yield a summarized response. In order to log the retriever object in the langchain flavor, it is also required to specify persist_dir and loader_fn, the same as logging the RetrievalQA chain. See the previous section for details about these parameters. See the following example for more details. Example: Log a LangChain Retriever import os import tempfile  from langchain.document_loaders import TextLoader from langchain.embeddings.openai import OpenAIEmbeddings from langchain.text_splitter import CharacterTextSplitter from langchain.vectorstores import FAISS  import mlflow  assert ""OPENAI_API_KEY"" in os.environ, ""Please set the OPENAI_API_KEY environment variable.""  with tempfile.TemporaryDirectory() as temp_dir:     persist_dir = os.path.join(temp_dir, ""faiss_index"")      # Create the vector db, persist the db to a local fs folder     loader = TextLoader(""tests/langchain/state_of_the_union.txt"")     documents = loader.load()     text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)     docs = text_splitter.split_documents(documents)     embeddings = OpenAIEmbeddings()     db = FAISS.from_documents(docs, embeddings)     db.save_local(persist_dir)      def load_retriever(persist_directory):         embeddings = OpenAIEmbeddings()         vectorstore = FAISS.load_local(persist_directory, embeddings)         return vectorstore.as_retriever()      # Log the retriever     with mlflow.start_run() as run:         logged_model = mlflow.langchain.log_model(             db.as_retriever(),             artifact_path=""retriever"",             loader_fn=load_retriever,             persist_dir=persist_dir,         )  # Load the retriever chain loaded_model = mlflow.pyfunc.load_model(logged_model.model_uri) print(loaded_model.predict([{""query"": ""What did the president say about Ketanji Brown Jackson""}]))    Output (truncated)  [     [         {             ""page_content"": ""Tonight. I call..."",             ""metadata"": {""source"": ""/state.txt""},         },         {             ""page_content"": ""A former top..."",             ""metadata"": {""source"": ""/state.txt""},         },     ] ]       John Snow Labs (johnsnowlabs) (Experimental)   Attention The johnsnowlabs flavor is in active development and is marked as Experimental. Public APIs may change and new features are subject to be added as additional functionality is brought to the flavor.  The johnsnowlabs model flavor gives you access to 20.000+ state-of-the-art enterprise NLP models in 200+ languages for medical, finance, legal and many more domains. You can use mlflow.johnsnowlabs.log_model() to log and export your model as mlflow.pyfunc.PyFuncModel. This enables you to integrate any John Snow Labs model into the MLflow framework. You can easily deploy your models for inference with MLflows serve functionalities. Models are interpreted as a generic Python function for inference via mlflow.pyfunc.load_model(). You can also use the mlflow.johnsnowlabs.load_model() function to load a saved or logged MLflow Model with the johnsnowlabs flavor from an stored artifact. Features include: LLM's, Text Summarization, Question Answering, Named Entity Recognition, Relation Extraction, Sentiment Analysis, Spell Checking, Image Classification, Automatic Speech Recognition and much more, powered by the latest Transformer Architectures. The models are provided by John Snow Labs and requires a John Snow Labs Enterprise NLP License. You can reach out to us for a research or industry license. Example: Export a John Snow Labs to MLflow format import json import os  import pandas as pd from johnsnowlabs import nlp  import mlflow from mlflow.pyfunc import spark_udf  # 1) Write your raw license.json string into the 'JOHNSNOWLABS_LICENSE_JSON' env variable for MLflow creds = {     ""AWS_ACCESS_KEY_ID"": ""..."",     ""AWS_SECRET_ACCESS_KEY"": ""..."",     ""SPARK_NLP_LICENSE"": ""..."",     ""SECRET"": ""..."", } os.environ[""JOHNSNOWLABS_LICENSE_JSON""] = json.dumps(creds)  # 2) Install enterprise libraries nlp.install() # 3) Start a Spark session with enterprise libraries spark = nlp.start()  # 4) Load a model and test it nlu_model = ""en.classify.bert_sequence.covid_sentiment"" model_save_path = ""my_model"" johnsnowlabs_model = nlp.load(nlu_model) johnsnowlabs_model.predict([""I hate COVID,"", ""I love COVID""])  # 5) Export model with pyfunc and johnsnowlabs flavors with mlflow.start_run():     model_info = mlflow.johnsnowlabs.log_model(johnsnowlabs_model, model_save_path)  # 6) Load model with johnsnowlabs flavor mlflow.johnsnowlabs.load_model(model_info.model_uri)  # 7) Load model with pyfunc flavor mlflow.pyfunc.load_model(model_save_path)  pandas_df = pd.DataFrame({""text"": [""Hello World""]}) spark_df = spark.createDataFrame(pandas_df).coalesce(1) pyfunc_udf = spark_udf(     spark=spark,     model_uri=model_save_path,     env_manager=""virtualenv"",     result_type=""string"", ) new_df = spark_df.withColumn(""prediction"", pyfunc_udf(*pandas_df.columns))  # 9) You can now use the mlflow models serve command to serve the model see next section  # 10)  You can also use x command to deploy model inside of a container see next section    To deploy the John Snow Labs model as a container   Start the Docker Container  docker run -p 5001:8080 -e JOHNSNOWLABS_LICENSE_JSON=your_json_string ""mlflow-pyfunc""    Query server  curl http://127.0.0.1:5001/invocations -H 'Content-Type: application/json' -d '{   ""dataframe_split"": {       ""columns"": [""text""],       ""data"": [[""I hate covid""], [""I love covid""]]   } }'     To deploy the John Snow Labs model without a container   Export env variable and start server  export JOHNSNOWLABS_LICENSE_JSON=your_json_string mlflow models serve -m <model_uri>    Query server  curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '{   ""dataframe_split"": {       ""columns"": [""text""],       ""data"": [[""I hate covid""], [""I love covid""]]   } }'      Diviner (diviner)  The diviner model flavor enables logging of diviner models in MLflow format via the mlflow.diviner.save_model() and mlflow.diviner.log_model() methods. These methods also add the python_function flavor to the MLflow Models that they produce, allowing the model to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can only be scored with a DataFrame input. You can also use the mlflow.diviner.load_model() method to load MLflow Models with the diviner model flavor in native diviner formats.  Diviner Types  Diviner is a library that provides an orchestration framework for performing time series forecasting on groups of related series. Forecasting in diviner is accomplished through wrapping popular open source libraries such as prophet and pmdarima. The diviner library offers a simplified set of APIs to simultaneously generate distinct time series forecasts for multiple data groupings using a single input DataFrame and a unified high-level API.   Metrics and Parameters logging for Diviner  Unlike other flavors that are supported in MLflow, Diviner has the concept of grouped models. As a collection of many (perhaps thousands) of individual forecasting models, the burden to the tracking server to log individual metrics and parameters for each of these models is significant. For this reason, metrics and parameters are exposed for retrieval from Diviner's APIs as Pandas DataFrames, rather than discrete primitive values. To illustrate, let us assume we are forecasting hourly electricity consumption from major cities around the world. A sample of our input data looks like this:         country city datetime watts    US NewYork 2022-03-01 00:01:00 23568.9  US NewYork 2022-03-01 00:02:00 22331.7  US Boston 2022-03-01 00:01:00 14220.1  US Boston 2022-03-01 00:02:00 14183.4  CA Toronto 2022-03-01 00:01:00 18562.2  CA Toronto 2022-03-01 00:02:00 17681.6  MX MexicoCity 2022-03-01 00:01:00 19946.8  MX MexicoCity 2022-03-01 00:02:00 19444.0    If we were to fit a model on this data, supplying the grouping keys as: grouping_keys = [""country"", ""city""]   We will have a model generated for each of the grouping keys that have been supplied: [(""US"", ""NewYork""), (""US"", ""Boston""), (""CA"", ""Toronto""), (""MX"", ""MexicoCity"")]   With a model constructed for each of these, entering each of their metrics and parameters wouldn't be an issue for the MLflow tracking server. What would become a problem, however, is if we modeled each major city on the planet and ran this forecasting scenario every day. If we were to adhere to the conditions of the World Bank, that would mean just over 10,000 models as of 2022. After a mere few weeks of running this forecasting every day we would have a very large metrics table. To eliminate this issue for large-scale forecasting, the metrics and parameters for diviner are extracted as a grouping key indexed Pandas DataFrame, as shown below for example (float values truncated for visibility):              grouping_key_columns country city mse rmse mae mape mdape smape    "('country', 'city')" CA Toronto 8276851.6 2801.7 2417.7 0.16 0.16 0.159  "('country', 'city')" MX MexicoCity 3548872.4 1833.8 1584.5 0.15 0.16 0.159  "('country', 'city')" US NewYork 3167846.4 1732.4 1498.2 0.15 0.16 0.158  "('country', 'city')" US Boston 14082666.4 3653.2 3156.2 0.15 0.16 0.159    There are two recommended means of logging the metrics and parameters from a diviner model :  Writing the DataFrames to local storage and using mlflow.log_artifacts()  import os import mlflow import tempfile  with tempfile.TemporaryDirectory() as tmpdir:     params = model.extract_model_params()     metrics = model.cross_validate_and_score(         horizon=""72 hours"",         period=""240 hours"",         initial=""480 hours"",         parallel=""threads"",         rolling_window=0.1,         monthly=False,     )     params.to_csv(f""{tmpdir}/params.csv"", index=False, header=True)     metrics.to_csv(f""{tmpdir}/metrics.csv"", index=False, header=True)      mlflow.log_artifacts(tmpdir, artifact_path=""data"")    Writing directly as a JSON artifact using mlflow.log_dict()   Note The parameters extract from diviner models may require casting (or dropping of columns) if using the pd.DataFrame.to_dict() approach due to the inability of this method to serialize objects.  import mlflow  params = model.extract_model_params() metrics = model.cross_validate_and_score(     horizon=""72 hours"",     period=""240 hours"",     initial=""480 hours"",     parallel=""threads"",     rolling_window=0.1,     monthly=False, ) params[""t_scale""] = params[""t_scale""].astype(str) params[""start""] = params[""start""].astype(str) params = params.drop(""stan_backend"", axis=1)  mlflow.log_dict(params.to_dict(), ""params.json"") mlflow.log_dict(metrics.to_dict(), ""metrics.json"")   Logging of the model artifact is shown in the pyfunc example below.   Diviner pyfunc usage  The MLflow Diviner flavor includes an implementation of the pyfunc interface for Diviner models. To control prediction behavior, you can specify configuration arguments in the first row of a Pandas DataFrame input. As this configuration is dependent upon the underlying model type (i.e., the diviner.GroupedProphet.forecast() method has a different signature than does diviner.GroupedPmdarima.predict()), the Diviner pyfunc implementation attempts to coerce arguments to the types expected by the underlying model.  Note Diviner models support both "full group" and "partial group" forecasting. If a column named ""groups"" is present in the configuration DataFrame submitted to the pyfunc flavor, the grouping key values in the first row will be used to generate a subset of forecast predictions. This functionality removes the need to filter a subset from the full output of all groups forecasts if the results of only a few (or one) groups are needed.  For a GroupedPmdarima model, an example configuration for the pyfunc predict() method is: import mlflow import pandas as pd from pmdarima.arima.auto import AutoARIMA from diviner import GroupedPmdarima  with mlflow.start_run():     base_model = AutoARIMA(out_of_sample_size=96, maxiter=200)     model = GroupedPmdarima(model_template=base_model).fit(         df=df,         group_key_columns=[""country"", ""city""],         y_col=""watts"",         datetime_col=""datetime"",         silence_warnings=True,     )      mlflow.diviner.save_model(diviner_model=model, path=""/tmp/diviner_model"")  diviner_pyfunc = mlflow.pyfunc.load_model(model_uri=""/tmp/diviner_model"")  predict_conf = pd.DataFrame(     {         ""n_periods"": 120,         ""groups"": [             (""US"", ""NewYork""),             (""CA"", ""Toronto""),             (""MX"", ""MexicoCity""),         ],  # NB: List of tuples required.         ""predict_col"": ""wattage_forecast"",         ""alpha"": 0.1,         ""return_conf_int"": True,         ""on_error"": ""warn"",     },     index=[0], )  subset_forecasts = diviner_pyfunc.predict(predict_conf)    Note There are several instances in which a configuration DataFrame submitted to the pyfunc predict() method will cause an MlflowException to be raised:   If neither horizon or n_periods are provided. The value of n_periods or horizon is not an integer. If the model is of type GroupedProphet, frequency as a string type must be provided. If both horizon and n_periods are provided with different values.       Transformers (transformers) (Experimental)   Attention The transformers flavor is in active development and is marked as Experimental. Public APIs may change and new features are subject to be added as additional functionality is brought to the flavor.  The transformers model flavor enables logging of transformers models, components, and pipelines in MLflow format via the mlflow.transformers.save_model() and mlflow.transformers.log_model() functions. Use of these functions also adds the python_function flavor to the MLflow Models that they produce, allowing the model to be interpreted as a generic Python function for inference via mlflow.pyfunc.load_model(). You can also use the mlflow.transformers.load_model() function to load a saved or logged MLflow Model with the transformers flavor in the native transformers formats.  Input and Output types for PyFunc  The transformers python_function (pyfunc) model flavor simplifies and standardizes both the inputs and outputs of pipeline inference. This conformity allows for serving and batch inference by coercing the data structures that are required for transformers inference pipelines to formats that are compatible with json serialization and casting to Pandas DataFrames.  Note Certain TextGenerationPipeline types, particularly instructional-based ones, may return the original prompt and included line-formatting carriage returns "n" in their outputs. For these pipeline types, if you would like to disable the prompt return, you can set the following in the model_config dictionary when saving or logging the model: "include_prompt": False. To remove the newline characters from within the body of the generated text output, you can add the "collapse_whitespace": True option to the model_config dictionary. If the pipeline type being saved does not inherit from TextGenerationPipeline, these options will not perform any modification to the output returned from pipeline inference.   Attention Not all transformers pipeline types are supported. See the table below for the list of currently supported Pipeline types that can be loaded as pyfunc. In the current version, audio and text-based large language models are supported for use with pyfunc, while computer vision, multi-modal, timeseries, reinforcement learning, and graph models are only supported for native type loading via mlflow.transformers.load_model() Future releases of MLflow will introduce pyfunc support for these additional types.  The table below shows the mapping of transformers pipeline types to the python_function (pyfunc) model flavor data type inputs and outputs.  Important The inputs and outputs of the pyfunc implementation of these pipelines are not guaranteed to match the input types and output types that would return from a native use of a given pipeline type. If your use case requires access to scores, top_k results, or other additional references within the output from a pipeline inference call, please use the native implementation by loading via mlflow.transformers.load_model() to receive the full output. Similarly, if your use case requires the use of raw tensor outputs or processing of outputs through an external processor module, load the model components directly as a dict by calling mlflow.transformers.load_model() and specify the return_type argument as 'components'.   Supported transformers Pipeline types for Pyfunc         Pipeline Type Input Type Output Type    Instructional Text Generation str or List[str] List[str]  Conversational str or List[str] List[str]  Summarization str or List[str] List[str]  Text Classification str or List[str] pd.DataFrame (dtypes: {'label': str, 'score': double})  Text Generation str or List[str] List[str]  Text2Text Generation str or List[str] List[str]  Token Classification str or List[str] List[str]  Translation str or List[str] List[str]  ZeroShot Classification* Dict[str, [List[str] | str]* pd.DataFrame (dtypes: {'sequence': str, 'labels': str, 'scores': double})  Table Question Answering** Dict[str, [List[str] | str]** List[str]  Question Answering*** Dict[str, str]*** List[str]  Fill Mask**** str or List[str]**** List[str]  Feature Extraction str or List[str] np.ndarray  AutomaticSpeechRecognition bytes*****, str, or np.ndarray List[str]  AudioClassification bytes*****, str, or np.ndarray pd.DataFrame (dtypes: {'label': str, 'score': double})    * A collection of these inputs can also be passed. The standard required key names are 'sequences' and 'candidate_labels', but these may vary. Check the input requirments for the architecture that you're using to ensure that the correct dictionary key names are provided. ** A collection of these inputs can also be passed. The reference table must be a json encoded dict (i.e. {'query': 'what did we sell most of?', 'table': json.dumps(table_as_dict)}) *** A collection of these inputs can also be passed. The standard required key names are 'question' and 'context'. Verify the expected input key names match the expected input to the model to ensure your inference request can be read properly. **** The mask syntax for the model that you've chosen is going to be specific to that model's implementation. Some are '[MASK]', while others are '<mask>'. Verify the expected syntax to avoid failed inference requests. ***** If using pyfunc in MLflow Model Serving for realtime inference, the raw audio in bytes format must be base64 encoded prior to submitting to the endpoint. String inputs will be interpreted as uri locations.   Using model_config and model signature params for transformers inference  For transformers inference, there are two ways to pass in additional arguments to the pipeline.  Use model_config when saving/logging the model. Optionally, specify model_config when calling load_model. Specify params at inference time when calling predict()  Use model_config to control how the model is loaded and inference performed for all input samples. Configuration in model_config is not overridable at predict() time unless a ModelSignature is indicated with the same parameters. Use ModelSignature with params schema, on the other hand, to allow downstream consumers to provide additional inference params that may be needed to compute the predictions for their specific samples.  Note If both model_config and ModelSignature with parameters are saved when logging model, both of them will be used for inference. The default parameters in ModelSignature will override the params in model_config. If extra params are provided at inference time, they take precedence over all params. We recommend using model_config for those parameters needed to run the model in general for all the samples. Then, add ModelSignature with parameters for those extra parameters that you want downstream consumers to indicated at per each of the samples.   Using model_config  import mlflow from mlflow.models import infer_signature from mlflow.transformers import generate_signature_output import transformers  architecture = ""mrm8488/t5-base-finetuned-common_gen"" model = transformers.pipeline(     task=""text2text-generation"",     tokenizer=transformers.T5TokenizerFast.from_pretrained(architecture),     model=transformers.T5ForConditionalGeneration.from_pretrained(architecture), ) data = ""pencil draw paper""  # Infer the signature signature = infer_signature(     data,     generate_signature_output(model, data), )  # Define an model_config model_config = {     ""num_beams"": 5,     ""max_length"": 30,     ""do_sample"": True,     ""remove_invalid_values"": True, }  # Saving model_config with the model mlflow.transformers.save_model(     model,     path=""text2text"",     model_config=model_config,     signature=signature, )  pyfunc_loaded = mlflow.pyfunc.load_model(""text2text"") # model_config will be applied result = pyfunc_loaded.predict(data)  # overriding some inference configuration with diferent values pyfunc_loaded = mlflow.pyfunc.load_model(     ""text2text"", model_config=dict(do_sample=False) )    Note Note that in the previous example, the user can't override the configuration do_sample when calling predict.   Specifying params at inference time  import mlflow from mlflow.models import infer_signature from mlflow.transformers import generate_signature_output import transformers  architecture = ""mrm8488/t5-base-finetuned-common_gen"" model = transformers.pipeline(     task=""text2text-generation"",     tokenizer=transformers.T5TokenizerFast.from_pretrained(architecture),     model=transformers.T5ForConditionalGeneration.from_pretrained(architecture), ) data = ""pencil draw paper""  # Define an model_config model_config = {     ""num_beams"": 5,     ""remove_invalid_values"": True, }  # Define the inference parameters params inference_params = {     ""max_length"": 30,     ""do_sample"": True, }  # Infer the signature including params signature_with_params = infer_signature(     data,     generate_signature_output(model, data),     params=inference_params, )  # Saving model with signature and model config mlflow.transformers.save_model(     model,     path=""text2text"",     model_config=model_config,     signature=signature_with_params, )  pyfunc_loaded = mlflow.pyfunc.load_model(""text2text"")  # Pass params at inference time params = {     ""max_length"": 20,     ""do_sample"": False, }  # In this case we only override max_length and do_sample, # other params will use the default one saved on ModelSignature # or in the model configuration. # The final params used for prediction is as follows: # { #    ""num_beams"": 5, #    ""max_length"": 20, #    ""do_sample"": False, #    ""remove_invalid_values"": True, # } result = pyfunc_loaded.predict(data, params=params)     Example of loading a transformers model as a python function  In the below example, a simple pre-trained model is used within a pipeline. After logging to MLflow, the pipeline is loaded as a pyfunc and used to generate a response from a passed-in list of strings. import mlflow import transformers  # Read a pre-trained conversation pipeline from HuggingFace hub conversational_pipeline = transformers.pipeline(model=""microsoft/DialoGPT-medium"")  # Define the signature signature = mlflow.models.infer_signature(     ""Hi there, chatbot!"",     mlflow.transformers.generate_signature_output(         conversational_pipeline, ""Hi there, chatbot!""     ), )  # Log the pipeline with mlflow.start_run():     model_info = mlflow.transformers.log_model(         transformers_model=conversational_pipeline,         artifact_path=""chatbot"",         task=""conversational"",         signature=signature,         input_example=""A clever and witty question"",     )  # Load the saved pipeline as pyfunc chatbot = mlflow.pyfunc.load_model(model_uri=model_info.model_uri)  # Ask the chatbot a question response = chatbot.predict(""What is machine learning?"")  print(response)  # >> [It's a new thing that's been around for a while.]      Save and Load options for transformers  The transformers flavor for MLflow provides support for saving either components of a model or a pipeline object that contains the customized components in an easy to use interface that is optimized for inference.  Note MLflow by default uses a 500 MB max_shard_size to save the model object in mlflow.transformers.save_model() or mlflow.transformers.log_model() APIs. You can use the environment variable MLFLOW_HUGGINGFACE_MODEL_MAX_SHARD_SIZE to override the value.   Note For component-based logging, the only requirement that must be met in the submitted dict is that a model is provided. All other elements of the dict are optional.   Logging a components-based model  The example below shows logging components of a transformers model via a dictionary mapping of specific named components. The names of the keys within the submitted dictionary must be in the set: {""model"", ""tokenizer"", ""feature_extractor"", ""image_processor""}. Processor type objects (some image processors, audio processors, and multi-modal processors) must be saved explicitly with the processor argument in the mlflow.transformers.save_model() or mlflow.transformers.log_model() APIs. After logging, the components are automatically inserted into the appropriate Pipeline type for the task being performed and are returned, ready for inference.  Note The components that are logged can be retrieved in their original structure (a dictionary) by setting the attribute return_type to "components" in the load_model() API.   Attention Not all model types are compatible with the pipeline API constructor via component elements. Incompatible models will raise an MLflowException error stating that the model is missing the name_or_path attribute. In the event that this occurs, please construct the model directly via the transformers.pipeline(<repo name>) API and save the pipeline object directly.  import mlflow import transformers  task = ""text-classification"" architecture = ""distilbert-base-uncased-finetuned-sst-2-english"" model = transformers.AutoModelForSequenceClassification.from_pretrained(architecture) tokenizer = transformers.AutoTokenizer.from_pretrained(architecture)  # Define the components of the model in a dictionary transformers_model = {""model"": model, ""tokenizer"": tokenizer}  # Log the model components with mlflow.start_run():     model_info = mlflow.transformers.log_model(         transformers_model=transformers_model,         artifact_path=""text_classifier"",         task=task,     )  # Load the components as a pipeline loaded_pipeline = mlflow.transformers.load_model(     model_info.model_uri, return_type=""pipeline"" )  print(type(loaded_pipeline).__name__) # >> TextClassificationPipeline  loaded_pipeline([""MLflow is awesome!"", ""Transformers is a great library!""])  # >> [{'label': 'POSITIVE', 'score': 0.9998478889465332}, # >>  {'label': 'POSITIVE', 'score': 0.9998030066490173}]     Saving a pipeline and loading components  Some use cases can benefit from the simplicity of defining a solution as a pipeline, but need the component-level access for performing a micro-services based deployment strategy where pre / post-processing is performed on containers that do not house the model itself. For this paradigm, a pipeline can be loaded as its constituent parts, as shown below. import transformers import mlflow  translation_pipeline = transformers.pipeline(     task=""translation_en_to_fr"",     model=transformers.T5ForConditionalGeneration.from_pretrained(""t5-small""),     tokenizer=transformers.T5TokenizerFast.from_pretrained(         ""t5-small"", model_max_length=100     ), )  with mlflow.start_run():     model_info = mlflow.transformers.log_model(         transformers_model=translation_pipeline,         artifact_path=""french_translator"",     )  translation_components = mlflow.transformers.load_model(     model_info.model_uri, return_type=""components"" )  for key, value in translation_components.items():     print(f""{key} -> {type(value).__name__}"")  # >> task -> str # >> model -> T5ForConditionalGeneration # >> tokenizer -> T5TokenizerFast  response = translation_pipeline(""MLflow is great!"")  print(response)  # >> [{'translation_text': 'MLflow est formidable!'}]  reconstructed_pipeline = transformers.pipeline(**translation_components)  reconstructed_response = reconstructed_pipeline(     ""transformers makes using Deep Learning models easy and fun!"" )  print(reconstructed_response)  # >> [{'translation_text': ""Les transformateurs rendent l'utilisation de modèles Deep Learning facile et amusante!""}]     Automatic Metadata and ModelCard logging  In order to provide as much information as possible for saved models, the transformers flavor will automatically fetch the ModelCard for any model or pipeline that is saved that has a stored card on the HuggingFace hub. This card will be logged as part of the model artifact, viewable at the same directory level as the MLmodel file and the stored model object. In addition to the ModelCard, the components that comprise any Pipeline (or the individual components if saving a dictionary of named components) will have their source types stored. The model type, pipeline type, task, and classes of any supplementary component (such as a Tokenizer or ImageProcessor) will be stored in the MLmodel file as well.   Automatic Signature inference  For pipelines that support pyfunc, there are 3 means of attaching a model signature to the MLmodel file.  Provide a model signature explicitly via setting a valid ModelSignature to the signature attribute. This can be generated via the helper utility mlflow.transformers.generate_signature_output() Provide an input_example. The signature will be inferred and validated that it matches the appropriate input type. The output type will be validated by performing inference automatically (if the model is a pyfunc supported type). Do nothing. The transformers flavor will automatically apply the appropriate general signature that the pipeline type supports (only for a single-entity; collections will not be inferred).    Scalability for inference  A common configuration for lowering the total memory pressure for pytorch models within transformers pipelines is to modify the processing data type. This is achieved through setting the torch_dtype argument when creating a Pipeline. For a full reference of these tunable arguments for configuration of pipelines, see the training docs .  Note This feature does not exist in versions of transformers < 4.26.x  In order to apply these configurations to a saved or logged run, there are two options:  Save a pipeline with the torch_dtype argument set to the encoding type of your choice.  Example: import transformers import torch import mlflow  task = ""translation_en_to_fr""  my_pipeline = transformers.pipeline(     task=task,     model=transformers.T5ForConditionalGeneration.from_pretrained(""t5-small""),     tokenizer=transformers.T5TokenizerFast.from_pretrained(         ""t5-small"", model_max_length=100     ),     framework=""pt"",     torch_dtype=torch.bfloat16, )  with mlflow.start_run():     model_info = mlflow.transformers.log_model(         transformers_model=my_pipeline,         artifact_path=""my_pipeline"",     )  # Illustrate that the torch data type is recorded in the flavor configuration print(model_info.flavors[""transformers""])   Result: {'transformers_version': '4.28.1',  'code': None,  'task': 'translation_en_to_fr',  'instance_type': 'TranslationPipeline',  'source_model_name': 't5-small',  'pipeline_model_type': 'T5ForConditionalGeneration',  'framework': 'pt',  'torch_dtype': 'torch.bfloat16',  'tokenizer_type': 'T5TokenizerFast',  'components': ['tokenizer'],  'pipeline': 'pipeline'}    Specify the torch_dtype argument when loading the model to override any values set during logging or saving.  Example: import transformers import torch import mlflow  task = ""translation_en_to_fr""  my_pipeline = transformers.pipeline(     task=task,     model=transformers.T5ForConditionalGeneration.from_pretrained(""t5-small""),     tokenizer=transformers.T5TokenizerFast.from_pretrained(         ""t5-small"", model_max_length=100     ),     framework=""pt"",     torch_dtype=torch.bfloat16, )  with mlflow.start_run():     model_info = mlflow.transformers.log_model(         transformers_model=my_pipeline,         artifact_path=""my_pipeline"",     )  loaded_pipeline = mlflow.transformers.load_model(     model_info.model_uri, return_type=""pipeline"", torch_dtype=torch.float64 )  print(loaded_pipeline.torch_dtype)   Result: torch.float64    Note Logging or saving a model in 'components' mode (using a dictionary to declare components) does not support setting the data type for a constructed pipeline. If you need to override the default behavior of how data is encoded, please save or log a pipeline object.   Note Overriding the data type for a pipeline when loading as a python_function (pyfunc) model flavor is not supported. The value set for torch_dtype during save_model() or log_model() will persist when loading as pyfunc.    Input data types for audio pipelines  Note that passing raw data to an audio pipeline (raw bytes) requires two separate elements of the same effective library. In order to use the bitrate transposition and conversion of the audio bytes data into numpy nd.array format, the library ffmpeg is required. Installing this package directly from pypi (pip install ffmpeg) does not install the underlying c dll's that are required to make ffmpeg function. Please consult with the documentation at the ffmpeg website for guidance on your given operating system. The Audio Pipeline types, when loaded as a python_function (pyfunc) model flavor have three input types available:  str  The string input type is meant for blob references (uri locations) that are accessible to the instance of the pyfunc model. This input mode is useful when doing large batch processing of audio inference in Spark due to the inherent limitations of handling large bytes data in Spark DataFrames. Ensure that you have ffmpeg installed in the environment that the pyfunc model is running in order to use str input uri-based inference. If this package is not properly installed (both from pypi and from the ffmpeg binaries), an Exception will be thrown at inference time.  Warning If using a uri (str) as an input type for a pyfunc model that you are intending to host for realtime inference through the MLflow Model Server, you must specify a custom model signature when logging or saving the model. The default signature input value type of bytes will, in MLflow Model serving, force the conversion of the uri string to bytes, which will cause an Exception to be thrown from the serving process stating that the soundfile is corrupt.  An example of specifying an appropriate uri-based input model signature for an audio model is shown below: from mlflow.models import infer_signature from mlflow.transformers import generate_signature_output  url = ""https://www.mywebsite.com/sound/files/for/transcription/file111.mp3"" signature = infer_signature(url, generate_signature_output(my_audio_pipeline, url)) with mlflow.start_run():     mlflow.transformers.log_model(         transformers_model=my_audio_pipeline,         artifact_path=""my_transcriber"",         signature=signature,     )    bytes  This is the default serialization format of audio files. It is the easiest format to utilize due to the fact that Pipeline implementations will automatically convert the audio bitrate from the file with the use of ffmpeg (a required dependency if using this format) to the bitrate required by the underlying model within the Pipeline. When using the pyfunc representation of the pipeline directly (not through serving), the sound file can be passed directly as bytes without any modification. When used through serving, the bytes data must be base64 encoded.  np.ndarray  This input format requires that both the bitrate has been set prior to conversion to numpy.ndarray (i.e., through the use of a package like librosa or pydub) and that the model has been saved with a signature that uses the np.ndarray format for the input.  Note Audio models being used for serving that intend to utilize pre-formatted audio in np.ndarray format must have the model saved with a signature configuration that reflects this schema. Failure to do so will result in type casting errors due to the default signature for audio transformers pipelines being set as expecting binary (bytes) data. The serving endpoint cannot accept a union of types, so a particular model instance must choose one or the other as an allowed input type.      SentenceTransformers (sentence_transformers) (Experimental)   Attention The sentence_transformers flavor is in active development and is marked as Experimental. Public APIs may change and new features are subject to be added as additional functionality is brought to the flavor.  The sentence_transformers model flavor enables logging of sentence-transformers models in MLflow format via the mlflow.sentence_transformers.save_model() and mlflow.sentence_transformers.log_model() functions. Use of these functions also adds the python_function flavor to the MLflow Models that they produce, allowing the model to be interpreted as a generic Python function for inference via mlflow.pyfunc.load_model(). You can also use the mlflow.sentence_transformers.load_model() function to load a saved or logged MLflow Model with the sentence_transformers flavor as a native sentence-transformers model. Example: from sentence_transformers import SentenceTransformer  import mlflow import mlflow.sentence_transformers  model = SentenceTransformer(""all-MiniLM-L6-v2"")  example_sentences = [""This is a sentence."", ""This is another sentence.""]  # Define the signature signature = mlflow.models.infer_signature(     model_input=example_sentences,     model_output=model.encode(example_sentences), )  # Log the model using mlflow with mlflow.start_run():     logged_model = mlflow.sentence_transformers.log_model(         model=model,         artifact_path=""sbert_model"",         signature=signature,         input_example=example_sentences,     )  # Load option 1: mlflow.pyfunc.load_model returns a PyFuncModel loaded_model = mlflow.pyfunc.load_model(logged_model.model_uri) embeddings1 = loaded_model.predict([""hello world"", ""i am mlflow""])  # Load option 2: mlflow.sentence_transformers.load_model returns a SentenceTransformer loaded_model = mlflow.sentence_transformers.load_model(logged_model.model_uri) embeddings2 = loaded_model.encode([""hello world"", ""i am mlflow""])  print(embeddings1)  """""" >> [[-3.44772562e-02  3.10232025e-02  6.73496164e-03  2.61089969e-02   ...   2.37922110e-02 -2.28897743e-02  3.89375277e-02  3.02067865e-02]  [ 4.81191138e-03 -9.33756605e-02  6.95968643e-02  8.09735525e-03   ...    6.57437667e-02 -2.72239652e-02  4.02687863e-02 -1.05599344e-01]] """"""      Model Evaluation  After building and training your MLflow Model, you can use the mlflow.evaluate() API to evaluate its performance on one or more datasets of your choosing. mlflow.evaluate() currently supports evaluation of MLflow Models with the python_function (pyfunc) model flavor for classification, regression, and numerous language modeling tasks (see Evaluating with LLMs), computing a variety of task-specific performance metrics, model performance plots, and model explanations. Evaluation results are logged to MLflow Tracking. The following example from the MLflow GitHub Repository uses mlflow.evaluate() to evaluate the performance of a classifier on the UCI Adult Data Set, logging a comprehensive collection of MLflow Metrics and Artifacts that provide insight into model performance and behavior: import xgboost import shap import mlflow from mlflow.models import infer_signature from sklearn.model_selection import train_test_split  # Load the UCI Adult Dataset X, y = shap.datasets.adult()  # Split the data into training and test sets X_train, X_test, y_train, y_test = train_test_split(     X, y, test_size=0.33, random_state=42 )  # Fit an XGBoost binary classifier on the training data split model = xgboost.XGBClassifier().fit(X_train, y_train)  # Create a model signature signature = infer_signature(X_test, model.predict(X_test))  # Build the Evaluation Dataset from the test set eval_data = X_test eval_data[""label""] = y_test  with mlflow.start_run() as run:     # Log the baseline model to MLflow     mlflow.sklearn.log_model(model, ""model"", signature=signature)     model_uri = mlflow.get_artifact_uri(""model"")      # Evaluate the logged model     result = mlflow.evaluate(         model_uri,         eval_data,         targets=""label"",         model_type=""classifier"",         evaluators=[""default""],     )      Evaluating with LLMs  As of MLflow 2.4.0, mlflow.evaluate() has built-in support for a variety of tasks with LLMs, including text summarization, text classification, question answering, and text generation. The following example uses mlflow.evaluate() to evaluate a model that answers questions about MLflow (note that you must have the OPENAI_API_TOKEN environment variable set in your current system environment in order to run the example): import os import pandas as pd  import mlflow import openai  # Create a question answering model using prompt engineering with OpenAI. Log the # prompt and the model to MLflow Tracking mlflow.start_run() system_prompt = (     ""Your job is to answer questions about MLflow. When you are asked a question about MLflow,""     "" respond to it. Make sure to include code examples. If the question is not related to""     "" MLflow, refuse to answer and say that the question is unrelated."" ) mlflow.log_param(""system_prompt"", system_prompt) logged_model = mlflow.openai.log_model(     model=""gpt-3.5-turbo"",     task=openai.ChatCompletion,     artifact_path=""model"",     messages=[         {""role"": ""system"", ""content"": system_prompt},         {""role"": ""user"", ""content"": ""{question}""},     ], )  # Evaluate the model on some example questions questions = pd.DataFrame(     {         ""question"": [             ""How do you create a run with MLflow?"",             ""How do you log a model with MLflow?"",             ""What is the capital of France?"",         ]     } ) mlflow.evaluate(     model=logged_model.model_uri,     model_type=""question-answering"",     data=questions, )  # Load and inspect the evaluation results results: pd.DataFrame = mlflow.load_table(     ""eval_results_table.json"", extra_columns=[""run_id"", ""params.system_prompt""] ) print(""Evaluation results:"") print(results)   MLflow also provides an Artifact View UI for comparing inputs and outputs across multiple models built with LLMs. For example, after evaluating multiple prompts for question answering (see the MLflow OpenAI question answering full example), you can navigate to the Artifact View to view the questions and compare the answers for each model:      For additional examples demonstrating the use of mlflow.evaluate() with LLMs, check out the MLflow LLMs example repository.   Evaluating with Extra Metrics  If the default set of metrics is insufficient, you can supply extra_metrics and custom_artifacts to mlflow.evaluate() to produce extra metrics and artifacts for the model(s) that you're evaluating. To define an extra metric, you should define an eval_fn function that takes in predictions, targets, and metrics as inputs and outputs a MetricValue object. predictions and targets are pandas.Series objects. If predictions or targets specified in mlflow.evaluate() is either numpy.array or List, they will be converted to pandas.Series. metrics is a dictionary mapping a metric name string to a MetricValue object. It contains the values from built-in metrics and can be used to compute your custom metric. The built-in metrics are available when model_type is defined for mlflow.evaluate(... model_type=""classifier""). {     ""accuracy_score"": MetricValue(         scores=None, justifications=None, aggregate_results={""accuracy_score"": 1.0}     ) }   The MetricValue class has three attributes:  scores: a list that contains per-row metrics. aggregate_results: a dictionary that maps the aggregation method names to the corresponding aggregated values. This is intended to be used to aggregate scores. justification: a per-row justification of the values in scores. This is optional, and is usually used with genai metrics.  The code block below demonstrates how to define a custom metric evaluation function: from mlflow.metrics import MetricValue   def my_metric_eval_fn(predictions, targets, metrics):     scores = np.abs(predictions - targets)     return MetricValue(         scores=list(scores),         aggregate_results={             ""mean"": np.mean(scores),             ""variance"": np.var(scores),             ""median"": np.median(scores),         },     )   Once you have defined an eval_fn, you then use make_metric() to wrap this eval_fn function into a metric. In addition to eval_fn, make_metric() requires an additional parameter , greater_is_better, for optimization purposes. This parameter indicates whether this is a metric we want to maximize or minimize. from mlflow.metrics import make_metric  mymetric = make_metric(eval_fn=my_metric_eval_fn, greater_is_better=False)   The extra metric allows you to either evaluate a model directly, or to evaluate an output dataframe. To evaluate the model directly, you will have to provide mlflow.evaluate() either a pyfunc model instance, a URI referring to a pyfunc model, or a callable function that takes in the data as input and outputs the predictions. def model(x):     return x[""inputs""]   eval_dataset = pd.DataFrame(     {         ""targets"": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],         ""inputs"": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],     } )  mlflow.evaluate(model, eval_dataset, targets=""targets"", extra_metrics=[mymetric])    To directly evaluate an output dataframe, you can omit the model parameter. However, you will needto set the predictions parameter in mlflow.evaluate() in order to evaluate an inference output dataframe.   eval_dataset = pd.DataFrame(     {         ""targets"": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],         ""predictions"": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],     } )  result = mlflow.evaluate(     data=eval_dataset,     predictions=""predictions"",     targets=""targets"",     extra_metrics=[mymetric], )   When your model has multiple outputs, the model must return a pandas DataFrame with multiple columns. You must specify one column among the model output columns as the predictions column using the predictions parameter, and other output columns of the model will be accessible from the eval_fn based on their column names. For example, if your model has two outputs retrieved_context and answer, you can specify answer as the predictions column, and retrieved_context column will be accessible as the context parameter from eval_fn via col_mapping: def eval_fn(predictions, targets, metrics, context):     scores = (predictions == targets) + context     return MetricValue(         scores=list(scores),         aggregate_results={""mean"": np.mean(scores), ""sum"": np.sum(scores)},     )   mymetric = make_metric(eval_fn=eval_fn, greater_is_better=False, name=""mymetric"")   def model(x):     return pd.DataFrame({""retrieved_context"": x[""inputs""] + 1, ""answer"": x[""inputs""]})   eval_dataset = pd.DataFrame(     {         ""targets"": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],         ""inputs"": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],     } )  config = {""col_mapping"": {""context"": ""retrieved_context""}}  result = mlflow.evaluate(     model,     eval_dataset,     predictions=""answer"",     targets=""targets"",     extra_metrics=[mymetric],     evaluator_config=config, )   However, you can also avoid using col_mapping if the parameter of eval_fn is the same as the output column name of the model. def eval_fn(predictions, targets, metrics, retrieved_context):     scores = (predictions == targets) + retrieved_context     return MetricValue(         scores=list(scores),         aggregate_results={""mean"": np.mean(scores), ""sum"": np.sum(scores)},     )   mymetric = make_metric(eval_fn=eval_fn, greater_is_better=False, name=""mymetric"")   def model(x):     return pd.DataFrame({""retrieved_context"": x[""inputs""] + 1, ""answer"": x[""inputs""]})   eval_dataset = pd.DataFrame(     {         ""targets"": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],         ""inputs"": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],     } )  result = mlflow.evaluate(     model,     eval_dataset,     predictions=""answer"",     targets=""targets"",     extra_metrics=[mymetric], )   col_mapping also allows you to pass additional parameters to the extra metric function, in this case passing a value k. def eval_fn(predictions, targets, metrics, k):     scores = k * (predictions == targets)     return MetricValue(scores=list(scores), aggregate_results={""mean"": np.mean(scores)})   weighted_mymetric = make_metric(eval_fn=eval_fn, greater_is_better=False)   def model(x):     return x[""inputs""]   eval_dataset = pd.DataFrame(     {         ""targets"": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],         ""inputs"": [1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0],     } )  config = {""col_mapping"": {""k"": 5}} mlflow.evaluate(     model,     eval_dataset,     targets=""targets"",     extra_metrics=[weighted_mymetric],     evaluator_config=config, )   The following short example from the MLflow GitHub Repository uses mlflow.evaluate() with an extra metric function to evaluate the performance of a regressor on the California Housing Dataset. import os  import matplotlib.pyplot as plt import numpy as np from sklearn.datasets import fetch_california_housing from sklearn.linear_model import LinearRegression from sklearn.model_selection import train_test_split  import mlflow from mlflow.models import infer_signature, make_metric  # loading the California housing dataset cali_housing = fetch_california_housing(as_frame=True)  # split the dataset into train and test partitions X_train, X_test, y_train, y_test = train_test_split(     cali_housing.data, cali_housing.target, test_size=0.2, random_state=123 )  # train the model lin_reg = LinearRegression().fit(X_train, y_train)  # Infer model signature predictions = lin_reg.predict(X_train) signature = infer_signature(X_train, predictions)  # creating the evaluation dataframe eval_data = X_test.copy() eval_data[""target""] = y_test   def squared_diff_plus_one(eval_df, _builtin_metrics):     """"""     This example custom metric function creates a metric based on the ``prediction`` and     ``target`` columns in ``eval_df`.     """"""     return np.sum(np.abs(eval_df[""prediction""] - eval_df[""target""] + 1) ** 2)   def sum_on_target_divided_by_two(_eval_df, builtin_metrics):     """"""     This example custom metric function creates a metric derived from existing metrics in     ``builtin_metrics``.     """"""     return builtin_metrics[""sum_on_target""] / 2   def prediction_target_scatter(eval_df, _builtin_metrics, artifacts_dir):     """"""     This example custom artifact generates and saves a scatter plot to ``artifacts_dir`` that     visualizes the relationship between the predictions and targets for the given model to a     file as an image artifact.     """"""     plt.scatter(eval_df[""prediction""], eval_df[""target""])     plt.xlabel(""Targets"")     plt.ylabel(""Predictions"")     plt.title(""Targets vs. Predictions"")     plot_path = os.path.join(artifacts_dir, ""example_scatter_plot.png"")     plt.savefig(plot_path)     return {""example_scatter_plot_artifact"": plot_path}   with mlflow.start_run() as run:     mlflow.sklearn.log_model(lin_reg, ""model"", signature=signature)     model_uri = mlflow.get_artifact_uri(""model"")     result = mlflow.evaluate(         model=model_uri,         data=eval_data,         targets=""target"",         model_type=""regressor"",         evaluators=[""default""],         extra_metrics=[             make_metric(                 eval_fn=squared_diff_plus_one,                 greater_is_better=False,             ),             make_metric(                 eval_fn=sum_on_target_divided_by_two,                 greater_is_better=True,             ),         ],         custom_artifacts=[prediction_target_scatter],     )  print(f""metrics:\n{result.metrics}"") print(f""artifacts:\n{result.artifacts}"")   For a more comprehensive extra metrics usage example, refer to this example from the MLflow GitHub Repository.   Evaluating with a Function  As of MLflow 2.8.0, mlflow.evaluate() supports evaluating a python function without requiring logging the model to MLflow. This is useful when you don't want to log the model and just want to evaluate it. The requirements for the function's input and output are the same as the requirements for a model's input and output. The following example uses mlflow.evaluate() to evaluate a function: import shap import xgboost from sklearn.model_selection import train_test_split  import mlflow  # Load the UCI Adult Dataset X, y = shap.datasets.adult()  # Split the data into training and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)  # Fit an XGBoost binary classifier on the training data split model = xgboost.XGBClassifier().fit(X_train, y_train)  # Build the Evaluation Dataset from the test set eval_data = X_test eval_data[""label""] = y_test   # Define a function that calls the model's predict method def fn(X):     return model.predict(X)   with mlflow.start_run() as run:     # Evaluate the function without logging the model     result = mlflow.evaluate(         fn,         eval_data,         targets=""label"",         model_type=""classifier"",         evaluators=[""default""],     )  print(f""metrics:\n{result.metrics}"") print(f""artifacts:\n{result.artifacts}"")     Evaluating with a Static Dataset  As of MLflow 2.8.0, mlflow.evaluate() supports evaluating a static dataset without specifying a model. This is useful when you save the model output to a column in a Pandas DataFrame or an MLflow PandasDataset, and want to evaluate the static dataset without re-running the model. If you are using a Pandas DataFrame, you must specify the column name that contains the model output using the top-level predictions parameter in mlflow.evaluate(): # Assume that the model output is saved to the pandas_df[""model_output""] column mlflow.evaluate(data=pandas_df, predictions=""model_output"", ...)   If you are using an MLflow PandasDataset, you must specify the column name that contains the model output using the predictions parameter in mlflow.data.from_pandas(), and specify None for the predictions parameter in mlflow.evaluate(): # Assume that the model output is saved to the pandas_df[""model_output""] column dataset = mlflow.data.from_pandas(pandas_df, predictions=""model_output"") mlflow.evaluate(data=pandas_df, predictions=None, ...)   When your model has multiple outputs, you must specify one column among the model output columns as the predictions column. The other output columns of the model will be treated as "input" columns. For example, if your model has two outputs named retrieved_context and answer, you can specify answer as the predictions column. The retrieved_context column will be treated as an "input" column when calculating the metrics. The following example uses mlflow.evaluate() to evaluate a static dataset: import shap import xgboost from sklearn.model_selection import train_test_split  import mlflow  # Load the UCI Adult Dataset X, y = shap.datasets.adult()  # Split the data into training and test sets X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)  # Fit an XGBoost binary classifier on the training data split model = xgboost.XGBClassifier().fit(X_train, y_train)  # Build the Evaluation Dataset from the test set y_test_pred = model.predict(X=X_test) eval_data = X_test eval_data[""label""] = y_test eval_data[""predictions""] = y_test_pred   with mlflow.start_run() as run:     # Evaluate the static dataset without providing a model     result = mlflow.evaluate(         data=eval_data,         targets=""label"",         predictions=""predictions"",         model_type=""classifier"",     )  print(f""metrics:\n{result.metrics}"") print(f""artifacts:\n{result.artifacts}"")     Performing Model Validation  You can also use the mlflow.evaluate() API to perform some checks on the metrics generated during model evaluation to validate the quality of your model. By specifying a validation_thresholds dictionary mapping metric names to mlflow.models.MetricThreshold objects, you can specify value thresholds that your model's evaluation metrics must exceed as well as absolute and relative gains your model must have in comparison to a specified baseline_model. If your model fails to clear specified thresholds, mlflow.evaluate() will throw a ModelValidationFailedException detailing the validation failure. import xgboost import shap from sklearn.model_selection import train_test_split from sklearn.dummy import DummyClassifier import mlflow from mlflow.models import infer_signature from mlflow.models import MetricThreshold  # load UCI Adult Data Set; segment it into training and test sets X, y = shap.datasets.adult() X_train, X_test, y_train, y_test = train_test_split(     X, y, test_size=0.33, random_state=42 )  # train a candidate XGBoost model candidate_model = xgboost.XGBClassifier().fit(X_train, y_train)  # train a baseline dummy model baseline_model = DummyClassifier(strategy=""uniform"").fit(X_train, y_train)  # create signature that is shared by the two models signature = infer_signature(X_test, y_test)  # construct an evaluation dataset from the test set eval_data = X_test eval_data[""label""] = y_test  # Define criteria for model to be validated against thresholds = {     ""accuracy_score"": MetricThreshold(         threshold=0.8,  # accuracy should be >=0.8         min_absolute_change=0.05,  # accuracy should be at least 0.05 greater than baseline model accuracy         min_relative_change=0.05,  # accuracy should be at least 5 percent greater than baseline model accuracy         greater_is_better=True,     ), }  with mlflow.start_run() as run:     candidate_model_uri = mlflow.sklearn.log_model(         candidate_model, ""candidate_model"", signature=signature     ).model_uri     baseline_model_uri = mlflow.sklearn.log_model(         baseline_model, ""baseline_model"", signature=signature     ).model_uri      mlflow.evaluate(         candidate_model_uri,         eval_data,         targets=""label"",         model_type=""classifier"",         validation_thresholds=thresholds,         baseline_model=baseline_model_uri,     )   Refer to mlflow.models.MetricThreshold to see details on how the thresholds are specified and checked. For a more comprehensive demonstration on how to use mlflow.evaluate() to perform model validation, refer to the Model Validation example from the MLflow GitHub Repository. The logged output within the MLflow UI for the comprehensive example is shown below. Note the two model artifacts that have been logged: 'baseline_model' and 'candidate_model' for comparison purposes in the example.   Note Limitations (when the default evaluator is used):  Model validation results are not included in the active MLflow run. No metrics are logged nor artifacts produced for the baseline model in the active MLflow run.   Additional information about model evaluation behaviors and outputs is available in the mlflow.evaluate() API docs.  Note There are plugins that support in-depth model validation with features that are not supported directly in MLflow. To learn more, see:  Model Validation with Giskard's plugin Model Validation with Trubrics' plugin.    Note Differences in the computation of Area under Curve Precision Recall score (metric name precision_recall_auc) between multi and binary classifiers: Multiclass classifier models, when evaluated, utilize the standard scoring metric from sklearn: sklearn.metrics.roc_auc_score to calculate the area under the precision recall curve. This algorithm performs a linear interpolation calculation utilizing the trapezoidal rule to estimate the area under the precision recall curve. It is well-suited for use in evaluating multi-class classification models to provide a single numeric value of the quality of fit. Binary classifier models, on the other hand, use the sklearn.metrics.average_precision_score to avoid the shortcomings of the roc_auc_score implementation when applied to heavily imbalanced classes in binary classification. Usage of the roc_auc_score for imbalanced datasets can give a misleading result (optimistically better than the model's actual ability to accurately predict the minority class membership). For additional information on the topic of why different algorithms are employed for this, as well as links to the papers that informed the implementation of these metrics within the sklearn.metrics module, refer to the documentation. For simplicity purposes, both methodologies evaluation metric results (whether for multi-class or binary classification) are unified in the single metric: precision_recall_auc.    Model Validation with Giskard's plugin  To extend the validation capabilities of MLflow and anticipate issues before they go to production, a plugin has been built by Giskard allowing users to:   scan a model in order to detect hidden vulnerabilities such as Performance bias, Unrobustness, Overconfidence, Underconfidence, Ethical bias, Data leakage, Stochasticity, Spurious correlation, and others explore samples in the data that highlight the vulnerabilities found log the vulnerabilities as well-defined and quantified metrics compare the metrics across different models   See the following plugin example notebooks for a demo:   Tabular ML models Text ML models (LLMs)   For more information on the plugin, see the giskard-mlflow docs.   Model Validation with Trubrics' plugin  To extend the validation capabilities of MLflow, a plugin has been built by Trubrics allowing users:   to use a large number of out-of-the-box validations to validate a run with any custom python functions to view all validation results in a .json file, for diagnosis of why an MLflow run could have failed   See the plugin example notebook for a demo. For more information on the plugin, see the trubrics-mlflow docs.    Model Customization  While MLflow's built-in model persistence utilities are convenient for packaging models from various popular ML libraries in MLflow Model format, they do not cover every use case. For example, you may want to use a model from an ML library that is not explicitly supported by MLflow's built-in flavors. Alternatively, you may want to package custom inference code and data to create an MLflow Model. Fortunately, MLflow provides two solutions that can be used to accomplish these tasks: Custom Python Models and Custom Flavors.  In this section:  Custom Python Models  Example: Creating a custom "add n" model Example: Saving an XGBoost model in MLflow format Example: Logging a transformers model with hf:/ schema to avoid copying large files   Custom Flavors  Example: Creating a custom "sktime" flavor Example: Using the custom "sktime" flavor      Custom Python Models  The mlflow.pyfunc module provides save_model() and log_model() utilities for creating MLflow Models with the python_function flavor that contain user-specified code and artifact (file) dependencies. These artifact dependencies may include serialized models produced by any Python ML library. Because these custom models contain the python_function flavor, they can be deployed to any of MLflow's supported production environments, such as SageMaker, AzureML, or local REST endpoints. The following examples demonstrate how you can use the mlflow.pyfunc module to create custom Python models. For additional information about model customization with MLflow's python_function utilities, see the python_function custom models documentation.  Example: Creating a custom "add n" model  This example defines a class for a custom model that adds a specified numeric value, n, to all columns of a Pandas DataFrame input. Then, it uses the mlflow.pyfunc APIs to save an instance of this model with n = 5 in MLflow Model format. Finally, it loads the model in python_function format and uses it to evaluate a sample input. import mlflow.pyfunc   # Define the model class class AddN(mlflow.pyfunc.PythonModel):     def __init__(self, n):         self.n = n      def predict(self, context, model_input, params=None):         return model_input.apply(lambda column: column + self.n)   # Construct and save the model model_path = ""add_n_model"" add5_model = AddN(n=5) mlflow.pyfunc.save_model(path=model_path, python_model=add5_model)  # Load the model in `python_function` format loaded_model = mlflow.pyfunc.load_model(model_path)  # Evaluate the model import pandas as pd  model_input = pd.DataFrame([range(10)]) model_output = loaded_model.predict(model_input) assert model_output.equals(pd.DataFrame([range(5, 15)]))     Example: Saving an XGBoost model in MLflow format  This example begins by training and saving a gradient boosted tree model using the XGBoost library. Next, it defines a wrapper class around the XGBoost model that conforms to MLflow's python_function inference API. Then, it uses the wrapper class and the saved XGBoost model to construct an MLflow Model that performs inference using the gradient boosted tree. Finally, it loads the MLflow Model in python_function format and uses it to evaluate test data. # Load training and test datasets from sys import version_info import xgboost as xgb from sklearn import datasets from sklearn.model_selection import train_test_split  PYTHON_VERSION = f""{version_info.major}.{version_info.minor}.{version_info.micro}"" iris = datasets.load_iris() x = iris.data[:, 2:] y = iris.target x_train, x_test, y_train, _ = train_test_split(x, y, test_size=0.2, random_state=42) dtrain = xgb.DMatrix(x_train, label=y_train)  # Train and save an XGBoost model xgb_model = xgb.train(params={""max_depth"": 10}, dtrain=dtrain, num_boost_round=10) xgb_model_path = ""xgb_model.pth"" xgb_model.save_model(xgb_model_path)  # Create an `artifacts` dictionary that assigns a unique name to the saved XGBoost model file. # This dictionary will be passed to `mlflow.pyfunc.save_model`, which will copy the model file # into the new MLflow Model's directory. artifacts = {""xgb_model"": xgb_model_path}  # Define the model class import mlflow.pyfunc   class XGBWrapper(mlflow.pyfunc.PythonModel):     def load_context(self, context):         import xgboost as xgb          self.xgb_model = xgb.Booster()         self.xgb_model.load_model(context.artifacts[""xgb_model""])      def predict(self, context, model_input, params=None):         input_matrix = xgb.DMatrix(model_input.values)         return self.xgb_model.predict(input_matrix)   # Create a Conda environment for the new MLflow Model that contains all necessary dependencies. import cloudpickle  conda_env = {     ""channels"": [""defaults""],     ""dependencies"": [         f""python={PYTHON_VERSION}"",         ""pip"",         {             ""pip"": [                 f""mlflow=={mlflow.__version__}"",                 f""xgboost=={xgb.__version__}"",                 f""cloudpickle=={cloudpickle.__version__}"",             ],         },     ],     ""name"": ""xgb_env"", }  # Save the MLflow Model mlflow_pyfunc_model_path = ""xgb_mlflow_pyfunc"" mlflow.pyfunc.save_model(     path=mlflow_pyfunc_model_path,     python_model=XGBWrapper(),     artifacts=artifacts,     conda_env=conda_env, )  # Load the model in `python_function` format loaded_model = mlflow.pyfunc.load_model(mlflow_pyfunc_model_path)  # Evaluate the model import pandas as pd  test_predictions = loaded_model.predict(pd.DataFrame(x_test)) print(test_predictions)     Example: Logging a transformers model with hf:/ schema to avoid copying large files  This example shows how to use a special schema hf:/ to log a transformers model from huggingface hub directly. This is useful when the model is too large and especially when you want to serve the model directly, but it doesn't save extra space if you want to download and test the model locally. import mlflow from mlflow.models import infer_signature import numpy as np import transformers   # Define a custom PythonModel class QAModel(mlflow.pyfunc.PythonModel):     def load_context(self, context):         """"""         This method initializes the tokenizer and language model         using the specified snapshot location from model context.         """"""         snapshot_location = context.artifacts[""bert-tiny-model""]         # Initialize tokenizer and language model         tokenizer = transformers.AutoTokenizer.from_pretrained(snapshot_location)         model = transformers.BertForQuestionAnswering.from_pretrained(snapshot_location)         self.pipeline = transformers.pipeline(             task=""question-answering"", model=model, tokenizer=tokenizer         )      def predict(self, context, model_input, params=None):         question = model_input[""question""][0]         if isinstance(question, np.ndarray):             question = question.item()         ctx = model_input[""context""][0]         if isinstance(ctx, np.ndarray):             ctx = ctx.item()         return self.pipeline(question=question, context=ctx)   # Log the model data = {""question"": ""Who's house?"", ""context"": ""The house is owned by Run.""} pyfunc_artifact_path = ""question_answering_model"" with mlflow.start_run() as run:     model_info = mlflow.pyfunc.log_model(         artifact_path=pyfunc_artifact_path,         python_model=QAModel(),         artifacts={""bert-tiny-model"": ""hf:/prajjwal1/bert-tiny""},         input_example=data,         signature=infer_signature(data, [""Run""]),         extra_pip_requirements=[""torch"", ""accelerate"", ""transformers"", ""numpy""],     )      Custom Flavors  You can also create custom MLflow Models by writing a custom flavor. As discussed in the Model API and Storage Format sections, an MLflow Model is defined by a directory of files that contains an MLmodel configuration file. This MLmodel file describes various model attributes, including the flavors in which the model can be interpreted. The MLmodel file contains an entry for each flavor name; each entry is a YAML-formatted collection of flavor-specific attributes. To create a new flavor to support a custom model, you define the set of flavor-specific attributes to include in the MLmodel configuration file, as well as the code that can interpret the contents of the model directory and the flavor's attributes. A detailed example of constructing a custom model flavor and its usage is shown below. New custom flavors not considered for official inclusion into MLflow should be introduced as separate GitHub repositories with documentation provided in the Community Model Flavors page.  Example: Creating a custom "sktime" flavor  This example illustrates the creation of a custom flavor for sktime time series library. The library provides a unified interface for multiple learning  tasks including time series forecasting. While the custom flavor in this example is specific in terms of the sktime inference API and model serialization format, its interface design is similar to many of the existing built-in flavors. Particularly, the interface for utilizing the custom model loaded as a python_function flavor for generating predictions uses a single-row Pandas DataFrame configuration argument to expose the paramters of the sktime inference API. The complete code for this example is included in the flavor.py module of the sktime example directory. Let's examine the custom flavor module in more detail. The first step is to import several modules inluding sktime library, various MLflow utilities as well as the MLflow pyfunc module which is required to add the pyfunc specification to the MLflow model configuration. Note also the import of the flavor module itself. This will be passed to the mlflow.models.Model.log() method to log the model as an artifact to the current MLflow run. import logging import os import pickle  import flavor import mlflow import numpy as np import pandas as pd import sktime import yaml from mlflow import pyfunc from mlflow.exceptions import MlflowException from mlflow.models import Model from mlflow.models.model import MLMODEL_FILE_NAME from mlflow.models.utils import _save_example from mlflow.protos.databricks_pb2 import INTERNAL_ERROR, INVALID_PARAMETER_VALUE from mlflow.tracking._model_registry import DEFAULT_AWAIT_MAX_SLEEP_SECONDS from mlflow.tracking.artifact_utils import _download_artifact_from_uri from mlflow.utils.environment import (     _CONDA_ENV_FILE_NAME,     _CONSTRAINTS_FILE_NAME,     _PYTHON_ENV_FILE_NAME,     _REQUIREMENTS_FILE_NAME,     _mlflow_conda_env,     _process_conda_env,     _process_pip_requirements,     _PythonEnv,     _validate_env_arguments, ) from mlflow.utils.file_utils import write_to from mlflow.utils.model_utils import (     _add_code_from_conf_to_system_path,     _get_flavor_configuration,     _validate_and_copy_code_paths,     _validate_and_prepare_target_save_path, ) from mlflow.utils.requirements_utils import _get_pinned_requirement from sktime.utils.multiindex import flatten_multiindex  _logger = logging.getLogger(__name__)   We continue by defining a set of important variables used throughout the code that follows. The flavor name needs to be provided for every custom flavor and should reflect the name of the library to be supported. It is saved as part of the flavor-specific attributes to the MLmodel configuration file. This example also defines some sktime specific variables. For illustration purposes, only a subset of the available predict methods to be exposed via the _SktimeModelWrapper class is included when loading the model in its python_function flavor (additional methods could be added in a similar fashion). Additionaly, the model serialization formats, namely pickle (default) and cloudpickle, are defined. Note that both serialization modules require using the same python environment (version) in whatever environment this model is used for inference to ensure that the model will load with the appropriate version of pickle (cloudpickle). FLAVOR_NAME = ""sktime""  SKTIME_PREDICT = ""predict"" SKTIME_PREDICT_INTERVAL = ""predict_interval"" SKTIME_PREDICT_QUANTILES = ""predict_quantiles"" SKTIME_PREDICT_VAR = ""predict_var"" SUPPORTED_SKTIME_PREDICT_METHODS = [     SKTIME_PREDICT,     SKTIME_PREDICT_INTERVAL,     SKTIME_PREDICT_QUANTILES,     SKTIME_PREDICT_VAR, ]  SERIALIZATION_FORMAT_PICKLE = ""pickle"" SERIALIZATION_FORMAT_CLOUDPICKLE = ""cloudpickle"" SUPPORTED_SERIALIZATION_FORMATS = [     SERIALIZATION_FORMAT_PICKLE,     SERIALIZATION_FORMAT_CLOUDPICKLE, ]   Similar to the MLflow built-in flavors, a custom flavor logs the model in MLflow format via the save_model() and log_model() functions. In the save_model() function, the sktime model is saved to a specified output directory. Additionally, save_model() leverages the mlflow.models.Model.add_flavor() and mlflow.models.Model.save() methods to produce the MLmodel configuration containing the sktime and the python_function flavor. The resulting configuration has several flavor-specific attributes, such as the flavor name and sktime_version, which denotes the version of the sktime library that was used to train the model. An example of the output directoy for the custom sktime model is shown below: # Directory written by flavor.save_model(model, ""my_model"") my_model/ ├── MLmodel ├── conda.yaml ├── model.pkl ├── python_env.yaml └── requirements.txt   And its YAML-formatted MLmodel file describes the two flavors: flavors:   python_function:     env:       conda: conda.yaml       virtualenv: python_env.yaml     loader_module: flavor     model_path: model.pkl     python_version: 3.8.15   sktime:     code: null     pickled_model: model.pkl     serialization_format: pickle     sktime_version: 0.16.0   The save_model() function also provides flexibility to add additional paramters which can be added as flavor-specific attributes to the model configuration. In this example there is only one flavor-specific parameter for specifying the model serialization format. All other paramters are non-flavor specific (for a detailed description of these parameters take a look at mlflow.sklearn.save_model). Note: When creating your own custom flavor, be sure rename the sktime_model parameter in both the save_model() and log_model() functions to reflect the name of your custom model flavor. def save_model(     sktime_model,     path,     conda_env=None,     code_paths=None,     mlflow_model=None,     signature=None,     input_example=None,     pip_requirements=None,     extra_pip_requirements=None,     serialization_format=SERIALIZATION_FORMAT_PICKLE, ):     _validate_env_arguments(conda_env, pip_requirements, extra_pip_requirements)      if serialization_format not in SUPPORTED_SERIALIZATION_FORMATS:         raise MlflowException(             message=(                 f""Unrecognized serialization format: {serialization_format}. ""                 ""Please specify one of the following supported formats: ""                 ""{SUPPORTED_SERIALIZATION_FORMATS}.""             ),             error_code=INVALID_PARAMETER_VALUE,         )      _validate_and_prepare_target_save_path(path)     code_dir_subpath = _validate_and_copy_code_paths(code_paths, path)      if mlflow_model is None:         mlflow_model = Model()     if signature is not None:         mlflow_model.signature = signature     if input_example is not None:         _save_example(mlflow_model, input_example, path)      model_data_subpath = ""model.pkl""     model_data_path = os.path.join(path, model_data_subpath)     _save_model(         sktime_model, model_data_path, serialization_format=serialization_format     )      pyfunc.add_to_model(         mlflow_model,         loader_module=""flavor"",         model_path=model_data_subpath,         conda_env=_CONDA_ENV_FILE_NAME,         python_env=_PYTHON_ENV_FILE_NAME,         code=code_dir_subpath,     )      mlflow_model.add_flavor(         FLAVOR_NAME,         pickled_model=model_data_subpath,         sktime_version=sktime.__version__,         serialization_format=serialization_format,         code=code_dir_subpath,     )     mlflow_model.save(os.path.join(path, MLMODEL_FILE_NAME))      if conda_env is None:         if pip_requirements is None:             include_cloudpickle = (                 serialization_format == SERIALIZATION_FORMAT_CLOUDPICKLE             )             default_reqs = get_default_pip_requirements(include_cloudpickle)             inferred_reqs = mlflow.models.infer_pip_requirements(                 path, FLAVOR_NAME, fallback=default_reqs             )             default_reqs = sorted(set(inferred_reqs).union(default_reqs))         else:             default_reqs = None         conda_env, pip_requirements, pip_constraints = _process_pip_requirements(             default_reqs, pip_requirements, extra_pip_requirements         )     else:         conda_env, pip_requirements, pip_constraints = _process_conda_env(conda_env)      with open(os.path.join(path, _CONDA_ENV_FILE_NAME), ""w"") as f:         yaml.safe_dump(conda_env, stream=f, default_flow_style=False)      if pip_constraints:         write_to(os.path.join(path, _CONSTRAINTS_FILE_NAME), ""\n"".join(pip_constraints))      write_to(os.path.join(path, _REQUIREMENTS_FILE_NAME), ""\n"".join(pip_requirements))      _PythonEnv.current().to_yaml(os.path.join(path, _PYTHON_ENV_FILE_NAME))   def _save_model(model, path, serialization_format):     with open(path, ""wb"") as out:         if serialization_format == SERIALIZATION_FORMAT_PICKLE:             pickle.dump(model, out)         else:             import cloudpickle              cloudpickle.dump(model, out)   The save_model() function also writes the model dependencies to a requirements.txt and conda.yaml file in the model output directory. For this purpose the set of pip dependecies produced by this flavor need to be added to the get_default_pip_requirements() function. In this example only the minimum required dependencies are provided. In practice, additional requirements needed for preprocessing or post-processing steps could be included. Note that for any custom flavor, the mlflow.models.infer_pip_requirements() method in the save_model() function will return the default requirements defined in get_default_pip_requirements() as package imports are only inferred for built-in flavors. def get_default_pip_requirements(include_cloudpickle=False):     pip_deps = [_get_pinned_requirement(""sktime"")]     if include_cloudpickle:         pip_deps += [_get_pinned_requirement(""cloudpickle"")]      return pip_deps   def get_default_conda_env(include_cloudpickle=False):     return _mlflow_conda_env(         additional_pip_deps=get_default_pip_requirements(include_cloudpickle)     )   Next, we add the log_model() function. This function is little more than a wrapper around the mlflow.models.Model.log() method to enable logging our custom model as an artifact to the curren MLflow run. Any flavor-specific parameters (e.g. serialization_format) introduced in the save_model() function also need to be added in the log_model() function. We also need to pass the flavor module to the mlflow.models.Model.log() method which internally calls the save_model() function from above to persist the model. def log_model(     sktime_model,     artifact_path,     conda_env=None,     code_paths=None,     registered_model_name=None,     signature=None,     input_example=None,     await_registration_for=DEFAULT_AWAIT_MAX_SLEEP_SECONDS,     pip_requirements=None,     extra_pip_requirements=None,     serialization_format=SERIALIZATION_FORMAT_PICKLE,     **kwargs, ):     return Model.log(         artifact_path=artifact_path,         flavor=flavor,         registered_model_name=registered_model_name,         sktime_model=sktime_model,         conda_env=conda_env,         code_paths=code_paths,         signature=signature,         input_example=input_example,         await_registration_for=await_registration_for,         pip_requirements=pip_requirements,         extra_pip_requirements=extra_pip_requirements,         serialization_format=serialization_format,         **kwargs,     )   To interpret model directories produced by save_model(), the custom flavor must also define a load_model() function. The load_model() function reads the MLmodel configuration from the specified model directory and uses the configuration attributes to load and return the sktime model from its serialized representation. def load_model(model_uri, dst_path=None):     local_model_path = _download_artifact_from_uri(         artifact_uri=model_uri, output_path=dst_path     )     flavor_conf = _get_flavor_configuration(         model_path=local_model_path, flavor_name=FLAVOR_NAME     )     _add_code_from_conf_to_system_path(local_model_path, flavor_conf)     sktime_model_file_path = os.path.join(         local_model_path, flavor_conf[""pickled_model""]     )     serialization_format = flavor_conf.get(         ""serialization_format"", SERIALIZATION_FORMAT_PICKLE     )     return _load_model(         path=sktime_model_file_path, serialization_format=serialization_format     )   def _load_model(path, serialization_format):     with open(path, ""rb"") as pickled_model:         if serialization_format == SERIALIZATION_FORMAT_PICKLE:             return pickle.load(pickled_model)         elif serialization_format == SERIALIZATION_FORMAT_CLOUDPICKLE:             import cloudpickle              return cloudpickle.load(pickled_model)   The _load_pyfunc() function will be called by the mlflow.pyfunc.load_model() method to load the custom model flavor as a pyfunc type. The MLmodel flavor configuration is used to pass any flavor-specific attributes to the _load_model() function (i.e., the path to the python_function flavor in the model directory and the model serialization format). def _load_pyfunc(path):     try:         sktime_flavor_conf = _get_flavor_configuration(             model_path=path, flavor_name=FLAVOR_NAME         )         serialization_format = sktime_flavor_conf.get(             ""serialization_format"", SERIALIZATION_FORMAT_PICKLE         )     except MlflowException:         _logger.warning(             ""Could not find sktime flavor configuration during model ""             ""loading process. Assuming 'pickle' serialization format.""         )         serialization_format = SERIALIZATION_FORMAT_PICKLE      pyfunc_flavor_conf = _get_flavor_configuration(         model_path=path, flavor_name=pyfunc.FLAVOR_NAME     )     path = os.path.join(path, pyfunc_flavor_conf[""model_path""])      return _SktimeModelWrapper(         _load_model(path, serialization_format=serialization_format)     )   The final step is to create the model wrapper class defining the python_function flavor. The design of the wrapper class determines how the flavor's inference API is exposed when making predictions using the python_function flavor. Just like the built-in flavors, the predict() method of the sktime wrapper class accepts a single-row Pandas DataFrame configuration argument. For an example of how to construct this configuration DataFrame refer to the usage example in the next section. A detailed description of the supported paramaters and input formats is provided in the flavor.py module docstrings. class _SktimeModelWrapper:     def __init__(self, sktime_model):         self.sktime_model = sktime_model      def predict(self, dataframe, params=None) -> pd.DataFrame:         df_schema = dataframe.columns.values.tolist()          if len(dataframe) > 1:             raise MlflowException(                 f""The provided prediction pd.DataFrame contains {len(dataframe)} rows. ""                 ""Only 1 row should be supplied."",                 error_code=INVALID_PARAMETER_VALUE,             )          # Convert the configuration dataframe into a dictionary to simplify the         # extraction of parameters passed to the sktime predcition methods.         attrs = dataframe.to_dict(orient=""index"").get(0)         predict_method = attrs.get(""predict_method"")          if not predict_method:             raise MlflowException(                 f""The provided prediction configuration pd.DataFrame columns ({df_schema}) do not ""                 ""contain the required column `predict_method` for specifying the prediction method."",                 error_code=INVALID_PARAMETER_VALUE,             )          if predict_method not in SUPPORTED_SKTIME_PREDICT_METHODS:             raise MlflowException(                 ""Invalid `predict_method` value.""                 f""The supported prediction methods are {SUPPORTED_SKTIME_PREDICT_METHODS}"",                 error_code=INVALID_PARAMETER_VALUE,             )          # For inference parameters 'fh', 'X', 'coverage', 'alpha', and 'cov'         # the respective sktime default value is used if the value was not         # provided in the configuration dataframe.         fh = attrs.get(""fh"", None)          # Any model that is trained with exogenous regressor elements will need         # to provide `X` entries as a numpy ndarray to the predict method.         X = attrs.get(""X"", None)          # When the model is served via REST API the exogenous regressor must be         # provided as a list to the configuration DataFrame to be JSON serializable.         # Below we convert the list back to ndarray type as required by sktime         # predict methods.         if isinstance(X, list):             X = np.array(X)          # For illustration purposes only a subset of the available sktime prediction         # methods is exposed. Additional methods (e.g. predict_proba) could be added         # in a similar fashion.         if predict_method == SKTIME_PREDICT:             predictions = self.sktime_model.predict(fh=fh, X=X)          if predict_method == SKTIME_PREDICT_INTERVAL:             coverage = attrs.get(""coverage"", 0.9)             predictions = self.sktime_model.predict_interval(                 fh=fh, X=X, coverage=coverage             )          if predict_method == SKTIME_PREDICT_QUANTILES:             alpha = attrs.get(""alpha"", None)             predictions = self.sktime_model.predict_quantiles(fh=fh, X=X, alpha=alpha)          if predict_method == SKTIME_PREDICT_VAR:             cov = attrs.get(""cov"", False)             predictions = self.sktime_model.predict_var(fh=fh, X=X, cov=cov)          # Methods predict_interval() and predict_quantiles() return a pandas         # MultiIndex column structure. As MLflow signature inference does not         # support MultiIndex column structure the columns must be flattened.         if predict_method in [SKTIME_PREDICT_INTERVAL, SKTIME_PREDICT_QUANTILES]:             predictions.columns = flatten_multiindex(predictions)          return predictions     Example: Using the custom "sktime" flavor  This example trains a sktime NaiveForecaster model using the Longley dataset for forecasting with exogenous variables. It shows a custom model type implementation that logs the training hyper-parameters, evaluation metrics and the trained model as an artifact. The single-row configuration DataFrame for this example defines an interval forecast with nominal coverage values [0.9,0.95], a future forecast horizon of four periods, and an exogenous regressor. import json  import flavor import pandas as pd from sktime.datasets import load_longley from sktime.forecasting.model_selection import temporal_train_test_split from sktime.forecasting.naive import NaiveForecaster from sktime.performance_metrics.forecasting import (     mean_absolute_error,     mean_absolute_percentage_error, )  import mlflow  ARTIFACT_PATH = ""model""  with mlflow.start_run() as run:     y, X = load_longley()     y_train, y_test, X_train, X_test = temporal_train_test_split(y, X)      forecaster = NaiveForecaster()     forecaster.fit(         y_train,         X=X_train,     )      # Extract parameters     parameters = forecaster.get_params()      # Evaluate model     y_pred = forecaster.predict(fh=[1, 2, 3, 4], X=X_test)     metrics = {         ""mae"": mean_absolute_error(y_test, y_pred),         ""mape"": mean_absolute_percentage_error(y_test, y_pred),     }      print(f""Parameters: \n{json.dumps(parameters, indent=2)}"")     print(f""Metrics: \n{json.dumps(metrics, indent=2)}"")      # Log parameters and metrics     mlflow.log_params(parameters)     mlflow.log_metrics(metrics)      # Log model using custom model flavor with pickle serialization (default).     flavor.log_model(         sktime_model=forecaster,         artifact_path=ARTIFACT_PATH,         serialization_format=""pickle"",     )     model_uri = mlflow.get_artifact_uri(ARTIFACT_PATH)  # Load model in native sktime flavor and pyfunc flavor loaded_model = flavor.load_model(model_uri=model_uri) loaded_pyfunc = flavor.pyfunc.load_model(model_uri=model_uri)  # Convert test data to 2D numpy array so it can be passed to pyfunc predict using # a single-row Pandas DataFrame configuration argument X_test_array = X_test.to_numpy()  # Create configuration DataFrame predict_conf = pd.DataFrame(     [         {             ""fh"": [1, 2, 3, 4],             ""predict_method"": ""predict_interval"",             ""coverage"": [0.9, 0.95],             ""X"": X_test_array,         }     ] )  # Generate interval forecasts with native sktime flavor and pyfunc flavor print(     f""\nNative sktime 'predict_interval':\n${loaded_model.predict_interval(fh=[1, 2, 3], X=X_test, coverage=[0.9, 0.95])}"" ) print(f""\nPyfunc 'predict_interval':\n${loaded_pyfunc.predict(predict_conf)}"")  # Print the run id wich is used for serving the model to a local REST API endpoint print(f""\nMLflow run id:\n{run.info.run_id}"")   When opening the MLflow runs detail page the serialized model artifact  will show up, such as:      To serve the model to a local REST API endpoint run the following MLflow CLI command substituting the run id printed during execution of the previous block (for more details refer to the Deploy MLflow models section): mlflow models serve -m runs:/<run_id>/model --env-manager local --host 127.0.0.1   An example of requesting a prediction from the served model is shown below. The exogenous regressor needs to be provided as a list to be JSON serializable. The wrapper instance will convert the list back to numpy ndarray type as required by sktime inference API. import pandas as pd import requests  from sktime.datasets import load_longley from sktime.forecasting.model_selection import temporal_train_test_split  y, X = load_longley() y_train, y_test, X_train, X_test = temporal_train_test_split(y, X)  # Define local host and endpoint url host = ""127.0.0.1"" url = f""http://{host}:5000/invocations""  # Create configuration DataFrame X_test_list = X_test.to_numpy().tolist() predict_conf = pd.DataFrame(     [         {             ""fh"": [1, 2, 3, 4],             ""predict_method"": ""predict_interval"",             ""coverage"": [0.9, 0.95],             ""X"": X_test_list,         }     ] )  # Create dictionary with pandas DataFrame in the split orientation json_data = {""dataframe_split"": predict_conf.to_dict(orient=""split"")}  # Score model response = requests.post(url, json=json_data) print(f""\nPyfunc 'predict_interval':\n${response.json()}"")       Built-In Deployment Tools  MLflow provides tools for deploying MLflow models on a local machine and to several production environments. Not all deployment methods are available for all model flavors.  In this section:  Deploy MLflow models Deploy a python_function model on Microsoft Azure ML Deploy a python_function model on Amazon SageMaker Export a python_function model as an Apache Spark UDF    Deploy MLflow models  MLflow can deploy models locally as local REST API endpoints or to directly score files. In addition, MLflow can package models as self-contained Docker images with the REST API endpoint. The image can be used to safely deploy the model to various environments such as Kubernetes. You deploy MLflow model locally or generate a Docker image using the CLI interface to the mlflow.models module. The REST API defines 4 endpoints:  /ping used for health check /health (same as /ping) /version used for getting the mlflow version /invocations used for scoring  The REST API server accepts csv or json input. The input format must be specified in Content-Type header. The value of the header must be either application/json or application/csv. The csv input must be a valid pandas.DataFrame csv representation. For example, data = pandas_df.to_csv(). The json input must be a dictionary with exactly one of the following fields that further specify the type and encoding of the input data  dataframe_split field with pandas DataFrames in the split orientation. For example, data = {""dataframe_split"": pandas_df.to_dict(orient='split'). dataframe_records field with pandas DataFrame in the records orientation. For example, data = {""dataframe_records"": pandas_df.to_dict(orient='records').*We do not recommend using this format because it is not guaranteed to preserve column ordering.* instances field with tensor input formatted as described in TF Serving's API docs where the provided inputs will be cast to Numpy arrays. inputs field with tensor input formatted as described in TF Serving's API docs where the provided inputs will be cast to Numpy arrays.  The json input also has an optional field params that can be used to pass additional parameters. Valid parameters types are Union[DataType, List[DataType], None] where DataType is MLflow data types. In order to pass params, a valid Model Signature with params must be defined.  Note Since JSON loses type information, MLflow will cast the JSON input to the input type specified in the model's schema if available. If your model is sensitive to input types, it is recommended that a schema is provided for the model to ensure that type mismatch errors do not occur at inference time. In particular, DL models are typically strict about input types and will need model schema in order for the model to score correctly. For complex data types, see Encoding complex data below.  Example requests: # split-oriented DataFrame input curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '{   ""dataframe_split"": {       ""columns"": [""a"", ""b"", ""c""],       ""data"": [[1, 2, 3], [4, 5, 6]]   } }'  # record-oriented DataFrame input (fine for vector rows, loses ordering for JSON records) curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '{   ""dataframe_records"": [     {""a"": 1,""b"": 2,""c"": 3},     {""a"": 4,""b"": 5,""c"": 6}   ] }'  # numpy/tensor input using TF serving's ""instances"" format curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '{     ""instances"": [         {""a"": ""s1"", ""b"": 1, ""c"": [1, 2, 3]},         {""a"": ""s2"", ""b"": 2, ""c"": [4, 5, 6]},         {""a"": ""s3"", ""b"": 3, ""c"": [7, 8, 9]}     ] }'  # numpy/tensor input using TF serving's ""inputs"" format curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '{     ""inputs"": {""a"": [""s1"", ""s2"", ""s3""], ""b"": [1, 2, 3], ""c"": [[1, 2, 3], [4, 5, 6], [7, 8, 9]]} }'  # inference with params curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '{     ""inputs"": {""question"": [""What color is it?""],                ""context"": [""Some people said it was green but I know that it is pink.""]},     ""params"": {""max_answer_len"": 10} }'   For more information about serializing pandas DataFrames, see pandas.DataFrame.to_json. For more information about serializing tensor inputs using the TF serving format, see TF serving's request format docs.  Serving with MLServer  Python models can be deployed using Seldon's MLServer as alternative inference server. MLServer is integrated with two leading open source model deployment tools, Seldon Core and KServe (formerly known as KFServing), and can be used to test and deploy models using these frameworks. This is especially powerful when building docker images since the docker image built with MLServer can be deployed directly with both of these frameworks. MLServer exposes the same scoring API through the /invocations endpoint. In addition, it supports the standard V2 Inference Protocol.  Note To use MLServer with MLflow, please install mlflow as: pip install mlflow[extras]    To serve a MLflow model using MLServer, you can use the --enable-mlserver flag, such as: mlflow models serve -m my_model --enable-mlserver   Similarly, to build a Docker image built with MLServer you can use the --enable-mlserver flag, such as: mlflow models build-docker -m my_model --enable-mlserver -n my-model   To read more about the integration between MLflow and MLServer, please check the end-to-end example in the MLServer documentation or visit the MLServer docs.   Encoding complex data  Complex data types, such as dates or binary, do not have a native JSON representation. If you include a model signature, MLflow can automatically decode supported data types from JSON. The following data type conversions are supported:  binary: data is expected to be base64 encoded, MLflow will automatically base64 decode. datetime: data is expected as string according to ISO 8601 specification. MLflow will parse this into the appropriate datetime representation on the given platform.  Example requests: # record-oriented DataFrame input with binary column ""b"" curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '[     {""a"": 0, ""b"": ""dGVzdCBiaW5hcnkgZGF0YSAw""},     {""a"": 1, ""b"": ""dGVzdCBiaW5hcnkgZGF0YSAx""},     {""a"": 2, ""b"": ""dGVzdCBiaW5hcnkgZGF0YSAy""} ]'  # record-oriented DataFrame input with datetime column ""b"" curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '[     {""a"": 0, ""b"": ""2020-01-01T00:00:00Z""},     {""a"": 1, ""b"": ""2020-02-01T12:34:56Z""},     {""a"": 2, ""b"": ""2021-03-01T00:00:00Z""} ]'     Command Line Interface  MLflow also has a CLI that supports the following commands:  serve deploys the model as a local REST API server. build_docker packages a REST API endpoint serving the model as a docker image. predict uses the model to generate a prediction for a local CSV or JSON file. Note that this method only supports DataFrame input.  For more info, see: mlflow models --help mlflow models serve --help mlflow models predict --help mlflow models build-docker --help     Environment Management Tools  MLflow currently supports the following environment management tools to restore model environments:  localUse the local environment. No extra tools are required.  virtualenv (preferred)Create environments using virtualenv and pyenv (for python version management). Virtualenv and pyenv (for Linux and macOS) or pyenv-win (for Windows) must be installed for this mode of environment reconstruction.  virtualenv installation instructions pyenv installation instructions pyenv-win installation instructions   condaCreate environments using conda. Conda must be installed for this mode of environment reconstruction.  Warning By using conda, you're responsible for adhering to Anaconda's terms of service.   conda installation instructions    The mlflow models CLI commands provide an optional --env-manager argument that selects a specific environment management configuration to be used, as shown below: # Use virtualenv mlflow models predict ... --env-manager=virtualenv # Use conda mlflow models serve ... --env-manager=conda      Deploy a python_function model on Microsoft Azure ML  The MLflow plugin azureml-mlflow can deploy models to Azure ML, either to Azure Kubernetes Service (AKS) or Azure Container Instances (ACI) for real-time serving. The resulting deployment accepts the following data formats as input:  JSON-serialized pandas DataFrames in the split orientation. For example, data = pandas_df.to_json(orient='split'). This format is specified using a Content-Type request header value of application/json.   Warning The TensorSpec input format is not fully supported for deployments on Azure Machine Learning at the moment. Be aware that many autolog() implementations may use TensorSpec for model's signatures when logging models and hence those deployments will fail in Azure ML.  Deployments can be generated using both the Python API or MLflow CLI. In both cases, a JSON configuration file can be indicated with the details of the deployment you want to achieve. If not indicated, then a default deployment is done using Azure Container Instances (ACI) and a minimal configuration. The full specification of this configuration file can be checked at Deployment configuration schema. Also, you will also need the Azure ML MLflow Tracking URI of your particular Azure ML Workspace where you want to deploy your model. You can obtain this URI in several ways:  Through the Azure ML Studio:  Navigate to Azure ML Studio and select the workspace you are working on. Click on the name of the workspace at the upper right corner of the page. Click "View all properties in Azure Portal" on the pane popup. Copy the MLflow tracking URI value from the properties section.   Programmatically, using Azure ML SDK with the method Workspace.get_mlflow_tracking_uri(). If you are running inside Azure ML Compute, like for instance a Compute Instance, you can get this value also from the environment variable os.environ[""MLFLOW_TRACKING_URI""]. Manually, for a given Subscription ID, Resource Group and Azure ML Workspace, the URI is as follows: azureml://eastus.api.azureml.ms/mlflow/v1.0/subscriptions/<SUBSCRIPTION_ID>/resourceGroups/<RESOURCE_GROUP_NAME>/providers/Microsoft.MachineLearningServices/workspaces/<WORKSPACE_NAME>  Configuration example for ACI deployment {   ""computeType"": ""aci"",   ""containerResourceRequirements"":   {     ""cpu"": 1,     ""memoryInGB"": 1   },   ""location"": ""eastus2"", }    Remarks: If containerResourceRequirements is not indicated, a deployment with minimal compute configuration is applied (cpu: 0.1 and memory: 0.5). If location is not indicated, it defaults to the location of the workspace.    Configuration example for an AKS deployment {   ""computeType"": ""aks"",   ""computeTargetName"": ""aks-mlflow"" }    Remarks: In above example, aks-mlflow is the name of an Azure Kubernetes Cluster registered/created in Azure Machine Learning.    The following examples show how to create a deployment in ACI. Please, ensure you have azureml-mlflow installed before continuing. Example: Workflow using the Python API import json from mlflow.deployments import get_deploy_client  # Create the deployment configuration. # If no deployment configuration is provided, then the deployment happens on ACI. deploy_config = {""computeType"": ""aci""}  # Write the deployment configuration into a file. deployment_config_path = ""deployment_config.json"" with open(deployment_config_path, ""w"") as outfile:     outfile.write(json.dumps(deploy_config))  # Set the tracking uri in the deployment client. client = get_deploy_client(""<azureml-mlflow-tracking-url>"")  # MLflow requires the deployment configuration to be passed as a dictionary. config = {""deploy-config-file"": deployment_config_path} model_name = ""mymodel"" model_version = 1  # define the model path and the name is the service name # if model is not registered, it gets registered automatically and a name is autogenerated using the ""name"" parameter below client.create_deployment(     model_uri=f""models:/{model_name}/{model_version}"",     config=config,     name=""mymodel-aci-deployment"", )  # After the model deployment completes, requests can be posted via HTTP to the new ACI # webservice's scoring URI. print(""Scoring URI is: %s"", webservice.scoring_uri)  # The following example posts a sample input from the wine dataset # used in the MLflow ElasticNet example: # https://github.com/mlflow/mlflow/tree/master/examples/sklearn_elasticnet_wine  # `sample_input` is a JSON-serialized pandas DataFrame with the `split` orientation import requests import json  # `sample_input` is a JSON-serialized pandas DataFrame with the `split` orientation sample_input = {     ""columns"": [         ""alcohol"",         ""chlorides"",         ""citric acid"",         ""density"",         ""fixed acidity"",         ""free sulfur dioxide"",         ""pH"",         ""residual sugar"",         ""sulphates"",         ""total sulfur dioxide"",         ""volatile acidity"",     ],     ""data"": [[8.8, 0.045, 0.36, 1.001, 7, 45, 3, 20.7, 0.45, 170, 0.27]], } response = requests.post(     url=webservice.scoring_uri,     data=json.dumps(sample_input),     headers={""Content-type"": ""application/json""}, ) response_json = json.loads(response.text) print(response_json)   Example: Workflow using the MLflow CLI echo ""{ computeType: aci }"" > deployment_config.json mlflow deployments create --name <deployment-name> -m models:/<model-name>/<model-version> -t <azureml-mlflow-tracking-url> --deploy-config-file deployment_config.json  # After the deployment completes, requests can be posted via HTTP to the new ACI # webservice's scoring URI.  scoring_uri=$(az ml service show --name <deployment-name> -v | jq -r "".scoringUri"")  # The following example posts a sample input from the wine dataset # used in the MLflow ElasticNet example: # https://github.com/mlflow/mlflow/tree/master/examples/sklearn_elasticnet_wine  # `sample_input` is a JSON-serialized pandas DataFrame with the `split` orientation sample_input=' {     ""columns"": [         ""alcohol"",         ""chlorides"",         ""citric acid"",         ""density"",         ""fixed acidity"",         ""free sulfur dioxide"",         ""pH"",         ""residual sugar"",         ""sulphates"",         ""total sulfur dioxide"",         ""volatile acidity""     ],     ""data"": [         [8.8, 0.045, 0.36, 1.001, 7, 45, 3, 20.7, 0.45, 170, 0.27]     ] }'  echo $sample_input | curl -s -X POST $scoring_uri\ -H 'Cache-Control: no-cache'\ -H 'Content-Type: application/json'\ -d @-   You can also test your deployments locally first using the option run-local: mlflow deployments run-local --name <deployment-name> -m models:/<model-name>/<model-version> -t <azureml-mlflow-tracking-url>   For more info, see: mlflow deployments help -t azureml     Deploy a python_function model on Amazon SageMaker  The mlflow.deployments and mlflow.sagemaker modules can deploy python_function models locally in a Docker container with SageMaker compatible environment and remotely on SageMaker. To deploy remotely to SageMaker you need to set up your environment and user accounts. To export a custom model to SageMaker, you need a MLflow-compatible Docker image to be available on Amazon ECR. MLflow provides a default Docker image definition; however, it is up to you to build the image and upload it to ECR. MLflow includes the utility function build_and_push_container to perform this step. Once built and uploaded, you can use the MLflow container for all MLflow Models. Model webservers deployed using the mlflow.deployments module accept the following data formats as input, depending on the deployment flavor:  python_function: For this deployment flavor, the endpoint accepts the same formats described in the local model deployment documentation. mleap: For this deployment flavor, the endpoint accepts only JSON-serialized pandas DataFrames in the split orientation. For example, data = pandas_df.to_json(orient='split'). This format is specified using a Content-Type request header value of application/json.   Commands   mlflow deployments run-local -t sagemaker deploys the model locally in a Docker container. The image and the environment should be identical to how the model would be run remotely and it is therefore useful for testing the model prior to deployment. mlflow sagemaker build-and-push-container builds an MLfLow Docker image and uploads it to ECR. The caller must have the correct permissions set up. The image is built locally and requires Docker to be present on the machine that performs this step. mlflow deployments create -t sagemaker deploys the model on Amazon SageMaker. MLflow uploads the Python Function model into S3 and starts an Amazon SageMaker endpoint serving the model.  Example workflow using the MLflow CLI mlflow sagemaker build-and-push-container  # build the container (only needs to be called once) mlflow deployments run-local -t sagemaker --name <deployment-name> -m <path-to-model>  # test the model locally mlflow deployments sagemaker create -t  # deploy the model remotely   For more info, see: mlflow sagemaker --help mlflow sagemaker build-and-push-container --help mlflow deployments run-local --help mlflow deployments help -t sagemaker      Export a python_function model as an Apache Spark UDF  You can output a python_function model as an Apache Spark UDF, which can be uploaded to a Spark cluster and used to score the model. Example from pyspark.sql.functions import struct from pyspark.sql import SparkSession  spark = SparkSession.builder.getOrCreate() pyfunc_udf = mlflow.pyfunc.spark_udf(spark, ""<path-to-model>"") df = spark_df.withColumn(""prediction"", pyfunc_udf(struct([...])))   If a model contains a signature, the UDF can be called without specifying column name arguments. In this case, the UDF will be called with column names from signature, so the evaluation dataframe's column names must match the model signature's column names. Example from pyspark.sql import SparkSession  spark = SparkSession.builder.getOrCreate() pyfunc_udf = mlflow.pyfunc.spark_udf(spark, ""<path-to-model-with-signature>"") df = spark_df.withColumn(""prediction"", pyfunc_udf())   If a model contains a signature with tensor spec inputs, you will need to pass a column of array type as a corresponding UDF argument. The values in this column must be comprised of one-dimensional arrays. The UDF will reshape the array values to the required shape with 'C' order (i.e. read / write the elements using C-like index order) and cast the values as the required tensor spec type. For example, assuming a model requires input 'a' of shape (-1, 2, 3) and input 'b' of shape (-1, 4, 5). In order to perform inference on this data, we need to prepare a Spark DataFrame with column 'a' containing arrays of length 6 and column 'b' containing arrays of length 20. We can then invoke the UDF like following example code: Example from pyspark.sql import SparkSession  spark = SparkSession.builder.getOrCreate() # Assuming the model requires input 'a' of shape (-1, 2, 3) and input 'b' of shape (-1, 4, 5) model_path = ""<path-to-model-requiring-multidimensional-inputs>"" pyfunc_udf = mlflow.pyfunc.spark_udf(spark, model_path) # The `spark_df` has column 'a' containing arrays of length 6 and # column 'b' containing arrays of length 20 df = spark_df.withColumn(""prediction"", pyfunc_udf(struct(""a"", ""b"")))   The resulting UDF is based on Spark's Pandas UDF and is currently limited to producing either a single value, an array of values, or a struct containing multiple field values of the same type per observation. By default, we return the first numeric column as a double. You can control what result is returned by supplying result_type argument. The following values are supported:  'int' or IntegerType: The leftmost integer that can fit in int32 result is returned or an exception is raised if there are none. 'long' or LongType: The leftmost long integer that can fit in int64 result is returned or an exception is raised if there are none. ArrayType (IntegerType | LongType): Return all integer columns that can fit into the requested size. 'float' or FloatType: The leftmost numeric result cast to float32 is returned or an exception is raised if there are no numeric columns. 'double' or DoubleType: The leftmost numeric result cast to double is returned or an exception is raised if there are no numeric columns. ArrayType ( FloatType | DoubleType ): Return all numeric columns cast to the requested type. An exception is raised if there are no numeric columns. 'string' or StringType: Result is the leftmost column cast as string. ArrayType ( StringType ): Return all columns cast as string. 'bool' or 'boolean' or BooleanType: The leftmost column cast to bool is returned or an exception is raised if the values cannot be coerced. 'field1 FIELD1_TYPE, field2 FIELD2_TYPE, ...': A struct type containing multiple fields separated by comma, each field type must be one of types listed above.  Example from pyspark.sql import SparkSession  spark = SparkSession.builder.getOrCreate() # Suppose the PyFunc model `predict` method returns a dict like: # `{'prediction': 1-dim_array, 'probability': 2-dim_array}` # You can supply result_type to be a struct type containing # 2 fields 'prediction' and 'probability' like following. pyfunc_udf = mlflow.pyfunc.spark_udf(     spark, ""<path-to-model>"", result_type=""prediction float, probability: array<float>"" ) df = spark_df.withColumn(""prediction"", pyfunc_udf())   Example from pyspark.sql.types import ArrayType, FloatType from pyspark.sql.functions import struct from pyspark.sql import SparkSession  spark = SparkSession.builder.getOrCreate() pyfunc_udf = mlflow.pyfunc.spark_udf(     spark, ""path/to/model"", result_type=ArrayType(FloatType()) ) # The prediction column will contain all the numeric columns returned by the model as floats df = spark_df.withColumn(""prediction"", pyfunc_udf(struct(""name"", ""age"")))   If you want to use conda to restore the python environment that was used to train the model, set the env_manager argument when calling mlflow.pyfunc.spark_udf(). Example from pyspark.sql.types import ArrayType, FloatType from pyspark.sql.functions import struct from pyspark.sql import SparkSession  spark = SparkSession.builder.getOrCreate() pyfunc_udf = mlflow.pyfunc.spark_udf(     spark,     ""path/to/model"",     result_type=ArrayType(FloatType()),     env_manager=""conda"",  # Use conda to restore the environment used in training ) df = spark_df.withColumn(""prediction"", pyfunc_udf(struct(""name"", ""age"")))      Deployment to Custom Targets  In addition to the built-in deployment tools, MLflow provides a pluggable mlflow.deployments Python API and mlflow deployments CLI for deploying models to custom targets and environments. To deploy to a custom target, you must first install an appropriate third-party Python plugin. See the list of known community-maintained plugins here.  Commands  The mlflow deployments CLI contains the following commands, which can also be invoked programmatically using the mlflow.deployments Python API:  Create: Deploy an MLflow model to a specified custom target Delete: Delete a deployment Update: Update an existing deployment, for example to deploy a new model version or change the deployment's configuration (e.g. increase replica count) List: List IDs of all deployments Get: Print a detailed description of a particular deployment Run Local: Deploy the model locally for testing Help: Show the help string for the specified target  For more info, see: mlflow deployments --help mlflow deployments create --help mlflow deployments delete --help mlflow deployments update --help mlflow deployments list --help mlflow deployments get --help mlflow deployments run-local --help mlflow deployments help --help      Community Model Flavors  Go to the Community Model Flavors page to get an overview of other useful MLflow flavors, which are developed and maintained by the MLflow community.        Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
model-registry.html,"   Documentation  MLflow Model Registry       MLflow Model Registry  The MLflow Model Registry component is a centralized model store, set of APIs, and UI, to collaboratively manage the full lifecycle of an MLflow Model. It provides model lineage (which MLflow experiment and run produced the model), model versioning, model aliasing, model tagging, and annotations.  Table of Contents  Concepts Model Registry Workflows  UI Workflow  Register a Model Find Registered Models Deploy and Organize Models   API Workflow  Adding an MLflow Model to the Model Registry Deploy and Organize Models with Aliases and Tags Fetching an MLflow Model from the Model Registry Serving an MLflow Model from Model Registry Promoting an MLflow Model across environments Adding or Updating an MLflow Model Descriptions Renaming an MLflow Model Listing and Searching MLflow Models Deleting MLflow Models Registering a Model Saved Outside MLflow Registering an Unsupported Machine Learning Model Transitioning an MLflow Model's Stage Archiving an MLflow Model        Concepts  The Model Registry introduces a few concepts that describe and facilitate the full lifecycle of an MLflow Model.  ModelAn MLflow Model is created from an experiment or run that is logged with one of the model flavor's mlflow.<model_flavor>.log_model() methods. Once logged, this model can then be registered with the Model Registry.  Registered ModelAn MLflow Model can be registered with the Model Registry. A registered model has a unique name, contains versions, associated transitional stages, model lineage, and other metadata.  Model VersionEach registered model can have one or many versions. When a new model is added to the Model Registry, it is added as version 1. Each new model registered to the same model name increments the version number.    Model AliasModel aliases allow you to assign a mutable, named reference to a particular version of a registered model. By assigning an alias to a specific model version, you can use the alias to refer that model version via a model URI or the model registry API. For example, you can create an alias named champion that points to version 1 of a model named MyModel. You can then refer to version 1 of MyModel by using the URI models:/MyModel@champion. Aliases are especially useful for deploying models. For example, you could assign a champion alias to the model version intended for production traffic and target this alias in production workloads. You can then update the model serving production traffic by reassigning the champion alias to a different model version.  TagsTags are key-value pairs that you associate with registered models and model versions, allowing you to label and categorize them by function or status. For example, you could apply a tag with key ""task"" and value ""question-answering"" (displayed in the UI as task:question-answering) to registered models intended for question answering tasks. At the model version level, you could tag versions undergoing pre-deployment validation with validation_status:pending and those cleared for deployment with validation_status:approved.  Annotations and DescriptionsYou can annotate the top-level model and each version individually using Markdown, including description and any relevant information useful for the team such as algorithm descriptions, dataset employed or methodology.  Model StageEach distinct model version can be assigned one stage at any given time. MLflow provides predefined stages for common use-cases such as Staging, Production or Archived. You can transition a model version from one stage to another stage.     Model Registry Workflows  If running your own MLflow server, you must use a database-backed backend store in order to access the model registry via the UI or API. See here for more information. Before you can add a model to the Model Registry, you must log it using the log_model methods of the corresponding model flavors. Once a model has been logged, you can add, modify, update, or delete the model in the Model Registry through the UI or the API.  UI Workflow  This section demonstrates how to use the MLflow Model Registry UI to manage your MLflow models.  Register a Model  Follow the steps below to register your MLflow model in the Model Registry.  Open the details page for the MLflow Run containing the logged MLflow model you'd like to register. Select the model folder containing the intended MLflow model in the Artifacts section.        Click the Register Model button, which will trigger a form to pop up. In the Model dropdown menu on the form, you can either select "Create New Model", which creates a new registered model with your MLflow model as its initial version, or select an existing registered model, which registers your model under it as a new version. The screenshot below demonstrates registering the MLflow model to a new registered model named ""iris_model_testing"".         Find Registered Models  After you've registered your models in the Model Registry, you can navigate to them in the following ways.  Navigate to the Registered Models page, which links to your registered models and correponding model versions.     Go to the Artifacts section of your MLflow Runs details page, click the model folder, and then click the model version at the top right to view the version created from that model.        Deploy and Organize Models  You can deploy and organize your models in the Model Registry using model aliases and tags. To set aliases and tags for model versions in your registered model, navigate to the overview page of your registered model, such as the one below.    You can add or edit aliases and tags for a specific model version by clicking on the corresponding Add link or pencil icon in the model verison table.    To learn more about a specific model version, navigate to the details page for that model version.    In this page, you can inspect model version details like the model signature, MLflow source run, and creation timestamp. You can also view and configure the verion's aliases, tags, and description.    API Workflow  An alternative way to interact with Model Registry is using the MLflow model flavor or MLflow Client Tracking API interface. In particular, you can register a model during an MLflow experiment run or after all your experiment runs.  Adding an MLflow Model to the Model Registry  There are three programmatic ways to add a model to the registry. First, you can use the mlflow.<model_flavor>.log_model() method. For example, in your code: from sklearn.datasets import make_regression from sklearn.ensemble import RandomForestRegressor from sklearn.metrics import mean_squared_error from sklearn.model_selection import train_test_split  import mlflow import mlflow.sklearn from mlflow.models import infer_signature  with mlflow.start_run() as run:     X, y = make_regression(n_features=4, n_informative=2, random_state=0, shuffle=False)     X_train, X_test, y_train, y_test = train_test_split(         X, y, test_size=0.2, random_state=42     )     params = {""max_depth"": 2, ""random_state"": 42}     model = RandomForestRegressor(**params)     model.fit(X_train, y_train)      # Infer the model signature     y_pred = model.predict(X_test)     signature = infer_signature(X_test, y_pred)      # Log parameters and metrics using the MLflow APIs     mlflow.log_params(params)     mlflow.log_metrics({""mse"": mean_squared_error(y_test, y_pred)})      # Log the sklearn model and register as version 1     mlflow.sklearn.log_model(         sk_model=model,         artifact_path=""sklearn-model"",         signature=signature,         registered_model_name=""sk-learn-random-forest-reg-model"",     )   In the above code snippet, if a registered model with the name doesn't exist, the method registers a new model and creates Version 1. If a registered model with the name exists, the method creates a new model version. The second way is to use the mlflow.register_model() method, after all your experiment runs complete and when you have decided which model is most suitable to add to the registry. For this method, you will need the run_id as part of the runs:URI argument. result = mlflow.register_model(     ""runs:/d16076a3ec534311817565e6527539c0/sklearn-model"", ""sk-learn-random-forest-reg"" )   If a registered model with the name doesn't exist, the method registers a new model, creates Version 1, and returns a ModelVersion MLflow object. If a registered model with the name exists, the method creates a new model version and returns the version object. And finally, you can use the create_registered_model() to create a new registered model. If the model name exists, this method will throw an MlflowException because creating a new registered model requires a unique name. from mlflow import MlflowClient  client = MlflowClient() client.create_registered_model(""sk-learn-random-forest-reg-model"")   The method above creates an empty registered model with no version associated. You can use create_model_version() as shown below to create a new version of the model. client = MlflowClient() result = client.create_model_version(     name=""sk-learn-random-forest-reg-model"",     source=""mlruns/0/d16076a3ec534311817565e6527539c0/artifacts/sklearn-model"",     run_id=""d16076a3ec534311817565e6527539c0"", )     Deploy and Organize Models with Aliases and Tags  Model aliases and tags help you deploy and organize your models in the Model Registry. Set and delete aliases on models To set, update, and delete aliases using the MLflow Client API, see the examples below: from mlflow import MlflowClient  client = MlflowClient()  # create ""champion"" alias for version 1 of model ""example-model"" client.set_registered_model_alias(""example-model"", ""champion"", 1)  # reassign the ""Champion"" alias to version 2 client.set_registered_model_alias(""example-model"", ""Champion"", 2)  # get a model version by alias client.get_model_version_by_alias(""example-model"", ""Champion"")  # delete the alias client.delete_registered_model_alias(""example-model"", ""Champion"")   Set and delete tags on models To set and delete tags using the MLflow Client API, see the examples below: from mlflow import MlflowClient  client = MlflowClient()  # Set registered model tag client.set_registered_model_tag(""example-model"", ""task"", ""classification"")  # Delete registered model tag client.delete_registered_model_tag(""example-model"", ""task"")  # Set model version tag client.set_model_version_tag(""example-model"", ""1"", ""validation_status"", ""approved"")  # Delete model version tag client.delete_model_version_tag(""example-model"", ""1"", ""validation_status"")   For more details on alias and tag client APIs, see the mlflow.client API documentation.   Fetching an MLflow Model from the Model Registry  After you have registered an MLflow model, you can fetch that model using mlflow.<model_flavor>.load_model(), or more generally, load_model(). You can use the loaded model for one off predictions or in inference workloads such as batch inference. Fetch a specific model version To fetch a specific model version, just supply that version number as part of the model URI. import mlflow.pyfunc  model_name = ""sk-learn-random-forest-reg-model"" model_version = 1  model = mlflow.pyfunc.load_model(model_uri=f""models:/{model_name}/{model_version}"")  model.predict(data)   Fetch a model version by alias To fetch a model version by alias, specify the model alias in the model URI, and it will fetch the model version currently under it. import mlflow.pyfunc  model_name = ""sk-learn-random-forest-reg-model"" alias = ""champion""  champion_version = mlflow.pyfunc.load_model(f""models:/{model_name}@{alias}"")  champion_version.predict(data)   Note that model alias assignments can be updated independently of your production code. If the champion alias in the snippet above is reassigned to a new model version in the Model Registry, the next execution of this snippet will automatically pick up the new model version. This allows you to decouple model deployments from your inference workloads. Fetch the latest model version in a specific stage To fetch a model version by stage, simply provide the model stage as part of the model URI, and it will fetch the most recent version of the model in that stage. import mlflow.pyfunc  model_name = ""sk-learn-random-forest-reg-model"" stage = ""Staging""  model = mlflow.pyfunc.load_model(model_uri=f""models:/{model_name}/{stage}"")  model.predict(data)     Serving an MLflow Model from Model Registry  After you have registered an MLflow model, you can serve the model as a service on your host. #!/usr/bin/env sh  # Set environment variable for the tracking URL where the Model Registry resides export MLFLOW_TRACKING_URI=http://localhost:5000  # Serve the production model from the model registry mlflow models serve -m ""models:/sk-learn-random-forest-reg-model@champion""     Promoting an MLflow Model across environments  Over the course of a model's lifecycle, it might progress through various separate environments like development, testing, staging, production, and so on. This segregation facilitates continuous integration and deployment for the model. In MLflow, you can use registered models to set up environments for your MLflow Models, where each registered model corresponds to a specific environment. Furthermore, you can configure access controls for the registered models using MLflow Authentication. Then, to promote MLflow Models across environments, you can use the copy_model_version() method to copy model versions across registered models. from mlflow import MlflowClient  client = MlflowClient() client.copy_model_version(     src_model_uri=""models:/regression-model-staging@candidate"",     dst_name=""regression-model-production"", )   This code snippet copies the model version with the candidate alias in the regression-model-staging model to the regression-model-production model as the latest version.   Adding or Updating an MLflow Model Descriptions  At any point in a model's lifecycle development, you can update a model version's description using update_model_version(). client = MlflowClient() client.update_model_version(     name=""sk-learn-random-forest-reg-model"",     version=1,     description=""This model version is a scikit-learn random forest containing 100 decision trees"", )     Renaming an MLflow Model  As well as adding or updating a description of a specific version of the model, you can rename an existing registered model using rename_registered_model(). client = MlflowClient() client.rename_registered_model(     name=""sk-learn-random-forest-reg-model"",     new_name=""sk-learn-random-forest-reg-model-100"", )     Listing and Searching MLflow Models  You can fetch a list of registered models in the registry with a simple method. from pprint import pprint  client = MlflowClient() for rm in client.search_registered_models():     pprint(dict(rm), indent=4)   This outputs: {   'creation_timestamp': 1582671933216,     'description': None,     'last_updated_timestamp': 1582671960712,     'latest_versions': [<ModelVersion: creation_timestamp=1582671933246, current_stage='Production', description='A random forest model containing 100 decision trees trained in scikit-learn', last_updated_timestamp=1582671960712, name='sk-learn-random-forest-reg-model', run_id='ae2cc01346de45f79a44a320aab1797b', source='./mlruns/0/ae2cc01346de45f79a44a320aab1797b/artifacts/sklearn-model', status='READY', status_message=None, user_id=None, version=1>,                         <ModelVersion: creation_timestamp=1582671960628, current_stage='None', description=None, last_updated_timestamp=1582671960628, name='sk-learn-random-forest-reg-model', run_id='d994f18d09c64c148e62a785052e6723', source='./mlruns/0/d994f18d09c64c148e62a785052e6723/artifacts/sklearn-model', status='READY', status_message=None, user_id=None, version=2>],     'name': 'sk-learn-random-forest-reg-model'}   With hundreds of models, it can be cumbersome to peruse the results returned from this call. A more efficient approach would be to search for a specific model name and list its version details using search_model_versions() method and provide a filter string such as ""name='sk-learn-random-forest-reg-model'"" client = MlflowClient() for mv in client.search_model_versions(""name='sk-learn-random-forest-reg-model'""):     pprint(dict(mv), indent=4)   This outputs: {     ""creation_timestamp"": 1582671933246,     ""current_stage"": ""Production"",     ""description"": ""A random forest model containing 100 decision trees ""     ""trained in scikit-learn"",     ""last_updated_timestamp"": 1582671960712,     ""name"": ""sk-learn-random-forest-reg-model"",     ""run_id"": ""ae2cc01346de45f79a44a320aab1797b"",     ""source"": ""./mlruns/0/ae2cc01346de45f79a44a320aab1797b/artifacts/sklearn-model"",     ""status"": ""READY"",     ""status_message"": None,     ""user_id"": None,     ""version"": 1, }  {     ""creation_timestamp"": 1582671960628,     ""current_stage"": ""None"",     ""description"": None,     ""last_updated_timestamp"": 1582671960628,     ""name"": ""sk-learn-random-forest-reg-model"",     ""run_id"": ""d994f18d09c64c148e62a785052e6723"",     ""source"": ""./mlruns/0/d994f18d09c64c148e62a785052e6723/artifacts/sklearn-model"",     ""status"": ""READY"",     ""status_message"": None,     ""user_id"": None,     ""version"": 2, }     Deleting MLflow Models   Note Deleting registered models or model versions is irrevocable, so use it judiciously.  You can either delete specific versions of a registered model or you can delete a registered model and all its versions. # Delete versions 1,2, and 3 of the model client = MlflowClient() versions = [1, 2, 3] for version in versions:     client.delete_model_version(         name=""sk-learn-random-forest-reg-model"", version=version     )  # Delete a registered model along with all its versions client.delete_registered_model(name=""sk-learn-random-forest-reg-model"")   While the above workflow API demonstrates interactions with the Model Registry, two exceptional cases require attention. One is when you have existing ML models saved from training without the use of MLflow. Serialized and persisted on disk in sklearn's pickled format, you want to register this model with the Model Registry. The second is when you use an ML framework without a built-in MLflow model flavor support, for instance, vaderSentiment, and want to register the model.   Registering a Model Saved Outside MLflow  Not everyone will start their model training with MLflow. So you may have some models trained before the use of MLflow. Instead of retraining the models, all you want to do is register your saved models with the Model Registry. This code snippet creates a sklearn model, which we assume that you had created and saved in native pickle format.  Note The sklearn library and pickle versions with which the model was saved should be compatible with the current MLflow supported built-in sklearn model flavor.  import numpy as np import pickle  from sklearn import datasets, linear_model from sklearn.metrics import mean_squared_error, r2_score  # source: https://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html  # Load the diabetes dataset diabetes_X, diabetes_y = datasets.load_diabetes(return_X_y=True)  # Use only one feature diabetes_X = diabetes_X[:, np.newaxis, 2]  # Split the data into training/testing sets diabetes_X_train = diabetes_X[:-20] diabetes_X_test = diabetes_X[-20:]  # Split the targets into training/testing sets diabetes_y_train = diabetes_y[:-20] diabetes_y_test = diabetes_y[-20:]   def print_predictions(m, y_pred):     # The coefficients     print(""Coefficients: \n"", m.coef_)     # The mean squared error     print(""Mean squared error: %.2f"" % mean_squared_error(diabetes_y_test, y_pred))     # The coefficient of determination: 1 is perfect prediction     print(""Coefficient of determination: %.2f"" % r2_score(diabetes_y_test, y_pred))   # Create linear regression object lr_model = linear_model.LinearRegression()  # Train the model using the training sets lr_model.fit(diabetes_X_train, diabetes_y_train)  # Make predictions using the testing set diabetes_y_pred = lr_model.predict(diabetes_X_test) print_predictions(lr_model, diabetes_y_pred)  # save the model in the native sklearn format filename = ""lr_model.pkl"" pickle.dump(lr_model, open(filename, ""wb""))   Coefficients: [938.23786125] Mean squared error: 2548.07 Coefficient of determination: 0.47   Once saved in pickled format, we can load the sklearn model into memory using pickle API and register the loaded model with the Model Registry. import mlflow from mlflow.models import infer_signature import numpy as np from sklearn import datasets  # load the model into memory loaded_model = pickle.load(open(filename, ""rb""))  # create a signature for the model based on the input and output data diabetes_X, diabetes_y = datasets.load_diabetes(return_X_y=True) diabetes_X = diabetes_X[:, np.newaxis, 2] signature = infer_signature(diabetes_X, diabetes_y)  # log and register the model using MLflow scikit-learn API mlflow.set_tracking_uri(""sqlite:///mlruns.db"") reg_model_name = ""SklearnLinearRegression"" print(""--"") mlflow.sklearn.log_model(     loaded_model,     ""sk_learn"",     serialization_format=""cloudpickle"",     signature=signature,     registered_model_name=reg_model_name, )   -- Successfully registered model 'SklearnLinearRegression'. 2021/04/02 16:30:57 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation. Model name: SklearnLinearRegression, version 1 Created version '1' of model 'SklearnLinearRegression'.   Now, using MLflow fluent APIs, we reload the model from the Model Registry and score. # load the model from the Model Registry and score model_uri = f""models:/{reg_model_name}/1"" loaded_model = mlflow.sklearn.load_model(model_uri) print(""--"")  # Make predictions using the testing set diabetes_y_pred = loaded_model.predict(diabetes_X_test) print_predictions(loaded_model, diabetes_y_pred)   -- Coefficients: [938.23786125] Mean squared error: 2548.07 Coefficient of determination: 0.47     Registering an Unsupported Machine Learning Model  In some cases, you might use a machine learning framework without its built-in MLflow Model flavor support. For instance, the vaderSentiment library is a standard Natural Language Processing (NLP) library used for sentiment analysis. Since it lacks a built-in MLflow Model flavor, you cannot log or register the model using MLflow Model fluent APIs. To work around this problem, you can create an instance of a mlflow.pyfunc model flavor and embed your NLP model inside it, allowing you to save, log or register the model. Once registered, load the model from the Model Registry and score using the predict function. The code sections below demonstrate how to create a PythonFuncModel class with a vaderSentiment model embedded in it, save, log, register, and load from the Model Registry and score.  Note To use this example, you will need to pip install vaderSentiment.  from sys import version_info import cloudpickle import pandas as pd  import mlflow.pyfunc from vaderSentiment.vaderSentiment import SentimentIntensityAnalyzer  # # Good and readable paper from the authors of this package # http://comp.social.gatech.edu/papers/icwsm14.vader.hutto.pdf #  INPUT_TEXTS = [     {""text"": ""This is a bad movie. You don't want to see it! :-)""},     {""text"": ""Ricky Gervais is smart, witty, and creative!!!!!! :D""},     {""text"": ""LOL, this guy fell off a chair while sleeping and snoring in a meeting""},     {""text"": ""Men shoots himself while trying to steal a dog, OMG""},     {""text"": ""Yay!! Another good phone interview. I nailed it!!""},     {         ""text"": ""This is INSANE! I can't believe it. How could you do such a horrible thing?""     }, ]  PYTHON_VERSION = f""{version_info.major}.{version_info.minor}.{version_info.micro}""   def score_model(model):     # Use inference to predict output from the customized PyFunc model     for i, text in enumerate(INPUT_TEXTS):         text = INPUT_TEXTS[i][""text""]         m_input = pd.DataFrame([text])         scores = loaded_model.predict(m_input)         print(f""<{text}> -- {str(scores[0])}"")   # Define a class and extend from PythonModel class SocialMediaAnalyserModel(mlflow.pyfunc.PythonModel):     def __init__(self):         super().__init__()         # embed your vader model instance         self._analyser = SentimentIntensityAnalyzer()      # preprocess the input with prediction from the vader sentiment model     def _score(self, txt):         prediction_scores = self._analyser.polarity_scores(txt)         return prediction_scores      def predict(self, context, model_input, params=None):         # Apply the preprocess function from the vader model to score         model_output = model_input.apply(lambda col: self._score(col))         return model_output   model_path = ""vader"" reg_model_name = ""PyFuncVaderSentiments"" vader_model = SocialMediaAnalyserModel()  # Set the tracking URI to use local SQLAlchemy db file and start the run # Log MLflow entities and save the model mlflow.set_tracking_uri(""sqlite:///mlruns.db"")  # Save the conda environment for this model. conda_env = {     ""channels"": [""defaults"", ""conda-forge""],     ""dependencies"": [f""python={PYTHON_VERSION}"", ""pip""],     ""pip"": [         ""mlflow"",         f""cloudpickle=={cloudpickle.__version__}"",         ""vaderSentiment==3.3.2"",     ],     ""name"": ""mlflow-env"", }  # Save the model with mlflow.start_run(run_name=""Vader Sentiment Analysis"") as run:     model_path = f""{model_path}-{run.info.run_uuid}""     mlflow.log_param(""algorithm"", ""VADER"")     mlflow.log_param(""total_sentiments"", len(INPUT_TEXTS))     mlflow.pyfunc.save_model(         path=model_path, python_model=vader_model, conda_env=conda_env     )  # Use the saved model path to log and register into the model registry mlflow.pyfunc.log_model(     artifact_path=model_path,     python_model=vader_model,     registered_model_name=reg_model_name,     conda_env=conda_env, )  # Load the model from the model registry and score model_uri = f""models:/{reg_model_name}/1"" loaded_model = mlflow.pyfunc.load_model(model_uri) score_model(loaded_model)   Successfully registered model 'PyFuncVaderSentiments'. 2021/04/05 10:34:15 INFO mlflow.tracking._model_registry.client: Waiting up to 300 seconds for model version to finish creation. Created version '1' of model 'PyFuncVaderSentiments'.  <This is a bad movie. You don't want to see it! :-)> -- {'neg': 0.307, 'neu': 0.552, 'pos': 0.141, 'compound': -0.4047} <Ricky Gervais is smart, witty, and creative!!!!!! :D> -- {'neg': 0.0, 'neu': 0.316, 'pos': 0.684, 'compound': 0.8957} <LOL, this guy fell off a chair while sleeping and snoring in a meeting> -- {'neg': 0.0, 'neu': 0.786, 'pos': 0.214, 'compound': 0.5473} <Men shoots himself while trying to steal a dog, OMG> -- {'neg': 0.262, 'neu': 0.738, 'pos': 0.0, 'compound': -0.4939} <Yay!! Another good phone interview. I nailed it!!> -- {'neg': 0.0, 'neu': 0.446, 'pos': 0.554, 'compound': 0.816} <This is INSANE! I can't believe it. How could you do such a horrible thing?> -- {'neg': 0.357, 'neu': 0.643, 'pos': 0.0, 'compound': -0.8034}     Transitioning an MLflow Model's Stage  Over the course of the model's lifecycle, a model evolves—from development to staging to production. You can transition a registered model to one of the stages: Staging, Production or Archived. client = MlflowClient() client.transition_model_version_stage(     name=""sk-learn-random-forest-reg-model"", version=3, stage=""Production"" )   The accepted values for <stage> are: Staging|Archived|Production|None.   Archiving an MLflow Model  You can move models versions out of a Production stage into an Archived stage. At a later point, if that archived model is not needed, you can delete it. # Archive models version 3 from Production into Archived client = MlflowClient() client.transition_model_version_stage(     name=""sk-learn-random-forest-reg-model"", version=3, stage=""Archived"" )            Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
recipes.html,"   Documentation  MLflow Recipes       MLflow Recipes  MLflow Recipes (previously known as MLflow Pipelines) is a framework that enables data scientists to quickly develop high-quality models and deploy them to production. Compared to ad-hoc ML workflows, MLflow Recipes offers several major benefits:  Get started quickly: Predefined templates for common ML tasks, such as regression modeling, enable data scientists to get started quickly and focus on building great models, eliminating the large amount of boilerplate code that is traditionally required to curate datasets, engineer features, train & tune models, and package models for production deployment. Iterate faster: The intelligent recipe execution engine accelerates model development by caching results from each step of the process and re-running the minimal set of steps as changes are made. Easily ship to production: The modular, git-integrated recipe structure dramatically simplifies the handoff from development to production by ensuring that all model code, data, and configurations are easily reviewable and deployable by ML engineers.   Quickstarts   Prerequisites  MLflow Recipes is available as an extension of the MLflow Python library. You can install MLflow Recipes as follows:  Local: Install MLflow from PyPI: pip install mlflow. Note that MLflow Recipes requires Make, which may not be preinstalled on some Windows systems. Windows users must install Make before using MLflow Recipes. For more information about installing Make on Windows, see https://gnuwin32.sourceforge.net/install.html. Databricks: Install MLflow Recipes from a Databricks Notebook by running %pip install mlflow, or install MLflow Recipes on a Databricks Cluster by following the PyPI library installation instructions here and specifying the mlflow package string.  Note Databricks Runtime version 11.0 or greater is required in order to install MLflow Recipes on Databricks.      NYC taxi fare prediction example  The NYC taxi fare prediction example uses the MLflow Recipes Regression Template to develop and score models on the NYC Taxi (TLC) Trip Record Dataset. You can run the example locally by installing MLflow Recipes and running the Jupyter example regression notebook. You can run the example on Databricks by cloning the example repository with Databricks Repos and running the Databricks example regression notebook. To build and score models for your own use cases, we recommend using the MLflow Recipes Regression Template. For more information, see the Regression Template reference guide.   Classification problem example  The Classification problem example uses the MLflow Recipes Classification Template to develop and score models on the Wine Quality Dataset. You can run the example locally by installing MLflow Recipes and running the Jupyter example classification notebook. You can run the example on Databricks by cloning the example repository with Databricks Repos and running the Databricks example classification notebook. To build and score models for your own use cases, we recommend using the MLflow Recipes Classification Template. For more information, see the Classification Template reference guide.    Key concepts   Steps: A Step represents an individual ML operation, such as ingesting data, fitting an estimator, evaluating a model against test data, or deploying a model for real-time scoring. Each Step accepts a collection of well-defined inputs and produce well-defined outputs according to user-defined configurations and code.   Recipes: A Recipe is an ordered composition of Steps used to solve an ML problem or perform an MLOps task, such as developing a regression model or performing batch model scoring on production data. MLflow Recipes provides APIs and a CLI for running recipes and inspecting their results.   Templates: A Recipe Template is a git repository with a standardized, modular layout containing all of the customizable code and configurations for a Recipe. Configurations are defined in YAML format for easy review via the recipe.yaml file and Profile YAML files. Each template also defines its requirements, data science notebooks, and tests. MLflow Recipes includes predefined templates for a variety of model development and MLOps tasks.   Profiles: Profiles contain user-specific or environment-specific configurations for a Recipe, such as the particular set of hyperparameters being tuned by a data scientist in development or the MLflow Model Registry URI and credentials used to store production-worthy models. Each profile is represented as a YAML file in the Recipe Template (e.g. local.yaml and databricks.yaml).   Step Cards: Step Cards display the results produced by running a Step, including dataset profiles, model performance & explainability plots, overviews of the best model parameters found during tuning, and more. Step Cards and their corresponding dataset and model information are also logged to MLflow Tracking.    Usage   Model development workflow  The general model development workflow for using MLflow Recipes is as follows:  Clone a Recipe Template git repository corresponding to the ML problem that you wish to solve. Follow the template's README file for template-specific instructions.   [Local] Clone the MLflow Recipes Regression Template into a local directory.  git clone https://github.com/mlflow/recipes-regression-template    [Databricks] Clone the MLflow Recipes Regression Template git repository using Databricks Repos.      Edit required fields marked by FIXME::REQUIRED comments in recipe.yaml and profiles/*.yaml. The recipe is runnable once all required fields are filled with proper values. You may proceed to step 3 if this is the first time going through this step. Otherwise, continue to edit the YAML config files as well as steps/*.py files, filling out areas marked by FIXME::OPTIONAL as you see fit to customize the recipe steps to your ML problem for better model performance.     Run the recipe by selecting a desired profile. Profiles are used to quickly switch environment specific recipe settings, such as ingest data location. When a recipe run completes, you may inspect the run results. MLflow Recipes creates and displays an interactive Step Card with the results of the last executed step. Each Recipe Template also includes a Databricks Notebook and a Jupyter Notebook for running the recipe and inspecting its results.    Example API and CLI workflows for running the MLflow Recipes Regression Template and inspecting results. Note that recipes must be run from within their corresponding git repositories.  import os from mlflow.recipes import Recipe from mlflow.pyfunc import PyFuncModel  os.chdir(""~/recipes-regression-template"") regression_recipe = Recipe(profile=""local"") # Run the full recipe regression_recipe.run() # Inspect the model training results regression_recipe.inspect(step=""train"") # Load the trained model regression_model_recipe: PyFuncModel = regression_recipe.get_artifact(""model"")    git clone https://github.com/mlflow/recipes-regression-template cd recipes-regression-template # Run the full recipe mlflow recipes run --profile local # Inspect the model training results mlflow recipes inspect --step train --profile local # Inspect the resulting model performance evaluations mlflow recipes inspect --step evaluate --profile local      An example step card produced by running the evaluate step of the MLflow Recipes Regression Template. The step card results indicate that the trained model passed all performance validations and is ready for registration with the MLflow Model Registry.     An example MLflow run view page, showing artifacts logged from the Recipe's steps.     Example recipe run from the Databricks Notebook included in the MLflow Recipes Regression Template.    Note Data profiling is often best viewed with "quantiles" mode. To switch it on, on the Facet data profile, find Chart to show, click the selector below, and choose Quantiles.    Iterate over step 2 and 3: make changes to an individual step, and test them by running the step and observing the results it produces. Use Recipe.inspect() to visualize the overall Recipe dependency graph and artifacts each step produces. Use Recipe.get_artifact() to further inspect individual step outputs in a notebook. MLflow Recipes intelligently caches results from each Recipe Step, ensuring that steps are only executed if their inputs, code, or configurations have changed, or if such changes have occurred in dependent steps. Once you are satisfied with the results of your changes, commit them to a branch of the Recipe Repository in order to ensure reproducibility, and share or review the changes with your team.     Example Recipe.inspect() output, showing the dependency graph of recipe steps and artifacts each step produces.     Note Before testing changes in a staging or production environment, it is recommended that you commit the changes to a branch of the Recipe Repository to ensure reproducibility.   Note By default, MLflow Recipes caches results from each Recipe Step within the .mlflow subdirectory of the home folder on the local filesystem. The MLFLOW_RECIPES_EXECUTION_DIRECTORY environment variable can be used to specify an alternative location for caching results.       Development environments  We recommend using one of the following environment configurations to develop models with MLflow Recipes:  [Databricks] Edit YAML config and Python files in Databricks Repos. Open separate browser tabs for each file module that you wish to modify. For example, one for the recipe config file recipe.yaml, one for the profile config file profile/databricks.yaml, one for the driver notebook notebooks/databricks.py, and one for the current step (e.g. train) under development steps/train.py. Use notebooks/databricks.py as the driver to run recipe steps and inspect its output. Pin the workspace browser for easy file navigation.    [Local with Jupyter Notebook] Use notebooks/jupyter.ipynb as the driver to run recipe steps and inspect its output. Edit recipe.yaml, steps/*.py and profiles/*.yaml accordingly with an editor of your choice. To run the entire recipe, either run notebooks/jupyter.ipynb or on commandline, invoke mlflow recipes run --profile local (change the current working directory to the project root first).   [Edit locally with IDE (VSCode) and run on Databricks] Edit files on your local machine with VSCode and Jupyter plugin. Use dbx to sync them to Databricks Repos as demonstrated below. On Databricks, use the notebooks/databricks.py notebook as the driver to run recipe steps and inspect their outputs.   Example workflow for efficiently editing a recipe on a local machine and synchronizing changes to Databricks Repos  # Install the Databricks CLI, which is used to remotely access your Databricks Workspace pip install databricks-cli # Configure remote access to your Databricks Workspace databricks configure # Install dbx, which is used to automatically sync changes to and from Databricks Repos pip install dbx # Clone the MLflow Recipes Regression Template git clone https://github.com/mlflow/recipes-regression-template # Enter the MLflow Recipes Regression Template directory and configure dbx within it cd recipes-regression-template dbx configure # Use dbx to enable syncing from the repository directory to Databricks Repos dbx sync repo -d recipes-regression-template # Iteratively make changes to files in the repository directory and observe that they # are automatically synced to Databricks Repos         Recipe Templates  MLflow Recipes currently offers the following predefined templates that can be easily customized to develop and deploy high-quality, production-ready models for your use cases:  MLflow Recipes Regression Template: The MLflow Recipes Regression Template is designed for developing and scoring regression models. For more information, see the Regression Template reference guide. MLflow Recipes Classification Template: The MLflow Recipes Classification Template is designed for developing and scoring classification models. For more information, see the Classification Template reference guide.  Additional recipes for a variety of ML problems and MLOps tasks are under active development.   Detailed reference guide   Template structure  Recipe Templates are git repositories with a standardized, modular layout. The following example provides an overview of the recipe repository structure. It is adapted from the MLflow Recipes Regression Template. ├── recipe.yaml ├── requirements.txt ├── steps │   ├── ingest.py │   ├── split.py │   ├── transform.py │   ├── train.py │   ├── custom_metrics.py ├── profiles │   ├── local.yaml │   ├── databricks.yaml ├── tests │   ├── ingest_test.py │   ├── ... │   ├── train_test.py │   ├── ...   The main components of the Recipe Template layout, which are common across all recipes, are:   recipe.yaml: The main recipe configuration file that declaratively defines the attributes and behavior of each recipe step, such as the input dataset to use for training a model or the performance criteria for promoting a model to production. For reference, see the recipe.yaml configuration file from the MLflow Recipes Regression Template. requirements.txt: A pip requirements file specifying packages that must be installed in order to run the recipe. steps: A directory containing Python code modules used by the recipe steps. For example, the MLflow Recipes Regression Template defines the estimator type and parameters to use when training a model in steps/train.py and defines custom metric computations in steps/custom_metrics.py.   profiles: A directory containing Profile customizations for the configurations defined in recipe.yaml. For example, the MLflow Recipes Regression Template defines a profiles/local.yaml profile that customizes the dataset used for local model development and specifies a local MLflow Tracking store for logging model content. The MLflow Recipes Regression Template also defines a profiles/databricks.yaml profile for development on Databricks. tests: A directory containing Python test code for recipe steps. For example, the MLflow Recipes Regression Template implements tests for the transformer and the estimator defined in the respective steps/transform.py and steps/train.py modules.    Shown below is an example recipe.yaml configuration file adapted from the MLflow Recipes Regression Template. recipe.yaml is the main configuration file for a recipe containing aggregated configurations for all recipe steps; Profile-based substitutions and overrides are supported using Jinja2 templating syntax.  recipe: ""regression/v1"" target_col: ""fare_amount"" primary_metrics: ""root_mean_squared_error"" steps:   ingest: {{INGEST_CONFIG}}   split:     split_ratios: {{SPLIT_RATIOS|default([0.75, 0.125, 0.125])}}   transform:     using: custom     transformer_method: transformer_fn   train:     using: custom     estimator_method: estimator_fn   evaluate:     validation_criteria:       - metric: root_mean_squared_error         threshold: 10       - metric: weighted_mean_squared_error         threshold: 20   register:     allow_non_validated_model: false custom_metrics:   - name: weighted_mean_squared_error     function: weighted_mean_squared_error     greater_is_better: False      Working with profiles  A profile is a collection of customizations for the configurations defined in the recipe's main recipe.yaml file. Profiles are defined as YAML files within the recipe repository's profiles directory. When running a recipe or inspecting its results, the desired profile is specified as an API or CLI argument.   Example API and CLI workflows for running recipes with different profile customizations  import os from mlflow.recipes import Recipe  os.chdir(""~/recipes-regression-template"") # Run the regression recipe to train and evaluate the performance of an ElasticNet regressor regression_recipe_local_elasticnet = Recipe(profile=""local-elasticnet"") regression_recipe_local_elasticnet.run() # Run the recipe again to train and evaluate the performance of an SGD regressor regression_recipe_local_sgd = Recipe(profile=""local-sgd"") regression_recipe_local_sgd.run() # After finding the best model type and updating the 'shared-workspace' profile accordingly, # run the recipe again to retrain the best model in a workspace where teammates can view it regression_recipe_shared = Recipe(profile=""shared-workspace"") regression_recipe_shared.run()    git clone https://github.com/mlflow/recipes-regression-template cd recipes-regression-template # Run the regression recipe to train and evaluate the performance of an ElasticNet regressor mlflow recipes run --profile local-elasticnet # Run the recipe again to train and evaluate the performance of an SGD regressor mlflow recipes run --profile local-sgd # After finding the best model type and updating the 'shared-workspace' profile accordingly, # run the recipe again to retrain the best model in a workspace where teammates can view it mlflow recipes run --profile shared-workspace    The following profile customizations are supported:    overrides If the recipe.yaml configuration file defines a Jinja2-templated attribute with a default value, a profile can override the value by mapping the attribute to a different value using YAML dictionary syntax. Note that override values may have arbitrarily nested types (e.g. lists, dictionaries, lists of dictionaries, …).  Example recipe.yaml configuration file defining an overrideable RMSE_THRESHOLD attribute for validating model performance with a default value of 10  steps:   evaluate:     validation_criteria:       - metric: root_mean_squared_error         # The maximum RMSE value on the test dataset that a model can have         # to be eligible for production deployment         threshold: {{RMSE_THRESHOLD|default(10)}}     Example prod.yaml profile that overrides RMSE_THRESHOLD with a custom value to more aggressively validate model quality for production  RMSE_THRESHOLD: 5.2          substitutions If the recipe.yaml configuration file defines a Jinja2-templated attribute without a default value, a profile must map the attribute to a specific value using YAML dictionary syntax. Note that substitute values may have arbitrarily nested types (e.g. lists, dictionaries, lists of dictionaries, …).  Example recipe.yaml configuration file defining a DATASET_INFO variable whose value must be specified by the selected recipe profile  # Specifies the dataset to use for model training ingest: {{INGEST_CONFIG}}     Example dev.yaml profile that provides a value for DATASET_INFO corresponding to a small dataset for development purposes  INGEST_CONFIG:     location: ./data/taxi-small.parquet     format: parquet          additions If the recipe.yaml configuration file does not define a particular attribute, a profile may define it instead. This capability is helpful for providing values of optional configurations that, if unspecified, a recipe would otherwise ignore.  Example local.yaml profile that specifies a sqlite-based MLflow Tracking store for local testing on a laptop  experiment:   tracking_uri: ""sqlite:///metadata/mlflow/mlruns.db""   name: ""sklearn_regression_experiment""   artifact_location: ""./metadata/mlflow/mlartifacts""           Warning If the recipe.yaml configuration file defines an attribute that cannot be overridden or substituted (i.e. because its value is not specified using Jinja2 templating syntax), a profile must not define it. Defining such an attribute in a profile produces an error.           Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
plugins.html,"   Documentation  MLflow Plugins       MLflow Plugins  As a framework-agnostic tool for machine learning, the MLflow Python API provides developer APIs for writing plugins that integrate with different ML frameworks and backends. Plugins provide a powerful mechanism for customizing the behavior of the MLflow Python client and integrating third-party tools, allowing you to:  Integrate with third-party storage solutions for experiment data, artifacts, and models Integrate with third-party authentication providers, e.g. read HTTP authentication credentials from a special file Use the MLflow client to communicate with other REST APIs, e.g. your organization's existing experiment-tracking APIs Automatically capture additional metadata as run tags, e.g. the git repository associated with a run Add new backend to execute MLflow Project entrypoints.  The MLflow Python API supports several types of plugins:  Tracking Store: override tracking backend logic, e.g. to log to a third-party storage solution ArtifactRepository: override artifact logging logic, e.g. to log to a third-party storage solution Run context providers: specify context tags to be set on runs created via the mlflow.start_run() fluent API. Model Registry Store: override model registry backend logic, e.g. to log to a third-party storage solution MLflow Project backend: override the local execution backend to execute a project on your own cluster (Databricks, kubernetes, etc.) MLflow ModelEvaluator: Define custom model evaluator, which can be used in mlflow.evaluate() API.   Table of Contents  Using an MLflow Plugin  Install the Plugin Run Code Using the Plugin Use Plugin for Client Side Authentication   Writing Your Own MLflow Plugins  Defining a Plugin Testing Your Plugin Distributing Your Plugin   Community Plugins  SQL Server Plugin Aliyun(Alibaba Cloud) OSS Plugin XetHub Plugin Deployment Plugins Model Evaluation Plugins Project Backend Plugins Tracking Store Plugins Artifact Repository Plugins      Using an MLflow Plugin  MLflow plugins are Python packages that you can install using PyPI or conda. This example installs a Tracking Store plugin from source and uses it within an example script.  Install the Plugin  To get started, clone MLflow and install this example plugin: git clone https://github.com/mlflow/mlflow cd mlflow pip install -e tests/resources/mlflow-test-plugin     Run Code Using the Plugin  This plugin defines a custom Tracking Store for tracking URIs with the file-plugin scheme. The plugin implementation delegates to MLflow's built-in file-based run storage. To use the plugin, you can run any code that uses MLflow, setting the tracking URI to one with a file-plugin:// scheme: MLFLOW_TRACKING_URI=file-plugin:$(PWD)/mlruns python examples/quickstart/mlflow_tracking.py   Launch the MLflow UI: cd .. mlflow server --backend-store-uri ./mlflow/mlruns   View results at http://localhost:5000. You should see a newly-created run with a param named "param1" and a metric named "foo":      Use Plugin for Client Side Authentication  MLflow provides RequestAuthProvider plugin to customize auth header for outgoing http request. To use it, implement the RequestAuthProvider class and override the get_name and get_auth methods. get_name should return the name of your auth provider, while get_auth should return the auth object that will be added to the http request. from mlflow.tracking.request_auth.abstract_request_auth_provider import (     RequestAuthProvider, )   class DummyAuthProvider(RequestAuthProvider):     def get_name(self):         return ""dummy_auth_provider_name""      def get_auth(self):         return DummyAuth()   Once you have the implemented request auth provider class, register it in the entry_points and install the plugin. setup(     entry_points={         ""mlflow.request_auth_provider"": ""dummy-backend=DummyAuthProvider"",     }, )   Then set environment variable MLFLOW_TRACKING_AUTH to enable the injection of custom auth. The value of this environment variable should match the name of the auth provider. export MLFLOW_TRACKING_AUTH=dummy_auth_provider_name      Writing Your Own MLflow Plugins   Defining a Plugin  You define an MLflow plugin as a standalone Python package that can be distributed for installation via PyPI or conda. See https://github.com/mlflow/mlflow/tree/master/tests/resources/mlflow-test-plugin for an example package that implements all available plugin types. The example package contains a setup.py that declares a number of entry points: setup(     name=""mflow-test-plugin"",     # Require MLflow as a dependency of the plugin, so that plugin users can simply install     # the plugin and then immediately use it with MLflow     install_requires=[""mlflow""],     ...,     entry_points={         # Define a Tracking Store plugin for tracking URIs with scheme 'file-plugin'         ""mlflow.tracking_store"": ""file-plugin=mlflow_test_plugin.file_store:PluginFileStore"",         # Define a ArtifactRepository plugin for artifact URIs with scheme 'file-plugin'         ""mlflow.artifact_repository"": ""file-plugin=mlflow_test_plugin.local_artifact:PluginLocalArtifactRepository"",         # Define a RunContextProvider plugin. The entry point name for run context providers         # is not used, and so is set to the string ""unused"" here         ""mlflow.run_context_provider"": ""unused=mlflow_test_plugin.run_context_provider:PluginRunContextProvider"",         # Define a RequestHeaderProvider plugin. The entry point name for request header providers         # is not used, and so is set to the string ""unused"" here         ""mlflow.request_header_provider"": ""unused=mlflow_test_plugin.request_header_provider:PluginRequestHeaderProvider"",         # Define a RequestAuthProvider plugin. The entry point name for request auth providers         # is not used, and so is set to the string ""unused"" here         ""mlflow.request_auth_provider"": ""unused=mlflow_test_plugin.request_auth_provider:PluginRequestAuthProvider"",         # Define a Model Registry Store plugin for tracking URIs with scheme 'file-plugin'         ""mlflow.model_registry_store"": ""file-plugin=mlflow_test_plugin.sqlalchemy_store:PluginRegistrySqlAlchemyStore"",         # Define a MLflow Project Backend plugin called 'dummy-backend'         ""mlflow.project_backend"": ""dummy-backend=mlflow_test_plugin.dummy_backend:PluginDummyProjectBackend"",         # Define a MLflow model deployment plugin for target 'faketarget'         ""mlflow.deployments"": ""faketarget=mlflow_test_plugin.fake_deployment_plugin"",         # Define a MLflow model evaluator with name ""dummy_evaluator""         ""mlflow.model_evaluator"": ""dummy_evaluator=mlflow_test_plugin.dummy_evaluator:DummyEvaluator"",     }, )   Each element of this entry_points dictionary specifies a single plugin. You can choose to implement one or more plugin types in your package, and need not implement them all. The type of plugin defined by each entry point and its corresponding reference implementation in MLflow are described below. You can work from the reference implementations when writing your own plugin:         Description Entry-point group Entry-point name and value Reference Implementation    Plugins for overriding definitions of tracking APIs like mlflow.log_metric, mlflow.start_run for a specific tracking URI scheme. mlflow.tracking_store The entry point value (e.g. mlflow_test_plugin.local_store:PluginFileStore) specifies a custom subclass of mlflow.tracking.store.AbstractStore (e.g., the PluginFileStore class within the mlflow_test_plugin module). The entry point name (e.g. file-plugin) is the tracking URI scheme with which to associate the custom AbstractStore implementation. Users who install the example plugin and set a tracking URI of the form file-plugin://<path> will use the custom AbstractStore implementation defined in PluginFileStore. The full tracking URI is passed to the PluginFileStore constructor.  FileStore  Plugins for defining artifact read/write APIs like mlflow.log_artifact, MlflowClient.download_artifacts for a specified artifact URI scheme (e.g. the scheme used by your in-house blob storage system). mlflow.artifact_repository The entry point value (e.g. mlflow_test_plugin.local_artifact:PluginLocalArtifactRepository) specifies a custom subclass of mlflow.store.artifact.artifact_repo.ArtifactRepository (e.g., the PluginLocalArtifactRepository class within the mlflow_test_plugin module). The entry point name (e.g. file-plugin) is the artifact URI scheme with which to associate the custom ArtifactRepository implementation. Users who install the example plugin and log to a run whose artifact URI is of the form file-plugin://<path> will use the custom ArtifactRepository implementation defined in PluginLocalArtifactRepository. The full artifact URI is passed to the PluginLocalArtifactRepository constructor.  LocalArtifactRepository  Plugins for specifying custom context tags at run creation time, e.g. tags identifying the git repository associated with a run. mlflow.run_context_provider The entry point name is unused. The entry point value (e.g. mlflow_test_plugin.run_context_provider:PluginRunContextProvider) specifies a custom subclass of mlflow.tracking.context.abstract_context.RunContextProvider (e.g., the PluginRunContextProvider class within the mlflow_test_plugin module) to register. GitRunContext, DefaultRunContext  Plugins for specifying custom context request headers to attach to outgoing requests, e.g. headers identifying the client's environment. mlflow.request_header_provider The entry point name is unused. The entry point value (e.g. mlflow_test_plugin.request_header_provider:PluginRequestHeaderProvider) specifies a custom subclass of mlflow.tracking.request_header.abstract_request_header_provider.RequestHeaderProvider (e.g., the PluginRequestHeaderProvider class within the mlflow_test_plugin module) to register. DatabricksRequestHeaderProvider  Plugins for specifying custom request auth to attach to outgoing requests. mlflow.request_auth_provider The entry point name is unused. The entry point value (e.g. mlflow_test_plugin.request_auth_provider:PluginRequestAuthProvider) specifies a custom subclass of mlflow.tracking.request_auth.abstract_request_auth_provider.RequestAuthProvider (e.g., the PluginRequestAuthProvider class within the mlflow_test_plugin module) to register. N/A (will be added soon)  Plugins for overriding definitions of Model Registry APIs like mlflow.register_model. mlflow.model_registry_store The entry point value (e.g. mlflow_test_plugin.sqlalchemy_store:PluginRegistrySqlAlchemyStore) specifies a custom subclass of mlflow.tracking.model_registry.AbstractStore (e.g., the PluginRegistrySqlAlchemyStore class within the mlflow_test_plugin module) The entry point name (e.g. file-plugin) is the tracking URI scheme with which to associate the custom AbstractStore implementation. Users who install the example plugin and set a tracking URI of the form file-plugin://<path> will use the custom AbstractStore implementation defined in PluginFileStore. The full tracking URI is passed to the PluginFileStore constructor.  SqlAlchemyStore  Plugins for running MLflow projects against custom execution backends (e.g. to run projects against your team's in-house cluster or job scheduler). mlflow.project.backend The entry point value (e.g. mlflow_test_plugin.dummy_backend:PluginDummyProjectBackend) specifies a custom subclass of mlflow.project.backend.AbstractBackend) N/A (will be added soon)  Plugins for deploying models to custom serving tools. mlflow.deployments The entry point name (e.g. redisai) is the target name. The entry point value (e.g. mlflow_test_plugin.fake_deployment_plugin) specifies a module defining: 1) Exactly one subclass of mlflow.deployments.BaseDeploymentClient (e.g., the PluginDeploymentClient class). MLflow's mlflow.deployments.get_deploy_client API directly returns an instance of this subclass to the user, so you're encouraged to write clear user-facing method and class docstrings as part of your plugin implementation. 1) The run_local and target_help functions, with the target parameter excluded, as shown here PluginDeploymentClient.  Plugins for MLflow Model Evaluation mlflow.model_evaluator The entry point name (e.g. dummy_evaluator) is the evaluator name which is used in the evaluators argument of the mlflow.evaluate API. The entry point value (e.g. dummy_evaluator:DummyEvaluator) must refer to a subclass of mlflow.models.evaluation.ModelEvaluator; the subclass must implement 2 methods: 1) can_evaluate: Accepts the keyword-only arguments model_type and evaluator_config. Returns True if the evaluator can evaluate the specified model type with the specified evaluator config. Returns False otherwise. 1) evaluate: Computes and logs metrics and artifacts, returning evaluation results as an instance of mlflow.models.EvaluationResult. Accepts the following arguments: model (a pyfunc model instance), model_type (identical to the model_type argument from mlflow.evaluate()), dataset (an instance of mlflow.models.evaluation.base._EvaluationDataset containing features and labels (optional) for model evaluation), run_id (the ID of the MLflow Run to which to log results), and evaluator_config (a dictionary of additional configurations for the evaluator). DummyEvaluator.  [Experimental] Plugins for custom mlflow server flask app configuration mlflow.server.app. mlflow.app The entry point <app_name>=<object_reference> (e.g. custom_app=mlflow_test_plugin.app:app) specifies a customized flask application. This can be useful for implementing request hooks for authentication/authorization, custom logging and custom flask configurations. The plugin must import mlflow.server.app (e.g. from mlflow.server import app) and may add custom configuration, middleware etc. to the app. The plugin should avoid altering the existing application routes, handlers and environment variables to avoid unexpected behavior. Users who install the example plugin will have a customized flask application. To run the customized flask application, use mlflow server --app-name <app_name>. app.      Testing Your Plugin  We recommend testing your plugin to ensure that it follows the contract expected by MLflow. For example, a Tracking Store plugin should contain tests verifying correctness of its log_metric, log_param, … etc implementations. See also the tests for MLflow's reference implementations as an example:  Example Tracking Store tests Example ArtifactRepository tests Example RunContextProvider tests Example Model Registry Store tests Example Custom MLflow Evaluator tests Example Custom MLflow server tests    Distributing Your Plugin  Assuming you've structured your plugin similarly to the example plugin, you can distribute it via PyPI. Congrats, you've now written and distributed your own MLflow plugin!    Community Plugins   SQL Server Plugin  The mlflow-dbstore plugin allows MLflow to use a relational database as an artifact store. As of now, it has only been tested with SQL Server as the artifact store. You can install MLflow with the SQL Server plugin via: pip install mlflow[sqlserver]   and then use MLflow as normal. The SQL Server artifact store support will be provided automatically. The plugin implements all of the MLflow artifact store APIs. To use SQL server as an artifact store, a database URI must be provided, as shown in the example below: db_uri = ""mssql+pyodbc://username:password@host:port/database?driver=ODBC+Driver+17+for+SQL+Server""  client.create_experiment(exp_name, artifact_location=db_uri) mlflow.set_experiment(exp_name)  mlflow.onnx.log_model(onnx, ""model"")   The first time an artifact is logged in the artifact store, the plugin automatically creates an artifacts table in the database specified by the database URI and stores the artifact there as a BLOB. Subsequent logged artifacts are stored in the same table. In the example provided above, the log_model operation creates three entries in the database table to store the ONNX model, the MLmodel file and the conda.yaml file associated with the model.   Aliyun(Alibaba Cloud) OSS Plugin  The aliyunstoreplugin allows MLflow to use Alibaba Cloud OSS storage as an artifact store. pip install mlflow[aliyun-oss]   and then use MLflow as normal. The Alibaba Cloud OSS artifact store support will be provided automatically. The plugin implements all of the MLflow artifact store APIs. It expects Aliyun Storage access credentials in the MLFLOW_OSS_ENDPOINT_URL, MLFLOW_OSS_KEY_ID and MLFLOW_OSS_KEY_SECRET environment variables, so you must set these variables on both your client application and your MLflow tracking server. To use Aliyun OSS as an artifact store, an OSS URI of the form oss://<bucket>/<path> must be provided, as shown in the example below: import mlflow import mlflow.pyfunc   class Mod(mlflow.pyfunc.PythonModel):     def predict(self, ctx, inp, params=None):         return 7   exp_name = ""myexp"" mlflow.create_experiment(exp_name, artifact_location=""oss://mlflow-test/"") mlflow.set_experiment(exp_name) mlflow.pyfunc.log_model(""model_test"", python_model=Mod())   In the example provided above, the log_model operation creates three entries in the OSS storage oss://mlflow-test/$RUN_ID/artifacts/model_test/, the MLmodel file and the conda.yaml file associated with the model.   XetHub Plugin  The xethub plugin allows MLflow to use XetHub storage as an artifact store. pip install mlflow[xethub]   and then use MLflow as normal. The XetHub artifact store support will be provided automatically. The plugin implements all of the MLflow artifact store APIs. It expects XetHub access credentials through xet login CLI command or in the XET_USER_EMAIL, XET_USER_NAME and XET_USER_TOKEN environment variables, so you must authenticate with XetHub for both your client application and your MLflow tracking server. To use XetHub as an artifact store, an XetHub URI of the form xet://<username>/<repo>/<branch> must be provided, as shown in the example below: import mlflow import mlflow.pyfunc   class Mod(mlflow.pyfunc.PythonModel):     def predict(self, ctx, inp, params=None):         return 7   exp_name = ""myexp"" mlflow.create_experiment(     exp_name, artifact_location=""xet://<your_username>/mlflow-test/main"" ) mlflow.set_experiment(exp_name) mlflow.pyfunc.log_model(""model_test"", python_model=Mod())   In the example provided above, the log_model operation creates three entries in the OSS storage xet://mlflow-test/$RUN_ID/artifacts/model_test/, the MLmodel file and the conda.yaml file associated with the model.   Deployment Plugins  The following known plugins provide support for deploying models to custom serving tools using MLflow's model deployment APIs. See the individual plugin pages for installation instructions, and see the Python API docs and CLI docs for usage instructions and examples.  mlflow-redisai mlflow-torchserve mlflow-algorithmia mlflow-ray-serve mlflow-azureml oci-mlflow Leverages Oracle Cloud Infrastructure (OCI) Model Deployment service for the deployment of MLflow models.    Model Evaluation Plugins  The following known plugins provide support for evaluating models with custom validation tools using MLflow's mlflow.evaluate() API:  mlflow-giskard: Detect hidden vulnerabilities in ML models, from tabular to LLMs, before moving to production. Anticipate issues such as Performance bias, Unrobustness, Overconfidence, Underconfidence, Ethical bias, Data leakage, Stochasticity, Spurious correlation, and others. Conduct model comparisons using a wide range of tests, either through custom or domain-specific test suites. mlflow-trubrics: validating ML models with Trubrics    Project Backend Plugins  The following known plugins provide support for running MLflow projects against custom execution backends.  mlflow-yarn Running mlflow on Hadoop/YARN oci-mlflow Running mlflow projects on Oracle Cloud Infrastructure (OCI)    Tracking Store Plugins  The following known plugins provide support for running MLflow Tracking Store against custom databases.  mlflow-elasticsearchstore Running MLflow Tracking Store with Elasticsearch  For additional information regarding this plugin, refer to <https://github.com/criteo/mlflow-elasticsearchstore/issues>. The library is available on PyPI here : <https://pypi.org/project/mlflow-elasticsearchstore/>   Artifact Repository Plugins   oci-mlflow Leverages Oracle Cloud Infrastructure (OCI) Object Storage service to store MLflow models artifacts.          Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
auth/index.html,"   Documentation  MLflow Authentication       MLflow Authentication   Note This feature is still experimental and may change in a future release without warning.  MLflow supports basic HTTP authentication to enable access control over experiments and registered models. Once enabled, any visitor will be required to login before they can view any resource from the Tracking Server.  Table of Contents  Overview How It Works  Permissions Permissions Database Admin Users Managing Permissions   Authenticating to MLflow  Using MLflow UI Using Environment Variables Using Credentials File Using REST API   Creating a New User  Using MLflow UI Using REST API Using MLflow AuthServiceClient   Configuration Custom Authentication   MLflow Authentication provides Python and REST API for managing users and permissions.   MLflow Authentication Python API MLflow Authentication REST API    Overview  To enable MLflow authentication, launch the MLflow UI with the following command: mlflow server --app-name basic-auth   Server admin can choose to disable this feature anytime by restarting the server without the app-name flag. Any users and permissions created will be persisted on a SQL database and will be back in service once the feature is re-enabled. Due to the nature of HTTP authentication, it is only supported on a remote Tracking Server, where users send requests to the server via REST APIs.   How It Works   Permissions  The available permissions are:          Permission Can read Can update Can delete Can manage    READ Yes No No No  EDIT Yes Yes No No  MANAGE Yes Yes Yes Yes  NO_PERMISSIONS No No No No    The default permission for all users is READ. It can be changed in the configuration file. Permissions can be granted on individual resources for each user. Supported resources include Experiment and Registered Model. To access an API endpoint, an user must have the required permission. Otherwise, a 403 Forbidden response will be returned. Required Permissions for accessing experiments:         API Endpoint Method Required permission    Create Experiment 2.0/mlflow/experiments/create POST None  Get Experiment 2.0/mlflow/experiments/get GET can_read  Get Experiment By Name 2.0/mlflow/experiments/get-by-name GET can_read  Delete Experiment 2.0/mlflow/experiments/delete POST can_delete  Restore Experiment 2.0/mlflow/experiments/restore POST can_delete  Update Experiment 2.0/mlflow/experiments/update POST can_update  Search Experiments 2.0/mlflow/experiments/search POST None  Search Experiments 2.0/mlflow/experiments/search GET None  Set Experiment Tag 2.0/mlflow/experiments/set-experiment-tag POST can_update  Create Run 2.0/mlflow/runs/create POST can_update  Get Run 2.0/mlflow/runs/get GET can_read  Update Run 2.0/mlflow/runs/update POST can_update  Delete Run 2.0/mlflow/runs/delete POST can_delete  Restore Run 2.0/mlflow/runs/restore POST can_delete  Search Runs 2.0/mlflow/runs/search POST None  Set Tag 2.0/mlflow/runs/set-tag POST can_update  Delete Tag 2.0/mlflow/runs/delete-tag POST can_update  Log Metric 2.0/mlflow/runs/log-metric POST can_update  Log Param 2.0/mlflow/runs/log-parameter POST can_update  Log Batch 2.0/mlflow/runs/log-batch POST can_update  Log Model 2.0/mlflow/runs/log-model POST can_update  List Artifacts 2.0/mlflow/artifacts/list GET can_read  Get Metric History 2.0/mlflow/metrics/get-history GET can_read    Required Permissions for accessing registered models:         API Endpoint Method Required permission    Create Registered Model 2.0/mlflow/registered-models/create POST None  Rename Registered Model 2.0/mlflow/registered-models/rename POST can_update  Update Registered Model 2.0/mlflow/registered-models/update PATCH can_update  Delete Registered Model 2.0/mlflow/registered-models/delete DELETE can_delete  Get Registered Model 2.0/mlflow/registered-models/get GET can_read  Search Registered Models 2.0/mlflow/registered-models/search GET None  Get Latest Versions 2.0/mlflow/registered-models/get-latest-versions POST can_read  Get Latest Versions 2.0/mlflow/registered-models/get-latest-versions GET can_read  Set Registered Model Tag 2.0/mlflow/registered-models/set-tag POST can_update  Delete Registered Model Tag 2.0/mlflow/registered-models/delete-tag DELETE can_update  Set Registered Model Alias 2.0/mlflow/registered-models/alias POST can_update  Delete Registered Model Alias 2.0/mlflow/registered-models/alias DELETE can_delete  Get Model Version By Alias 2.0/mlflow/registered-models/alias GET can_read  Create Model Version 2.0/mlflow/model-versions/create POST can_update  Update Model Version 2.0/mlflow/model-versions/update PATCH can_update  Transition Model Version Stage 2.0/mlflow/model-versions/transition-stage POST can_update  Delete Model Version 2.0/mlflow/model-versions/delete DELETE can_delete  Get Model Version 2.0/mlflow/model-versions/get GET can_read  Search Model Versions 2.0/mlflow/model-versions/search GET None  Get Model Version Download Uri 2.0/mlflow/model-versions/get-download-uri GET can_read  Set Model Version Tag 2.0/mlflow/model-versions/set-tag POST can_update  Delete Model Version Tag 2.0/mlflow/model-versions/delete-tag DELETE can_delete    MLflow Authentication introduces several new API endpoints to manage users and permissions.         API Endpoint Method Required permission    Create User 2.0/mlflow/users/create POST None  Get User 2.0/mlflow/users/get GET Only readable by that user  Update User Password 2.0/mlflow/users/update-password PATCH Only updatable by that user  Update User Admin 2.0/mlflow/users/update-admin PATCH Only admin  Delete User 2.0/mlflow/users/delete DELETE Only admin  Create Experiment Permission 2.0/mlflow/experiments/permissions/create POST can_manage  Get Experiment Permission 2.0/mlflow/experiments/permissions/get GET can_manage  Update Experiment Permission 2.0/mlflow/experiments/permissions/update PATCH can_manage  Delete Experiment Permission 2.0/mlflow/experiments/permissions/delete DELETE can_manage  Create Registered Model Permission 2.0/mlflow/registered-models/permissions/create POST can_manage  Get Registered Model Permission 2.0/mlflow/registered-models/permissions/get GET can_manage  Update Registered Model Permission 2.0/mlflow/registered-models/permissions/update PATCH can_manage  Delete Registered Model Permission 2.0/mlflow/registered-models/permissions/delete DELETE can_manage    Some APIs will also have their behaviour modified. For example, the creator of an experiment will automatically be granted MANAGE permission on that experiment, so that the creator can grant or revoke other users' access to that experiment.         API Endpoint Method Effect    Create Experiment 2.0/mlflow/experiments/create POST Automatically grants MANAGE permission to the creator.  Create Registered Model 2.0/mlflow/registered-models/create POST Automatically grants MANAGE permission to the creator.  Search Experiments 2.0/mlflow/experiments/search POST Only returns experiments which the user has READ permission on.  Search Experiments 2.0/mlflow/experiments/search GET Only returns experiments which the user has READ permission on.  Search Runs 2.0/mlflow/runs/search POST Only returns experiments which the user has READ permission on.  Search Registered Models 2.0/mlflow/registered-models/search GET Only returns registered models which the user has READ permission on.  Search Model Versions 2.0/mlflow/model-versions/search GET Only returns registered models which the user has READ permission on.      Permissions Database  All users and permissions are stored in a database in basic_auth.db, relative to the directory where MLflow server is launched. The location can be changed in the configuration file. To run migrations, use the following command: python -m mlflow.server.auth db upgrade --url <database_url>     Admin Users  Admin users have unrestricted access to all MLflow resources, including creating or deleting users, updating password and admin status of other users, granting or revoking permissions from other users, and managing permissions for all MLflow resources, even if NO_PERMISSIONS is explicitly set to that admin account. MLflow has a built-in admin user that will be created the first time that the MLflow authentication feature is enabled.  Note It is recommended that you update the default admin password as soon as possible after creation.  The default admin user credentials are as follows:       Username Password    admin password    Multiple admin users can exist by promoting other users to admin, using the 2.0/mlflow/users/update-admin endpoint.  Example  # authenticate as built-in admin user export MLFLOW_TRACKING_USERNAME=admin export MLFLOW_TRACKING_PASSWORD=password    from mlflow.server import get_app_client  tracking_uri = ""http://localhost:5000/""  auth_client = get_app_client(""basic-auth"", tracking_uri=tracking_uri) auth_client.create_user(username=""user1"", password=""pw1"") auth_client.update_user_admin(username=""user1"", is_admin=True)     Managing Permissions  MLflow provides REST APIs and a client class AuthServiceClient to manage users and permissions. To instantiate AuthServiceClient, it is recommended that you use mlflow.server.get_app_client().  Example  export MLFLOW_TRACKING_USERNAME=admin export MLFLOW_TRACKING_PASSWORD=password    from mlflow import MlflowClient from mlflow.server import get_app_client  tracking_uri = ""http://localhost:5000/""  auth_client = get_app_client(""basic-auth"", tracking_uri=tracking_uri) auth_client.create_user(username=""user1"", password=""pw1"") auth_client.create_user(username=""user2"", password=""pw2"")  client = MlflowClient(tracking_uri=tracking_uri) experiment_id = client.create_experiment(name=""experiment"")  auth_client.create_experiment_permission(     experiment_id=experiment_id, username=""user2"", permission=""MANAGE"" )      Authenticating to MLflow   Using MLflow UI  When a user first visits the MLflow UI on a browser, they will be prompted to login. There is no limit to how many login attempts can be made. Currently, MLflow UI does not display any information about the current user. Once a user is logged in, the only way to log out is to close the browser.      Using Environment Variables  MLflow provides two environment variables for authentication: MLFLOW_TRACKING_USERNAME and MLFLOW_TRACKING_PASSWORD. To use basic authentication, you must set both environment variables. export MLFLOW_TRACKING_USERNAME=username export MLFLOW_TRACKING_PASSWORD=password   import mlflow  mlflow.set_tracking_uri(""https://<mlflow_tracking_uri>/"") with mlflow.start_run():     ...     Using Credentials File  You can save your credentials in a file to remove the need for setting environment variables every time. The credentials should be saved in ~/.mlflow/credentials using INI format. Note that the password will be stored unencrypted on disk, and is protected only by filesystem permissions. If the environment variables MLFLOW_TRACKING_USERNAME and MLFLOW_TRACKING_PASSWORD are configured, they override any credentials provided in the credentials file.  Credentials file format  [mlflow] mlflow_tracking_username = username mlflow_tracking_password = password      Using REST API  A user can authenticate using the HTTP Authorization request header. See https://developer.mozilla.org/en-US/docs/Web/HTTP/Authentication for more information. In Python, you can use the requests library: import requests  response = requests.get(     ""https://<mlflow_tracking_uri>/"",     auth=(""username"", ""password""), )      Creating a New User   Important To create a new user, you are required to authenticate with admin privileges.   Using MLflow UI  MLflow UI provides a simple page for creating new users at <tracking_uri>/signup.      Using REST API  Alternatively, you can send POST requests to the Tracking Server endpoint 2.0/users/create. In Python, you can use the requests library: import requests  response = requests.post(     ""https://<mlflow_tracking_uri>/api/2.0/mlflow/users/create"",     json={         ""username"": ""username"",         ""password"": ""password"",     }, )     Using MLflow AuthServiceClient  MLflow AuthServiceClient provides a function to create new users easily. import mlflow  auth_client = mlflow.server.get_app_client(     ""basic-auth"", tracking_uri=""https://<mlflow_tracking_uri>/"" ) auth_client.create_user(username=""username"", password=""password"")      Configuration  Authentication configuration is located at mlflow/server/auth/basic_auth.ini:       Variable Description    default_permission Default permission on all resources  database_uri Database location to store permission and user data  admin_username Default admin username if the admin is not already created  admin_password Default admin password if the admin is not already created  authorization_function Function to authenticate requests    The authorization_function setting supports pluggable authentication methods if you want to use another authentication method than HTTP basic auth. The value specifies module_name:function_name. The function has the following signature:  def authenticate_request() -> Union[Authorization, Response]:     ...    The function should return a werkzeug.datastructures.Authorization object if the request is authenticated, or a Response object (typically 401: Unauthorized) if the request is not authenticated. For an example of how to implement a custom authentication method, see tests/server/auth/jwt_auth.py. NOTE: This example is not intended for production use.   Custom Authentication  MLflow authentication is designed to be extensible. If your organization desires more advanced authentication logic (e.g., token-based authentication), it is possible to install a third party plugin or to create your own plugin. Your plugin should be an installable Python package. It should include an app factory that extends the MLflow app and, optionally, implement a client to manage permissions. The app factory function name will be passed to the --app argument in Flask CLI. See https://flask.palletsprojects.com/en/latest/cli/#application-discovery for more information.  Example: my_auth/__init__.py  from flask import Flask from mlflow.server import app   def create_app(app: Flask = app):     app.add_url_rule(...)     return app   class MyAuthClient:     ...    Then, the plugin should be installed in your Python environment: pip install my_auth   Then, register your plugin in mlflow/setup.py: setup(     ...,     entry_points=""""""         ...          [mlflow.app]         my-auth=my_auth:create_app          [mlflow.app.client]         my-auth=my_auth:MyAuthClient     """""", )   Then, you can start the MLflow server: mlflow server --app-name my-auth          Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
cli.html,"   Documentation  Command-Line Interface       Command-Line Interface  The MLflow command-line interface (CLI) provides a simple interface to various functionality in MLflow. You can use the CLI to run projects, start the tracking UI, create and list experiments, download run artifacts, serve MLflow Python Function and scikit-learn models, serve MLflow Python Function and scikit-learn models, and serve models on Microsoft Azure Machine Learning and Amazon SageMaker. Each individual command has a detailed help screen accessible via mlflow command_name --help.  Table of Contents  mlflow  artifacts db deployments doctor experiments gateway gc models recipes run runs sagemaker server      mlflow  mlflow [OPTIONS] COMMAND [ARGS]...   Options   --version  Show the version and exit.   artifacts  Upload, list, and download artifacts from an MLflow artifact repository. To manage artifacts for a run associated with a tracking server, set the MLFLOW_TRACKING_URI environment variable to the URL of the desired server. mlflow artifacts [OPTIONS] COMMAND [ARGS]...    download  Download an artifact file or directory to a local directory. The output is the name of the file or directory on the local filesystem. Either --artifact-uri or --run-id must be provided. mlflow artifacts download [OPTIONS]   Options   -r, --run-id <run_id>  Run ID from which to download    -a, --artifact-path <artifact_path>  For use with Run ID: if specified, a path relative to the run's root directory to download    -u, --artifact-uri <artifact_uri>  URI pointing to the artifact file or artifacts directory; use as an alternative to specifying –run_id and –artifact-path    -d, --dst-path <dst_path>  Path of the local filesystem destination directory to which to download the specified artifacts. If the directory does not exist, it is created. If unspecified the artifacts are downloaded to a new uniquely-named directory on the local filesystem, unless the artifacts already exist on the local filesystem, in which case their local path is returned directly    list  Return all the artifacts directly under run's root artifact directory, or a sub-directory. The output is a JSON-formatted list. mlflow artifacts list [OPTIONS]   Options   -r, --run-id <run_id>  Required Run ID to be listed    -a, --artifact-path <artifact_path>  If specified, a path relative to the run's root directory to list.    log-artifact  Log a local file as an artifact of a run, optionally within a run-specific artifact path. Run artifacts can be organized into directories, so you can place the artifact in a directory this way. mlflow artifacts log-artifact [OPTIONS]   Options   -l, --local-file <local_file>  Required Local path to artifact to log    -r, --run-id <run_id>  Required Run ID into which we should log the artifact.    -a, --artifact-path <artifact_path>  If specified, we will log the artifact into this subdirectory of the run's artifact directory.    log-artifacts  Log the files within a local directory as an artifact of a run, optionally within a run-specific artifact path. Run artifacts can be organized into directories, so you can place the artifact in a directory this way. mlflow artifacts log-artifacts [OPTIONS]   Options   -l, --local-dir <local_dir>  Required Directory of local artifacts to log    -r, --run-id <run_id>  Required Run ID into which we should log the artifact.    -a, --artifact-path <artifact_path>  If specified, we will log the artifact into this subdirectory of the run's artifact directory.     db  Commands for managing an MLflow tracking database. mlflow db [OPTIONS] COMMAND [ARGS]...    upgrade  Upgrade the schema of an MLflow tracking database to the latest supported version. IMPORTANT: Schema migrations can be slow and are not guaranteed to be transactional - always take a backup of your database before running migrations. The migrations README, which is located at https://github.com/mlflow/mlflow/blob/master/mlflow/store/db_migrations/README.md, describes large migrations and includes information about how to estimate their performance and recover from failures. mlflow db upgrade [OPTIONS] URL   Arguments   URL  Required argument     deployments  Deploy MLflow models to custom targets. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions in https://mlflow.org/docs/latest/plugins.html#community-plugins You can also write your own plugin for deployment to a custom target. For instructions on writing and distributing a plugin, see https://mlflow.org/docs/latest/plugins.html#writing-your-own-mlflow-plugins. mlflow deployments [OPTIONS] COMMAND [ARGS]...    create  Deploy the model at model_uri to the specified target. Additional plugin-specific arguments may also be passed to this command, via -C key=value mlflow deployments create [OPTIONS]   Options   --endpoint <endpoint>  Name of the endpoint    -C, --config <NAME=VALUE>  Extra target-specific config for the model deployment, of the form -C name=value. See documentation/help for your deployment target for a list of supported config options.    --name <name>  Required Name of the deployment    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    -m, --model-uri <URI>  Required URI to the model. A local path, a 'runs:/' URI, or a remote storage URI (e.g., an 's3://' URI). For more information about supported remote URIs for model artifacts, see https://mlflow.org/docs/latest/tracking.html#artifact-stores    -f, --flavor <flavor>  Which flavor to be deployed. This will be auto inferred if it's not given    create-endpoint  Create an endpoint with the specified name at the specified target. Additional plugin-specific arguments may also be passed to this command, via -C key=value mlflow deployments create-endpoint [OPTIONS]   Options   -C, --config <NAME=VALUE>  Extra target-specific config for the endpoint, of the form -C name=value. See documentation/help for your deployment target for a list of supported config options.    --endpoint <endpoint>  Required Name of the endpoint    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    delete  Delete the deployment with name given at –name from the specified target. mlflow deployments delete [OPTIONS]   Options   --endpoint <endpoint>  Name of the endpoint    -C, --config <NAME=VALUE>  Extra target-specific config for the model deployment, of the form -C name=value. See documentation/help for your deployment target for a list of supported config options.    --name <name>  Required Name of the deployment    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    delete-endpoint  Delete the specified endpoint at the specified target mlflow deployments delete-endpoint [OPTIONS]   Options   --endpoint <endpoint>  Required Name of the endpoint    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    explain  Generate explanations of model predictions on the specified input for the deployed model for the given input(s). Explanation output formats vary by deployment target, and can include details like feature importance for understanding/debugging predictions. Run mlflow deployments help or consult the documentation for your plugin for details on explanation format. For information about the input data formats accepted by this function, see the following documentation: https://www.mlflow.org/docs/latest/models.html#built-in-deployment-tools mlflow deployments explain [OPTIONS]   Options   --name <name>  Name of the deployment. Exactly one of –name or –endpoint must be specified.    --endpoint <endpoint>  Name of the endpoint. Exactly one of –name or –endpoint must be specified.    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    -I, --input-path <input_path>  Required Path to input json file for prediction    -O, --output-path <output_path>  File to output results to as a JSON file. If not provided, prints output to stdout.    get  Print a detailed description of the deployment with name given at --name in the specified target. mlflow deployments get [OPTIONS]   Options   --endpoint <endpoint>  Name of the endpoint    --name <name>  Required Name of the deployment    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    get-endpoint  Get details for the specified endpoint at the specified target mlflow deployments get-endpoint [OPTIONS]   Options   --endpoint <endpoint>  Required Name of the endpoint    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    help  Display additional help for a specific deployment target, e.g. info on target-specific config options and the target's URI format. mlflow deployments help [OPTIONS]   Options   -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    list  List the names of all model deployments in the specified target. These names can be used with the delete, update, and get commands. mlflow deployments list [OPTIONS]   Options   --endpoint <endpoint>  Name of the endpoint    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    list-endpoints  List all endpoints at the specified target mlflow deployments list-endpoints [OPTIONS]   Options   -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    predict  Predict the results for the deployed model for the given input(s) mlflow deployments predict [OPTIONS]   Options   --name <name>  Name of the deployment. Exactly one of –name or –endpoint must be specified.    --endpoint <endpoint>  Name of the endpoint. Exactly one of –name or –endpoint must be specified.    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    -I, --input-path <input_path>  Required Path to input json file for prediction    -O, --output-path <output_path>  File to output results to as a JSON file. If not provided, prints output to stdout.    run-local  Deploy the model locally. This has very similar signature to create API mlflow deployments run-local [OPTIONS]   Options   -C, --config <NAME=VALUE>  Extra target-specific config for the model deployment, of the form -C name=value. See documentation/help for your deployment target for a list of supported config options.    --name <name>  Required Name of the deployment    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    -m, --model-uri <URI>  Required URI to the model. A local path, a 'runs:/' URI, or a remote storage URI (e.g., an 's3://' URI). For more information about supported remote URIs for model artifacts, see https://mlflow.org/docs/latest/tracking.html#artifact-stores    -f, --flavor <flavor>  Which flavor to be deployed. This will be auto inferred if it's not given    update  Update the deployment with ID deployment_id in the specified target. You can update the URI of the model and/or the flavor of the deployed model (in which case the model URI must also be specified). Additional plugin-specific arguments may also be passed to this command, via -C key=value. mlflow deployments update [OPTIONS]   Options   --endpoint <endpoint>  Name of the endpoint    -C, --config <NAME=VALUE>  Extra target-specific config for the model deployment, of the form -C name=value. See documentation/help for your deployment target for a list of supported config options.    --name <name>  Required Name of the deployment    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins    -m, --model-uri <URI>  URI to the model. A local path, a 'runs:/' URI, or a remote storage URI (e.g., an 's3://' URI). For more information about supported remote URIs for model artifacts, see https://mlflow.org/docs/latest/tracking.html#artifact-stores    -f, --flavor <flavor>  Which flavor to be deployed. This will be auto inferred if it's not given    update-endpoint  Update the specified endpoint at the specified target. Additional plugin-specific arguments may also be passed to this command, via -C key=value mlflow deployments update-endpoint [OPTIONS]   Options   -C, --config <NAME=VALUE>  Extra target-specific config for the endpoint, of the form -C name=value. See documentation/help for your deployment target for a list of supported config options.    --endpoint <endpoint>  Required Name of the endpoint    -t, --target <target>  Required Deployment target URI. Run mlflow deployments help –target-name <target-name> for more details on the supported URI format and config options for a given target. Support is currently installed for deployment to: sagemaker See all supported deployment targets and installation instructions at https://mlflow.org/docs/latest/plugins.html#community-plugins     doctor  Prints out useful information for debugging issues with MLflow. mlflow doctor [OPTIONS]   Options   --mask-envs  If set (the default behavior without setting this flag is not to obfuscate information), mask the MLflow environment variable values (e.g. "MLFLOW_ENV_VAR": "***") in the output to prevent leaking sensitive information.    experiments  Manage experiments. To manage experiments associated with a tracking server, set the MLFLOW_TRACKING_URI environment variable to the URL of the desired server. mlflow experiments [OPTIONS] COMMAND [ARGS]...    create  Create an experiment. All artifacts generated by runs related to this experiment will be stored under artifact location, organized under specific run_id sub-directories. Implementation of experiment and metadata store is dependent on backend storage. FileStore creates a folder for each experiment ID and stores metadata in meta.yaml. Runs are stored as subfolders. mlflow experiments create [OPTIONS]   Options   -n, --experiment-name <experiment_name>  Required    -l, --artifact-location <artifact_location>  Base location for runs to store artifact results. Artifacts will be stored at $artifact_location/$run_id/artifacts. See https://mlflow.org/docs/latest/tracking.html#where-runs-are-recorded for more info on the properties of artifact location. If no location is provided, the tracking server will pick a default.    csv  Generate CSV with all runs for an experiment mlflow experiments csv [OPTIONS]   Options   -x, --experiment-id <experiment_id>  Required    -o, --filename <filename>     delete  Mark an active experiment for deletion. This also applies to experiment's metadata, runs and associated data, and artifacts if they are store in default location. Use list command to view artifact location. Command will throw an error if experiment is not found or already marked for deletion. Experiments marked for deletion can be restored using restore command, unless they are permanently deleted. Specific implementation of deletion is dependent on backend stores. FileStore moves experiments marked for deletion under a .trash folder under the main folder used to instantiate FileStore. Experiments marked for deletion can be permanently deleted by clearing the .trash folder. It is recommended to use a cron job or an alternate workflow mechanism to clear .trash folder. mlflow experiments delete [OPTIONS]   Options   -x, --experiment-id <experiment_id>  Required    rename  Renames an active experiment. Returns an error if the experiment is inactive. mlflow experiments rename [OPTIONS]   Options   -x, --experiment-id <experiment_id>  Required    --new-name <new_name>  Required    restore  Restore a deleted experiment. This also applies to experiment's metadata, runs and associated data. The command throws an error if the experiment is already active, cannot be found, or permanently deleted. mlflow experiments restore [OPTIONS]   Options   -x, --experiment-id <experiment_id>  Required    search  Search for experiments in the configured tracking server. mlflow experiments search [OPTIONS]   Options   -v, --view <view>  Select view type for experiments. Valid view types are 'active_only' (default), 'deleted_only', and 'all'.     gateway  Manage the MLflow Gateway service mlflow gateway [OPTIONS] COMMAND [ARGS]...    start  Start the MLflow Gateway service mlflow gateway start [OPTIONS]   Options   --config-path <config_path>  Required The path to the gateway configuration file.    --host <host>  The network address to listen on (default: 127.0.0.1).    --port <port>  The port to listen on (default: 5000).    --workers <workers>  The number of workers.  Environment variables   MLFLOW_GATEWAY_CONFIG  Provide a default for --config-path      gc  Permanently delete runs in the deleted lifecycle stage from the specified backend store. This command deletes all artifacts and metadata associated with the specified runs. If the provided artifact URL is invalid, the artifact deletion will be bypassed, and the gc process will continue. mlflow gc [OPTIONS]   Options   --older-than <older_than>  Optional. Remove run(s) older than the specified time limit. Specify a string in #d#h#m#s format. Float values are also supported. For example: –older-than 1d2h3m4s, –older-than 1.2d3h4m5s    --backend-store-uri <PATH>  URI of the backend store from which to delete runs. Acceptable URIs are SQLAlchemy-compatible database connection strings (e.g. 'sqlite:///path/to/file.db') or local filesystem URIs (e.g. 'file:///absolute/path/to/directory'). By default, data will be deleted from the ./mlruns directory.    --run-ids <run_ids>  Optional comma separated list of runs to be permanently deleted. If run ids are not specified, data is removed for all runs in the deleted lifecycle stage.    --experiment-ids <experiment_ids>  Optional comma separated list of experiments to be permanently deleted including all of their associated runs. If experiment ids are not specified, data is removed for all experiments in the deleted lifecycle stage.    models  Deploy MLflow models locally. To deploy a model associated with a run on a tracking server, set the MLFLOW_TRACKING_URI environment variable to the URL of the desired server. mlflow models [OPTIONS] COMMAND [ARGS]...    build-docker  Builds a Docker image whose default entrypoint serves an MLflow model at port 8080, using the python_function flavor. The container serves the model referenced by --model-uri, if specified when build-docker is called. If --model-uri is not specified when build_docker is called, an MLflow Model directory must be mounted as a volume into the /opt/ml/model directory in the container. Building a Docker image with --model-uri: # Build a Docker image named 'my-image-name' that serves the model from run 'some-run-uuid' # at run-relative artifact path 'my-model' mlflow models build-docker --model-uri ""runs:/some-run-uuid/my-model"" --name ""my-image-name"" # Serve the model docker run -p 5001:8080 ""my-image-name""   Building a Docker image without --model-uri: # Build a generic Docker image named 'my-image-name' mlflow models build-docker --name ""my-image-name"" # Mount the model stored in '/local/path/to/artifacts/model' and serve it docker run --rm -p 5001:8080 -v /local/path/to/artifacts/model:/opt/ml/model ""my-image-name""    Warning The image built without --model-uri doesn't support serving models with RFunc / Java MLeap model server.  NB: by default, the container will start nginx and gunicorn processes. If you don't need the nginx process to be started (for instance if you deploy your container to Google Cloud Run), you can disable it via the DISABLE_NGINX environment variable: docker run -p 5001:8080 -e DISABLE_NGINX=true ""my-image-name""   See https://www.mlflow.org/docs/latest/python_api/mlflow.pyfunc.html for more information on the 'python_function' flavor. mlflow models build-docker [OPTIONS]   Options   -m, --model-uri <URI>  [Optional] URI to the model. A local path, a 'runs:/' URI, or a remote storage URI (e.g., an 's3://' URI). For more information about supported remote URIs for model artifacts, see https://mlflow.org/docs/latest/tracking.html#artifact-stores    -n, --name <name>  Name to use for built image    --env-manager <env_manager>  If specified, create an environment for MLmodel using the specified environment manager. The following values are supported:  - local: use the local environment - virtualenv: use virtualenv (and pyenv for Python version management) - conda: use conda  If unspecified, default to virtualenv.    --mlflow-home <PATH>  Path to local clone of MLflow project. Use for development only.    --install-mlflow  If specified and there is a conda or virtualenv environment to be activated mlflow will be installed into the environment after it has been activated. The version of installed mlflow will be the same as the one used to invoke this command.    --enable-mlserver  Enable serving with MLServer through the v2 inference protocol. You can use environment variables to configure MLServer. (See https://mlserver.readthedocs.io/en/latest/reference/settings.html)    generate-dockerfile  Generates a directory with Dockerfile whose default entrypoint serves an MLflow model at port 8080 using the python_function flavor. The generated Dockerfile is written to the specified output directory, along with the model (if specified). This Dockerfile defines an image that is equivalent to the one produced by mlflow models build-docker. mlflow models generate-dockerfile [OPTIONS]   Options   -m, --model-uri <URI>  [Optional] URI to the model. A local path, a 'runs:/' URI, or a remote storage URI (e.g., an 's3://' URI). For more information about supported remote URIs for model artifacts, see https://mlflow.org/docs/latest/tracking.html#artifact-stores    -d, --output-directory <output_directory>  Output directory where the generated Dockerfile is stored.    --env-manager <env_manager>  If specified, create an environment for MLmodel using the specified environment manager. The following values are supported:  - local: use the local environment - virtualenv: use virtualenv (and pyenv for Python version management) - conda: use conda  If unspecified, default to virtualenv.    --mlflow-home <PATH>  Path to local clone of MLflow project. Use for development only.    --install-mlflow  If specified and there is a conda or virtualenv environment to be activated mlflow will be installed into the environment after it has been activated. The version of installed mlflow will be the same as the one used to invoke this command.    --enable-mlserver  Enable serving with MLServer through the v2 inference protocol. You can use environment variables to configure MLServer. (See https://mlserver.readthedocs.io/en/latest/reference/settings.html)    predict  Generate predictions in json format using a saved MLflow model. For information about the input data formats accepted by this function, see the following documentation: https://www.mlflow.org/docs/latest/models.html#built-in-deployment-tools. mlflow models predict [OPTIONS]   Options   -m, --model-uri <URI>  Required URI to the model. A local path, a 'runs:/' URI, or a remote storage URI (e.g., an 's3://' URI). For more information about supported remote URIs for model artifacts, see https://mlflow.org/docs/latest/tracking.html#artifact-stores    -i, --input-path <input_path>  CSV containing pandas DataFrame to predict against.    -o, --output-path <output_path>  File to output results to as json file. If not provided, output to stdout.    -t, --content-type <content_type>  Content type of the input file. Can be one of {'json', 'csv'}.    --env-manager <env_manager>  If specified, create an environment for MLmodel using the specified environment manager. The following values are supported:  - local: use the local environment - virtualenv: use virtualenv (and pyenv for Python version management) - conda: use conda  If unspecified, default to virtualenv.    --install-mlflow  If specified and there is a conda or virtualenv environment to be activated mlflow will be installed into the environment after it has been activated. The version of installed mlflow will be the same as the one used to invoke this command.    prepare-env  Performs any preparation necessary to predict or serve the model, for example downloading dependencies or initializing a conda environment. After preparation, calling predict or serve should be fast. mlflow models prepare-env [OPTIONS]   Options   -m, --model-uri <URI>  Required URI to the model. A local path, a 'runs:/' URI, or a remote storage URI (e.g., an 's3://' URI). For more information about supported remote URIs for model artifacts, see https://mlflow.org/docs/latest/tracking.html#artifact-stores    --env-manager <env_manager>  If specified, create an environment for MLmodel using the specified environment manager. The following values are supported:  - local: use the local environment - virtualenv: use virtualenv (and pyenv for Python version management) - conda: use conda  If unspecified, default to virtualenv.    --install-mlflow  If specified and there is a conda or virtualenv environment to be activated mlflow will be installed into the environment after it has been activated. The version of installed mlflow will be the same as the one used to invoke this command.    serve  Serve a model saved with MLflow by launching a webserver on the specified host and port. The command supports models with the python_function or crate (R Function) flavor. For information about the input data formats accepted by the webserver, see the following documentation: https://www.mlflow.org/docs/latest/models.html#built-in-deployment-tools.  Warning Models built using MLflow 1.x will require adjustments to the endpoint request payload if executed in an environment that has MLflow 2.x installed. In 1.x, a request payload was in the format: {'columns': [str], 'data': [[...]]}. 2.x models require payloads that are defined by the structural-defining keys of either dataframe_split, instances, inputs or dataframe_records. See the examples below for demonstrations of the changes to the invocation API endpoint in 2.0.   Note Requests made in pandas DataFrame structures can be made in either split or records oriented formats. See https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.to_json.html for detailed information on orientation formats for converting a pandas DataFrame to json.  Example: $ mlflow models serve -m runs:/my-run-id/model-path &  # records orientation input format for serializing a pandas DataFrame $ curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '{     ""dataframe_records"": [{""a"":1, ""b"":2}, {""a"":3, ""b"":4}, {""a"":5, ""b"":6}] }'  # split orientation input format for serializing a pandas DataFrame $ curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '{     ""dataframe_split"": {""columns"": [""a"", ""b""],                         ""index"": [0, 1, 2],                         ""data"": [[1, 2], [3, 4], [5, 6]]} }'  # inputs format for List submission of array, tensor, or DataFrame data $ curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '{     ""inputs"": [[1, 2], [3, 4], [5, 6]] }'  # instances format for submission of Tensor data curl http://127.0.0.1:5000/invocations -H 'Content-Type: application/json' -d '{     ""instances"": [         {""a"": ""t1"", ""b"": [1, 2, 3]},         {""a"": ""t2"", ""b"": [4, 5, 6]},         {""a"": ""t3"", ""b"": [7, 8, 9]}     ] }'   mlflow models serve [OPTIONS]   Options   -m, --model-uri <URI>  Required URI to the model. A local path, a 'runs:/' URI, or a remote storage URI (e.g., an 's3://' URI). For more information about supported remote URIs for model artifacts, see https://mlflow.org/docs/latest/tracking.html#artifact-stores    -p, --port <port>  The port to listen on (default: 5000).    -h, --host <HOST>  The network address to listen on (default: 127.0.0.1). Use 0.0.0.0 to bind to all addresses if you want to access the tracking server from other machines.    -t, --timeout <timeout>  Timeout in seconds to serve a request (default: 60).    -w, --workers <workers>  Number of gunicorn worker processes to handle requests (default: 1).    --env-manager <env_manager>  If specified, create an environment for MLmodel using the specified environment manager. The following values are supported:  - local: use the local environment - virtualenv: use virtualenv (and pyenv for Python version management) - conda: use conda  If unspecified, default to virtualenv.    --no-conda  If specified, use local environment.    --install-mlflow  If specified and there is a conda or virtualenv environment to be activated mlflow will be installed into the environment after it has been activated. The version of installed mlflow will be the same as the one used to invoke this command.    --enable-mlserver  Enable serving with MLServer through the v2 inference protocol. You can use environment variables to configure MLServer. (See https://mlserver.readthedocs.io/en/latest/reference/settings.html)  Environment variables   MLFLOW_PORT  Provide a default for --port     MLFLOW_HOST  Provide a default for --host     MLFLOW_SCORING_SERVER_REQUEST_TIMEOUT  Provide a default for --timeout     MLFLOW_WORKERS  Provide a default for --workers      recipes  Run MLflow Recipes and inspect recipe results. mlflow recipes [OPTIONS] COMMAND [ARGS]...    clean  Remove all recipe outputs from the cache, or remove the cached outputs of a particular recipe step if specified. After cached outputs are cleaned for a particular step, the step will be re-executed in its entirety the next time it is run. mlflow recipes clean [OPTIONS]   Options   -s, --step <step>  The name of the recipe step for which to remove cached outputs.    -p, --profile <profile>  Required The name of the recipe profile to use. Profiles customize the configuration of one or more recipe steps, and recipe executions with different profiles often produce different results.  Environment variables   MLFLOW_RECIPES_PROFILE  Provide a default for --profile     get-artifact  Get the location of an artifact output from the recipe. mlflow recipes get-artifact [OPTIONS]   Options   -a, --artifact <artifact>  Required The name of the artifact to retrieve.    -p, --profile <profile>  Required The name of the recipe profile to use. Profiles customize the configuration of one or more recipe steps, and recipe executions with different profiles often produce different results.  Environment variables   MLFLOW_RECIPES_PROFILE  Provide a default for --profile     inspect  Display a visual overview of the recipe graph, or display a summary of results from a particular recipe step if specified. If the specified step has not been executed, nothing is displayed. mlflow recipes inspect [OPTIONS]   Options   -s, --step <step>  The name of the recipe step to inspect.    -p, --profile <profile>  Required The name of the recipe profile to use. Profiles customize the configuration of one or more recipe steps, and recipe executions with different profiles often produce different results.  Environment variables   MLFLOW_RECIPES_PROFILE  Provide a default for --profile     run  Run the full recipe, or run a particular recipe step if specified, producing outputs and displaying a summary of results upon completion. mlflow recipes run [OPTIONS]   Options   -s, --step <step>  The name of the recipe step to run.    -p, --profile <profile>  Required The name of the recipe profile to use. Profiles customize the configuration of one or more recipe steps, and recipe executions with different profiles often produce different results.  Environment variables   MLFLOW_RECIPES_PROFILE  Provide a default for --profile      run  Run an MLflow project from the given URI. For local runs, the run will block until it completes. Otherwise, the project will run asynchronously. If running locally (the default), the URI can be either a Git repository URI or a local path. If running on Databricks, the URI must be a Git repository. By default, Git projects run in a new working directory with the given parameters, while local projects run from the project's root directory. mlflow run [OPTIONS] URI   Options   -e, --entry-point <NAME>  Entry point within project. [default: main]. If the entry point is not found, attempts to run the project file with the specified name as a script, using 'python' to run .py files and the default shell (specified by environment variable $SHELL) to run .sh files    -v, --version <VERSION>  Version of the project to run, as a Git commit reference for Git projects.    -P, --param-list <NAME=VALUE>  A parameter for the run, of the form -P name=value. Provided parameters that are not in the list of parameters for an entry point will be passed to the corresponding entry point as command-line arguments in the form –name value    -A, --docker-args <NAME=VALUE>  A docker run argument or flag, of the form -A name=value (e.g. -A gpus=all) or -A name (e.g. -A t). The argument will then be passed as docker run –name value or docker run –name respectively.    --experiment-name <experiment_name>  Name of the experiment under which to launch the run. If not specified, 'experiment-id' option will be used to launch run.    --experiment-id <experiment_id>  ID of the experiment under which to launch the run.    -b, --backend <BACKEND>  Execution backend to use for run. Supported values: 'local', 'databricks', kubernetes (experimental). Defaults to 'local'. If running against Databricks, will run against a Databricks workspace determined as follows: if a Databricks tracking URI of the form 'databricks://profile' has been set (e.g. by setting the MLFLOW_TRACKING_URI environment variable), will run against the workspace specified by <profile>. Otherwise, runs against the workspace specified by the default Databricks CLI profile. See https://github.com/databricks/databricks-cli for more info on configuring a Databricks CLI profile.    -c, --backend-config <FILE>  Path to JSON file (must end in '.json') or JSON string which will be passed as config to the backend. The exact content which should be provided is different for each execution backend and is documented at https://www.mlflow.org/docs/latest/projects.html.    --env-manager <env_manager>  If specified, create an environment for MLproject using the specified environment manager. The following values are supported:  - local: use the local environment - virtualenv: use virtualenv (and pyenv for Python version management) - conda: use conda  If unspecified, the appropriate environment manager is automatically selected based on the project configuration. For example, if MLproject.yaml contains a python_env key, virtualenv is used.    --storage-dir <storage_dir>  Only valid when backend is local. MLflow downloads artifacts from distributed URIs passed to parameters of type 'path' to subdirectories of storage_dir.    --run-id <RUN_ID>  If specified, the given run ID will be used instead of creating a new run. Note: this argument is used internally by the MLflow project APIs and should not be specified.    --run-name <RUN_NAME>  The name to give the MLflow Run associated with the project execution. If not specified, the MLflow Run name is left unset.    --build-image  Only valid for Docker projects. If specified, build a new Docker image that's based on the image specified by the image field in the MLproject file, and contains files in the project directory.  Default False    Arguments   URI  Required argument  Environment variables   MLFLOW_EXPERIMENT_NAME  Provide a default for --experiment-name     MLFLOW_EXPERIMENT_ID  Provide a default for --experiment-id     MLFLOW_TMP_DIR  Provide a default for --storage-dir     runs  Manage runs. To manage runs of experiments associated with a tracking server, set the MLFLOW_TRACKING_URI environment variable to the URL of the desired server. mlflow runs [OPTIONS] COMMAND [ARGS]...    delete  Mark a run for deletion. Return an error if the run does not exist or is already marked. You can restore a marked run with restore_run, or permanently delete a run in the backend store. mlflow runs delete [OPTIONS]   Options   --run-id <run_id>  Required    describe  All of run details will print to the stdout as JSON format. mlflow runs describe [OPTIONS]   Options   --run-id <run_id>  Required    list  List all runs of the specified experiment in the configured tracking server. mlflow runs list [OPTIONS]   Options   --experiment-id <experiment_id>  Required Specify the experiment ID for list of runs.    -v, --view <view>  Select view type for list experiments. Valid view types are 'active_only' (default), 'deleted_only', and 'all'.  Environment variables   MLFLOW_EXPERIMENT_ID  Provide a default for --experiment-id     restore  Restore a deleted run. Returns an error if the run is active or has been permanently deleted. mlflow runs restore [OPTIONS]   Options   --run-id <run_id>  Required     sagemaker  Serve models on SageMaker. To serve a model associated with a run on a tracking server, set the MLFLOW_TRACKING_URI environment variable to the URL of the desired server. mlflow sagemaker [OPTIONS] COMMAND [ARGS]...    build-and-push-container  Build new MLflow Sagemaker image, assign it a name, and push to ECR. This function builds an MLflow Docker image. The image is built locally and it requires Docker to run. The image is pushed to ECR under current active AWS account and to current active AWS region. mlflow sagemaker build-and-push-container [OPTIONS]   Options   --build, --no-build  Build the container if set.    --push, --no-push  Push the container to AWS ECR if set.    -c, --container <container>  image name    --env-manager <env_manager>  If specified, create an environment for MLmodel using the specified environment manager. The following values are supported:  - local: use the local environment - virtualenv: use virtualenv (and pyenv for Python version management) - conda: use conda  If unspecified, default to virtualenv.    --mlflow-home <PATH>  Path to local clone of MLflow project. Use for development only.    deploy-transform-job  Deploy model on Sagemaker as a batch transform job. Current active AWS account needs to have correct permissions setup. By default, unless the --async flag is specified, this command will block until either the batch transform job completes (definitively succeeds or fails) or the specified timeout elapses. mlflow sagemaker deploy-transform-job [OPTIONS]   Options   -n, --job-name <job_name>  Required Transform job name    -m, --model-uri <URI>  Required URI to the model. A local path, a 'runs:/' URI, or a remote storage URI (e.g., an 's3://' URI). For more information about supported remote URIs for model artifacts, see https://mlflow.org/docs/latest/tracking.html#artifact-stores    --input-data-type <input_data_type>  Required Input data type for the transform job    -u, --input-uri <input_uri>  Required S3 key name prefix or manifest of the input data    --content-type <content_type>  Required The multipurpose internet mail extension (MIME) type of the data    -o, --output-path <output_path>  Required The S3 path to store the output results of the Sagemaker transform job    --compression-type <compression_type>  The compression type of the transform data    -s, --split-type <split_type>  The method to split the transform job's data files into smaller batches    -a, --accept <accept>  The multipurpose internet mail extension (MIME) type of the output data    --assemble-with <assemble_with>  The method to assemble the results of the transform job as a single S3 object    --input-filter <input_filter>  A JSONPath expression used to select a portion of the input data for the transform job    --output-filter <output_filter>  A JSONPath expression used to select a portion of the output data from the transform job    -j, --join-resource <join_resource>  The source of the data to join with the transformed data    -e, --execution-role-arn <execution_role_arn>  SageMaker execution role    -b, --bucket <bucket>  S3 bucket to store model artifacts    -i, --image-url <image_url>  ECR URL for the Docker image    --region-name <region_name>  Name of the AWS region in which to deploy the transform job    -t, --instance-type <instance_type>  The type of SageMaker ML instance on which to perform the batch transform job. For a list of supported instance types, see https://aws.amazon.com/sagemaker/pricing/instance-types/.    -c, --instance-count <instance_count>  The number of SageMaker ML instances on which to perform the batch transform job    -v, --vpc-config <vpc_config>  Path to a file containing a JSON-formatted VPC configuration. This configuration will be used when creating the new SageMaker model associated with this application. For more information, see https://docs.aws.amazon.com/sagemaker/latest/dg/API_VpcConfig.html    -f, --flavor <flavor>  The name of the flavor to use for deployment. Must be one of the following: ['python_function', 'mleap']. If unspecified, a flavor will be automatically selected from the model's available flavors.    --archive  If specified, any SageMaker resources that become inactive after the finished batch transform job are preserved. These resources may include the associated SageMaker models and model artifacts. Otherwise, if –archive is unspecified, these resources are deleted. –archive must be specified when deploying asynchronously with –async.    --async  If specified, this command will return immediately after starting the deployment process. It will not wait for the deployment process to complete. The caller is responsible for monitoring the deployment process via native SageMaker APIs or the AWS console.    --timeout <timeout>  If the command is executed synchronously, the deployment process will return after the specified number of seconds if no definitive result (success or failure) is achieved. Once the function returns, the caller is responsible for monitoring the health and status of the pending deployment via native SageMaker APIs or the AWS console. If the command is executed asynchronously using the –async flag, this value is ignored.    push-model  Push an MLflow model to Sagemaker model registry. Current active AWS account needs to have correct permissions setup. mlflow sagemaker push-model [OPTIONS]   Options   -n, --model-name <model_name>  Required Sagemaker model name    -m, --model-uri <URI>  Required URI to the model. A local path, a 'runs:/' URI, or a remote storage URI (e.g., an 's3://' URI). For more information about supported remote URIs for model artifacts, see https://mlflow.org/docs/latest/tracking.html#artifact-stores    -e, --execution-role-arn <execution_role_arn>  SageMaker execution role    -b, --bucket <bucket>  S3 bucket to store model artifacts    -i, --image-url <image_url>  ECR URL for the Docker image    --region-name <region_name>  Name of the AWS region in which to push the Sagemaker model    -v, --vpc-config <vpc_config>  Path to a file containing a JSON-formatted VPC configuration. This configuration will be used when creating the new SageMaker model. For more information, see https://docs.aws.amazon.com/sagemaker/latest/dg/API_VpcConfig.html    -f, --flavor <flavor>  The name of the flavor to use for deployment. Must be one of the following: ['python_function', 'mleap']. If unspecified, a flavor will be automatically selected from the model's available flavors.    terminate-transform-job  Terminate the specified Sagemaker batch transform job. Unless --archive is specified, all SageMaker resources associated with the batch transform job are deleted as well. By default, unless the --async flag is specified, this command will block until either the termination process completes (definitively succeeds or fails) or the specified timeout elapses. mlflow sagemaker terminate-transform-job [OPTIONS]   Options   -n, --job-name <job_name>  Required Transform job name    -r, --region-name <region_name>  Name of the AWS region in which the transform job is deployed    --archive  If specified, resources associated with the application are preserved. These resources may include unused SageMaker models and model artifacts. Otherwise, if –archive is unspecified, these resources are deleted. –archive must be specified when deleting asynchronously with –async.    --async  If specified, this command will return immediately after starting the termination process. It will not wait for the termination process to complete. The caller is responsible for monitoring the termination process via native SageMaker APIs or the AWS console.    --timeout <timeout>  If the command is executed synchronously, the termination process will return after the specified number of seconds if no definitive result (success or failure) is achieved. Once the function returns, the caller is responsible for monitoring the health and status of the pending termination via native SageMaker APIs or the AWS console. If the command is executed asynchronously using the –async flag, this value is ignored.     server  Run the MLflow tracking server. The server listens on http://localhost:5000 by default and only accepts connections from the local machine. To let the server accept connections from other machines, you will need to pass --host 0.0.0.0 to listen on all network interfaces (or a specific interface address). mlflow server [OPTIONS]   Options   --backend-store-uri <PATH>  URI to which to persist experiment and run data. Acceptable URIs are SQLAlchemy-compatible database connection strings (e.g. 'sqlite:///path/to/file.db') or local filesystem URIs (e.g. 'file:///absolute/path/to/directory'). By default, data will be logged to the ./mlruns directory.    --registry-store-uri <URI>  URI to which to persist registered models. Acceptable URIs are SQLAlchemy-compatible database connection strings (e.g. 'sqlite:///path/to/file.db'). If not specified, backend-store-uri is used.    --default-artifact-root <URI>  Directory in which to store artifacts for any new experiments created. For tracking server backends that rely on SQL, this option is required in order to store artifacts. Note that this flag does not impact already-created experiments with any previous configuration of an MLflow server instance. By default, data will be logged to the mlflow-artifacts:/ uri proxy if the –serve-artifacts option is enabled. Otherwise, the default location will be ./mlruns.    --serve-artifacts, --no-serve-artifacts  Enables serving of artifact uploads, downloads, and list requests by routing these requests to the storage location that is specified by '–artifacts-destination' directly through a proxy. The default location that these requests are served from is a local './mlartifacts' directory which can be overridden via the '–artifacts-destination' argument. To disable artifact serving, specify –no-serve-artifacts. Default: True    --artifacts-only  If specified, configures the mlflow server to be used only for proxied artifact serving. With this mode enabled, functionality of the mlflow tracking service (e.g. run creation, metric logging, and parameter logging) is disabled. The server will only expose endpoints for uploading, downloading, and listing artifacts. Default: False    --artifacts-destination <URI>  The base artifact location from which to resolve artifact upload/download/list requests (e.g. 's3://my-bucket'). Defaults to a local './mlartifacts' directory. This option only applies when the tracking server is configured to stream artifacts and the experiment's artifact root location is http or mlflow-artifacts URI.    -h, --host <HOST>  The network address to listen on (default: 127.0.0.1). Use 0.0.0.0 to bind to all addresses if you want to access the tracking server from other machines.    -p, --port <port>  The port to listen on (default: 5000).    -w, --workers <workers>  Number of gunicorn worker processes to handle requests (default: 1).    --static-prefix <static_prefix>  A prefix which will be prepended to the path of all static paths.    --gunicorn-opts <gunicorn_opts>  Additional command line options forwarded to gunicorn processes.    --waitress-opts <waitress_opts>  Additional command line options for waitress-serve.    --expose-prometheus <expose_prometheus>  Path to the directory where metrics will be stored. If the directory doesn't exist, it will be created. Activate prometheus exporter to expose metrics on /metrics endpoint.    --app-name <app_name>  Application name to be used for the tracking server. If not specified, 'mlflow.server:app' will be used.  Options basic-auth | basic-auth      --dev  If enabled, run the server with debug logging and auto-reload. Should only be used for development purposes. Cannot be used with '–gunicorn-opts'. Unsupported on Windows.  Default False    Environment variables   MLFLOW_BACKEND_STORE_URI  Provide a default for --backend-store-uri     MLFLOW_REGISTRY_STORE_URI  Provide a default for --registry-store-uri     MLFLOW_DEFAULT_ARTIFACT_ROOT  Provide a default for --default-artifact-root     MLFLOW_SERVE_ARTIFACTS  Provide a default for --serve-artifacts     MLFLOW_ARTIFACTS_ONLY  Provide a default for --artifacts-only     MLFLOW_ARTIFACTS_DESTINATION  Provide a default for --artifacts-destination     MLFLOW_HOST  Provide a default for --host     MLFLOW_PORT  Provide a default for --port     MLFLOW_WORKERS  Provide a default for --workers     MLFLOW_STATIC_PREFIX  Provide a default for --static-prefix     MLFLOW_GUNICORN_OPTS  Provide a default for --gunicorn-opts     MLFLOW_EXPOSE_PROMETHEUS  Provide a default for --expose-prometheus           Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
search-runs.html,"   Documentation  Search Runs       Search Runs  The MLflow UI and API support searching runs within a single experiment or a group of experiments using a search filter API. This API is a simplified version of the SQL WHERE clause.  Table of Contents  Syntax  Example Expressions Identifier  Entity Names Containing Special Characters Entity Names Starting with a Number Run Attributes Datasets MLflow Tags   Comparator Constant   Programmatically Searching Runs  Python R Java      Syntax  A search filter is one or more expressions joined by the AND keyword. The syntax does not support OR. Each expression has three parts: an identifier on the left-hand side (LHS), a comparator, and constant on the right-hand side (RHS).  Example Expressions   Search for the subset of runs with logged accuracy metric greater than 0.92. metrics.accuracy > 0.92    Search for all completed runs. attributes.status = ""FINISHED""    Search for all failed runs. attributes.status = ""FAILED""    Search for runs created after UNIX timestamp 1670628787527. attributes.created > 1670628787527 attributes.Created > 1670628787527 attributes.start_time > 1670628787527    Search for the subset of runs with F1 score greater than 0.5. metrics.`f1 score` > 0.5    Search for runs created by user 'john@mlflow.com'. tags.`mlflow.user` = 'john@mlflow.com'    Search for runs with models trained using scikit-learn (assumes runs have a tag called model whose value starts with sklearn). tags.`model` LIKE 'sklearn%'    Search for runs with logistic regression models, ignoring case (assumes runs have a tag called type whose value contains logistic). tags.`type` ILIKE '%Logistic%'    Search for runs whose names contain alpha. attributes.`run_name` ILIKE ""%alpha%"" attributes.`run name` ILIKE ""%alpha%"" attributes.`Run name` ILIKE ""%alpha%"" attributes.`Run Name` ILIKE ""%alpha%""    Search for runs created using a Logistic Regression model, a learning rate (lambda) of 0.001, and recorded error metric under 0.05. params.alpha = ""0.3"" and params.lambda = ""0.001"" and metrics.error <= 0.05       Identifier  Required in the LHS of a search expression. Signifies an entity to compare against. An identifier has two parts separated by a period: the type of the entity and the name of the entity. The type of the entity is metrics, params, attributes, datasets, or tags. The entity name can contain alphanumeric characters and special characters. This section describes supported entity names and how to specify such names in search expressions.  In this section:  Entity Names Containing Special Characters Entity Names Starting with a Number Run Attributes Datasets MLflow Tags    Entity Names Containing Special Characters  When a metric, parameter, or tag name contains a special character like hyphen, space, period, and so on, enclose the entity name in double quotes or backticks. Examples params.""model-type""   metrics.`error rate`     Entity Names Starting with a Number  Unlike SQL syntax for column names, MLflow allows logging metrics, parameters, and tags names that have a leading number. If an entity name contains a leading number, enclose the entity name in double quotes. For example: metrics.""2019-04-02 error rate""     Run Attributes  You can search using the following run attributes contained in mlflow.entities.RunInfo: run_id, run_name, status, artifact_uri, user_id, start_time and end_time. The run_id, run_name, status, user_id and artifact_uri attributes have string values, while start_time and end_time are numeric. Other fields in mlflow.entities.RunInfo are not searchable. Run name, Run Name and run name are aliases for run_name. created and Created are aliases for start_time.  Note  The experiment ID is implicitly selected by the search API. A run's lifecycle_stage attribute is not allowed because it is already encoded as a part of the API's run_view_type field. To search for runs using run_id, it is more efficient to use get_run APIs.   Example attributes.artifact_uri = 'models:/mymodel/1' attributes.status = 'ACTIVE' # RHS value for start_time and end_time are unix timestamp attributes.start_time >= 1664067852747 attributes.end_time < 1664067852747 attributes.user_id = 'user1' attributes.run_name = 'my-run' attributes.run_id = 'a1b2c3d4' attributes.run_id IN ('a1b2c3d4', 'e5f6g7h8')     Datasets  You can search using the following dataset attributes contained in mlflow.entities.Dataset: name, digest. Additionally, you may search for a specific mlflow.entities.InputTag: with key mlflow.data.context under the alias context. All dataset attributes are string values. Other fields in mlflow.entities.Dataset are not searchable. Example datasets.name = 'mydataset' datasets.digest = 's8ds293b' datasets.digest IN ('s8ds293b', 'jks834s2') datasets.context = 'train'     MLflow Tags  You can search for MLflow tags by enclosing the tag name in double quotes or backticks. For example, to search by owner of an MLflow run, specify tags.""mlflow.user"" or tags.`mlflow.user`. Examples tags.""mlflow.user""   tags.`mlflow.parentRunId`      Comparator  There are two classes of comparators: numeric and string.  Numeric comparators (metrics): =, !=, >, >=, <, and <=. String comparators (params, tags, and attributes): =, !=, LIKE and ILIKE.    Constant  The search syntax requires the RHS of the expression to be a constant. The type of the constant depends on LHS.  If LHS is a metric, the RHS must be an integer or float number. If LHS is a parameter or tag, the RHS must be a string constant enclosed in single or double quotes.     Programmatically Searching Runs  The MLflow UI supports searching runs contained within the current experiment. To search runs across multiple experiments, use one of the client APIs.  Python  Use the MlflowClient.search_runs() or mlflow.search_runs() API to search programmatically. You can specify the list of columns to order by (for example, "metrics.rmse") in the order_by column. The column can contain an optional DESC or ASC value; the default is ASC. The default ordering is to sort by start_time DESC, then run_id. The mlflow.search_runs() API can be used to search for runs within specific experiments which can be identified by experiment IDs or experiment names, but not both at the same time.  Warning Using both experiment_ids and experiment_names in the same call will result in error unless one of them is None or []  For example, if you'd like to identify the best active run from experiment ID 0 by accuracy, use: from mlflow import MlflowClient from mlflow.entities import ViewType  run = MlflowClient().search_runs(     experiment_ids=""0"",     filter_string="""",     run_view_type=ViewType.ACTIVE_ONLY,     max_results=1,     order_by=[""metrics.accuracy DESC""], )[0]   To get all active runs from experiments IDs 3, 4, and 17 that used a CNN model with 10 layers and had a prediction accuracy of 94.5% or higher, use: from mlflow import MlflowClient from mlflow.entities import ViewType  query = ""params.model = 'CNN' and params.layers = '10' and metrics.`prediction accuracy` >= 0.945"" runs = MlflowClient().search_runs(     experiment_ids=[""3"", ""4"", ""17""],     filter_string=query,     run_view_type=ViewType.ACTIVE_ONLY, )   To search all known experiments for any MLflow runs created using the Inception model architecture: import mlflow from mlflow.entities import ViewType  all_experiments = [exp.experiment_id for exp in mlflow.search_experiments()] runs = mlflow.search_runs(     experiment_ids=all_experiments,     filter_string=""params.model = 'Inception'"",     run_view_type=ViewType.ALL, )   To get all runs from the experiment named "Social NLP Experiments", use: import mlflow  runs = mlflow.search_runs(experiment_names=[""Social NLP Experiments""])     R  The R API is similar to the Python API. library(mlflow) mlflow_search_runs(   filter = ""metrics.rmse < 0.9 and tags.production = 'true'"",   experiment_ids = as.character(1:2),   order_by = ""params.lr DESC"" )     Java  The Java API is similar to Python API. List<Long> experimentIds = Arrays.asList(""1"", ""2"", ""4"", ""8""); List<RunInfo> searchResult = client.searchRuns(experimentIds, ""metrics.accuracy_score < 99.90"");           Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
search-experiments.html,"   Documentation  Search Experiments       Search Experiments  mlflow.search_experiments() and MlflowClient.search_experiments() support the same filter string syntax as mlflow.search_runs() and MlflowClient.search_runs(), but the supported identifiers and comparators are different.  Table of Contents  Syntax  Identifier Comparator Examples      Syntax  See Search Runs Syntax for more information.  Identifier  The following identifiers are supported:  attributes.name: Experiment name attributes.creation_time: Experiment creation time attributes.last_update_time: Experiment last update time   Note attributes can be omitted. name is equivalent to attributes.name.    tags.<tag key>: Tag    Comparator  Comparators for string attributes and tags:  =: Equal !=: Not equal LIKE: Case-sensitive pattern match ILIKE: Case-insensitive pattern match  Comparators for numeric attributes:  =: Equal !=: Not equal <: Less than <=: Less than or equal to >: Greater than >=: Greater than or equal to    Examples  # Matches experiments with name equal to 'x' ""attributes.name = 'x'""  # or ""name = 'x'""  # Matches experiments with name starting with 'x' ""attributes.name LIKE 'x%'""  # Matches experiments with 'group' tag value not equal to 'x' ""tags.group != 'x'""  # Matches experiments with 'group' tag value containing 'x' or 'X' ""tags.group ILIKE '%x%'""  # Matches experiments with name starting with 'x' and 'group' tag value equal to 'y' ""attributes.name LIKE 'x%' AND tags.group = 'y'""           Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
python_api/index.html,"   Documentation  Python API       Python API  The MLflow Python API is organized into the following modules. The most common functions are exposed in the mlflow module, so we recommend starting there.   mlflow mlflow.artifacts mlflow.catboost mlflow.client mlflow.data mlflow.deployments mlflow.diviner mlflow.entities mlflow.environment_variables mlflow.fastai mlflow.gateway mlflow.gluon mlflow.h2o mlflow.johnsnowlabs mlflow.keras_core mlflow.langchain mlflow.lightgbm mlflow.metrics mlflow.mleap mlflow.models mlflow.onnx mlflow.paddle mlflow.pmdarima mlflow.projects mlflow.prophet mlflow.pyfunc mlflow.pyspark.ml mlflow.pytorch mlflow.recipes mlflow.sagemaker mlflow.sentence_transformers mlflow.server mlflow.shap mlflow.sklearn mlflow.spacy mlflow.spark mlflow.statsmodels mlflow.system_metrics mlflow.tensorflow mlflow.transformers mlflow.types mlflow.utils mlflow.xgboost mlflow.openai   See also the index of all functions and classes.  Log Levels  MLflow Python APIs log information during execution using the Python Logging API. You can configure the log level for MLflow logs using the following code snippet. Learn more about Python log levels at the Python language logging guide. import logging  logger = logging.getLogger(""mlflow"")  # Set log level to debugging logger.setLevel(logging.DEBUG)          Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
R-api.html,"   Documentation  R API       R API  The MLflow R API allows you to use MLflow Tracking, Projects and Models.  Prerequisites  To use the MLflow R API, you must install the MLflow Python package. pip install mlflow   Installing with an Available Conda Environment example: conda create -n mlflow-env python conda activate mlflow-env pip install mlflow   The above provided commands create a new Conda environment named mlflow-env, specifying the default Python version. It then activates this environment, making it the active working environment. Finally, it installs the MLflow package using pip, ensuring that MLflow is isolated within this environment, allowing for independent Python and package management for MLflow-related tasks. Optionally, you can set the MLFLOW_PYTHON_BIN and MLFLOW_BIN environment variables to specify the Python and MLflow binaries to use. By default, the R client automatically finds them using Sys.which('python') and Sys.which('mlflow'). export MLFLOW_PYTHON_BIN=/path/to/bin/python export MLFLOW_BIN=/path/to/bin/mlflow   You can use the R API to start the user interface, create experiment and search experiments, save models, run projects and serve models among many other functions available in the R API.   build_context_tags_from_databricks_job_info  Get information from a Databricks job execution context Parses the data from a job execution context when running on Databricks in a non-interactive mode. This function extracts relevant data that MLflow needs in order to properly utilize the MLflow APIs from this context. build_context_tags_from_databricks_job_info(job_info)    Arguments        Argument Description    job_info The job-related metadata from a running Databricks job      Value  A list of tags to be set by the run context when creating MLflow runs in the current Databricks Job environment    build_context_tags_from_databricks_notebook_info  Get information from Databricks Notebook environment Retrieves the notebook id, path, url, name, version, and type from the Databricks Notebook execution environment and sets them to a list to be used for setting the configured environment for executing an MLflow run in R from Databricks. build_context_tags_from_databricks_notebook_info(notebook_info)    Arguments        Argument Description    notebook_info The configuration data from the Databricks Notebook environment      Value  A list of tags to be set by the run context when creating MLflow runs in the current Databricks Notebook environment    mlflow_client  Initialize an MLflow Client Initializes and returns an MLflow client that communicates with the tracking server or store at the specified URI. mlflow_client(tracking_uri = NULL)    Arguments        Argument Description    tracking_uri The tracking URI. If not provided, defaults to the service set by mlflow_set_tracking_uri().       mlflow_create_experiment  Create Experiment Creates an MLflow experiment and returns its id. mlflow_create_experiment(   name,   artifact_location = NULL,   client = NULL,   tags = NULL )    Arguments        Argument Description    name The name of the experiment to create.  artifact_location Location where all artifacts for this experiment are stored. If not provided, the remote server will select an appropriate default.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.  tags Experiment tags to set on the experiment upon experiment creation.       mlflow_create_model_version  Create a model version Create a model version mlflow_create_model_version(   name,   source,   run_id = NULL,   tags = NULL,   run_link = NULL,   description = NULL,   client = NULL )    Arguments        Argument Description    name Register model under this name.  source URI indicating the location of the model artifacts.  run_id MLflow run ID for correlation, if source was generated by an experiment run in MLflow Tracking.  tags Additional metadata.  run_link MLflow run link - This is the exact link of the run that generated this model version.  description Description for model version.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_create_registered_model  Create registered model Creates a new registered model in the model registry mlflow_create_registered_model(   name,   tags = NULL,   description = NULL,   client = NULL )    Arguments        Argument Description    name The name of the model to create.  tags Additional metadata for the registered model (Optional).  description Description for the registered model (Optional).  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_delete_experiment  Delete Experiment Marks an experiment and associated runs, params, metrics, etc. for deletion. If the experiment uses FileStore, artifacts associated with experiment are also deleted. mlflow_delete_experiment(experiment_id, client = NULL)    Arguments        Argument Description    experiment_id ID of the associated experiment. This field is required.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_delete_model_version  Delete a model version Delete a model version mlflow_delete_model_version(name, version, client = NULL)    Arguments        Argument Description    name Name of the registered model.  version Model version number.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_delete_registered_model  Delete registered model Deletes an existing registered model by name mlflow_delete_registered_model(name, client = NULL)    Arguments        Argument Description    name The name of the model to delete  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_delete_run  Delete a Run Deletes the run with the specified ID. mlflow_delete_run(run_id, client = NULL)    Arguments        Argument Description    run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_delete_tag  Delete Tag Deletes a tag on a run. This is irreversible. Tags are run metadata that can be updated during a run and after a run completes. mlflow_delete_tag(key, run_id = NULL, client = NULL)    Arguments        Argument Description    key Name of the tag. Maximum size is 255 bytes. This field is required.  run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_download_artifacts  Download Artifacts Download an artifact file or directory from a run to a local directory if applicable, and return a local path for it. mlflow_download_artifacts(path, run_id = NULL, client = NULL)    Arguments        Argument Description    path Relative source path to the desired artifact.  run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_end_run  End a Run Terminates a run. Attempts to end the current active run if run_id is not specified. mlflow_end_run(   status = c(""FINISHED"", ""FAILED"", ""KILLED""),   end_time = NULL,   run_id = NULL,   client = NULL )    Arguments        Argument Description    status Updated status of the run. Defaults to FINISHED. Can also be set to "FAILED" or "KILLED".  end_time Unix timestamp of when the run ended in milliseconds.  run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_get_experiment  Get Experiment Gets metadata for an experiment and a list of runs for the experiment. Attempts to obtain the active experiment if both experiment_id and name are unspecified. mlflow_get_experiment(experiment_id = NULL, name = NULL, client = NULL)    Arguments        Argument Description    experiment_id ID of the experiment.  name The experiment name. Only one of name or experiment_id should be specified.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_get_latest_versions  Get latest model versions Retrieves a list of the latest model versions for a given model. mlflow_get_latest_versions(name, stages = list(), client = NULL)    Arguments        Argument Description    name Name of the model.  stages A list of desired stages. If the input list is NULL, return latest versions for ALL_STAGES.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_get_metric_history  Get Metric History Get a list of all values for the specified metric for a given run. mlflow_get_metric_history(metric_key, run_id = NULL, client = NULL)    Arguments        Argument Description    metric_key Name of the metric.  run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_get_model_version  Get a model version Get a model version mlflow_get_model_version(name, version, client = NULL)    Arguments        Argument Description    name Name of the registered model.  version Model version number.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_get_registered_model  Get a registered model Retrieves a registered model from the Model Registry. mlflow_get_registered_model(name, client = NULL)    Arguments        Argument Description    name The name of the model to retrieve.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_get_run  Get Run Gets metadata, params, tags, and metrics for a run. Returns a single value for each metric key: the most recently logged metric value at the largest step. mlflow_get_run(run_id = NULL, client = NULL)    Arguments        Argument Description    run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_get_tracking_uri  Get Remote Tracking URI Gets the remote tracking URI. mlflow_get_tracking_uri()     mlflow_id  Get Run or Experiment ID Extracts the ID of the run or experiment. mlflow_id(object) list(list(""mlflow_id""), list(""mlflow_run""))(object) list(list(""mlflow_id""), list(""mlflow_experiment""))(object)    Arguments        Argument Description    object An mlflow_run or mlflow_experiment object.       mlflow_list_artifacts  List Artifacts Gets a list of artifacts. mlflow_list_artifacts(path = NULL, run_id = NULL, client = NULL)    Arguments        Argument Description    path The run's relative artifact path to list from. If not specified, it is set to the root artifact path  run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_load_flavor  Load MLflow Model Flavor Loads an MLflow model using a specific flavor. This method is called internally by mlflow_load_model , but is exposed for package authors to extend the supported MLflow models. See https://mlflow.org/docs/latest/models.html#storage-format for more info on MLflow model flavors. mlflow_load_flavor(flavor, model_path)    Arguments        Argument Description    flavor An MLflow flavor object loaded by mlflo w_load_model , with class loaded from the flavor field in an MLmodel file.  model_path The path to the MLflow model wrapped in the correct class.       mlflow_load_model  Load MLflow Model Loads an MLflow model. MLflow models can have multiple model flavors. Not all flavors / models can be loaded in R. This method by default searches for a flavor supported by R/MLflow. mlflow_load_model(model_uri, flavor = NULL, client = mlflow_client())    Arguments        Argument Description    model_uri The location, in URI format, of the MLflow model.  flavor Optional flavor specification (string). Can be used to load a particular flavor in case there are multiple flavors available.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.      Details  The URI scheme must be supported by MLflow - i.e. there has to be an MLflow artifact repository corresponding to the scheme of the URI. The content is expected to point to a directory containing MLmodel. The following are examples of valid model uris:  file:///absolute/path/to/local/model file:relative/path/to/local/model s3://my_bucket/path/to/model runs:/<mlflow_run_id>/run-relative/path/to/model models:/<model_name>/<model_version> models:/<model_name>/<stage>  For more information about supported URI schemes, see the Artifacts Documentation at https://www.mlflow.org/docs/latest/tracking.html#artifact-stores.    mlflow_log_artifact  Log Artifact Logs a specific file or directory as an artifact for a run. mlflow_log_artifact(path, artifact_path = NULL, run_id = NULL, client = NULL)    Arguments        Argument Description    path The file or directory to log as an artifact.  artifact_path Destination path within the run's artifact URI.  run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.      Details  When logging to Amazon S3, ensure that you have the s3:PutObject, s3:GetObject, s3:ListBucket, and s3:GetBucketLocation permissions on your bucket. Additionally, at least the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables must be set to the corresponding key and secrets provided by Amazon IAM.    mlflow_log_batch  Log Batch Log a batch of metrics, params, and/or tags for a run. The server will respond with an error (non-200 status code) if any data failed to be persisted. In case of error (due to internal server error or an invalid request), partial data may be written. mlflow_log_batch(   metrics = NULL,   params = NULL,   tags = NULL,   run_id = NULL,   client = NULL )    Arguments        Argument Description    metrics A dataframe of metrics to log, containing the following columns: "key", "value", "step", "timestamp". This dataframe cannot contain any missing ('NA') entries.  params A dataframe of params to log, containing the following columns: "key", "value". This dataframe cannot contain any missing ('NA') entries.  tags A dataframe of tags to log, containing the following columns: "key", "value". This dataframe cannot contain any missing ('NA') entries.  run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_log_metric  Log Metric Logs a metric for a run. Metrics key-value pair that records a single float measure. During a single execution of a run, a particular metric can be logged several times. The MLflow Backend keeps track of historical metric values along two axes: timestamp and step. mlflow_log_metric(   key,   value,   timestamp = NULL,   step = NULL,   run_id = NULL,   client = NULL )    Arguments        Argument Description    key Name of the metric.  value Float value for the metric being logged.  timestamp Timestamp at which to log the metric. Timestamp is rounded to the nearest integer. If unspecified, the number of milliseconds since the Unix epoch is used.  step Step at which to log the metric. Step is rounded to the nearest integer. If unspecified, the default value of zero is used.  run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_log_model  Log Model Logs a model for this run. Similar to mlflow_save_model() but stores model as an artifact within the active run. mlflow_log_model(model, artifact_path, ...)    Arguments        Argument Description    model The model that will perform a prediction.  artifact_path Destination path where this MLflow compatible model will be saved.  ... Optional additional arguments passed to mlflow_save_model() when persisting the model. For example, conda_env = /path/to/conda.yaml may be passed to specify a conda dependencies file for flavors (e.g. keras) that support conda environments.       mlflow_log_param  Log Parameter Logs a parameter for a run. Examples are params and hyperparams used for ML training, or constant dates and values used in an ETL pipeline. A param is a STRING key-value pair. For a run, a single parameter is allowed to be logged only once. mlflow_log_param(key, value, run_id = NULL, client = NULL)    Arguments        Argument Description    key Name of the parameter.  value String value of the parameter.  run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_param  Read Command-Line Parameter Reads a command-line parameter passed to an MLflow project MLflow allows you to define named, typed input parameters to your R scripts via the mlflow_param API. This is useful for experimentation, e.g. tracking multiple invocations of the same script with different parameters. mlflow_param(name, default = NULL, type = NULL, description = NULL)    Arguments        Argument Description    name The name of the parameter.  default The default value of the parameter.  type Type of this parameter. Required if default is not set. If specified, must be one of "numeric", "integer", or "string".  description Optional description for the parameter.      Examples  # This parametrized script trains a GBM model on the Iris dataset and can be run as an MLflow # project. You can run this script (assuming it's saved at /some/directory/params_example.R) # with custom parameters via: # mlflow_run(entry_point = ""params_example.R"", uri = ""/some/directory"", #   parameters = list(num_trees = 200, learning_rate = 0.1)) install.packages(""gbm"") library(mlflow) library(gbm) # define and read input parameters num_trees <- mlflow_param(name = ""num_trees"", default = 200, type = ""integer"") lr <- mlflow_param(name = ""learning_rate"", default = 0.1, type = ""numeric"") # use params to fit a model ir.adaboost <- gbm(Species ~., data=iris, n.trees=num_trees, shrinkage=lr)      mlflow_predict  Generate Prediction with MLflow Model Performs prediction over a model loaded using mlflow_load_model() , to be used by package authors to extend the supported MLflow models. mlflow_predict(model, data, ...)    Arguments        Argument Description    model The loaded MLflow model flavor.  data A data frame to perform scoring.  ... Optional additional arguments passed to underlying predict methods.       mlflow_register_external_observer  Register an external MLflow observer Registers an external MLflow observer that will receive a register_tracking_event(event_name, data) callback on any model tracking event such as "create_run", "delete_run", or "log_metric". Each observer should have a register_tracking_event(event_name, data) callback accepting a character vector event_name specifying the name of the tracking event, and data containing a list of attributes of the event. The callback should be non-blocking, and ideally should complete instantaneously. Any exception thrown from the callback will be ignored. mlflow_register_external_observer(observer)    Arguments        Argument Description    observer The observer object (see example)      Examples  library(mlflow)  observer <- structure(list()) observer$register_tracking_event <- function(event_name, data) { print(event_name) print(data) } mlflow_register_external_observer(observer)      mlflow_rename_experiment  Rename Experiment Renames an experiment. mlflow_rename_experiment(new_name, experiment_id = NULL, client = NULL)    Arguments        Argument Description    new_name The experiment's name will be changed to this. The new name must be unique.  experiment_id ID of the associated experiment. This field is required.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_rename_registered_model  Rename a registered model Renames a model in the Model Registry. mlflow_rename_registered_model(name, new_name, client = NULL)    Arguments        Argument Description    name The current name of the model.  new_name The new name for the model.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_restore_experiment  Restore Experiment Restores an experiment marked for deletion. This also restores associated metadata, runs, metrics, and params. If experiment uses FileStore, underlying artifacts associated with experiment are also restored. mlflow_restore_experiment(experiment_id, client = NULL)    Arguments        Argument Description    experiment_id ID of the associated experiment. This field is required.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.      Details  Throws RESOURCE_DOES_NOT_EXIST if the experiment was never created or was permanently deleted.    mlflow_restore_run  Restore a Run Restores the run with the specified ID. mlflow_restore_run(run_id, client = NULL)    Arguments        Argument Description    run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_rfunc_serve  Serve an RFunc MLflow Model Serves an RFunc MLflow model as a local REST API server. This interface provides similar functionality to mlflow models serve cli command, however, it can only be used to deploy models that include RFunc flavor. The deployed server supports standard mlflow models interface with /ping and /invocation endpoints. In addition, R function models also support deprecated /predict endpoint for generating predictions. The /predict endpoint will be removed in a future version of mlflow. mlflow_rfunc_serve(   model_uri,   host = ""127.0.0.1"",   port = 8090,   daemonized = FALSE,   browse = !daemonized,   ... )    Arguments        Argument Description    model_uri The location, in URI format, of the MLflow model.  host Address to use to serve model, as a string.  port Port to use to serve model, as numeric.  daemonized Makes httpuv server daemonized so R interactive sessions are not blocked to handle requests. To terminate a daemonized server, call httpuv::stopDaemonizedServer() with the handle returned from this call.  browse Launch browser with serving landing page?  ... Optional arguments passed to mlflow_predict().      Details  The URI scheme must be supported by MLflow - i.e. there has to be an MLflow artifact repository corresponding to the scheme of the URI. The content is expected to point to a directory containing MLmodel. The following are examples of valid model uris:  file:///absolute/path/to/local/model file:relative/path/to/local/model s3://my_bucket/path/to/model runs:/<mlflow_run_id>/run-relative/path/to/model models:/<model_name>/<model_version> models:/<model_name>/<stage>  For more information about supported URI schemes, see the Artifacts Documentation at https://www.mlflow.org/docs/latest/tracking.html#artifact-stores.   Examples  library(mlflow)  # save simple model with constant prediction mlflow_save_model(function(df) 1, ""mlflow_constant"")  # serve an existing model over a web interface mlflow_rfunc_serve(""mlflow_constant"")  # request prediction from server httr::POST(""http://127.0.0.1:8090/predict/"")      mlflow_run  Run an MLflow Project Wrapper for the mlflow run CLI command. See https://www.mlflow.org/docs/latest/cli.html#mlflow-run for more info. mlflow_run(   uri = ""."",   entry_point = NULL,   version = NULL,   parameters = NULL,   experiment_id = NULL,   experiment_name = NULL,   backend = NULL,   backend_config = NULL,   env_manager = NULL,   storage_dir = NULL )    Arguments        Argument Description    uri A directory containing modeling scripts, defaults to the current directory.  entry_point Entry point within project, defaults to main if not specified.  version Version of the project to run, as a Git commit reference for Git projects.  parameters A list of parameters.  experiment_id ID of the experiment under which to launch the run.  experiment_name Name of the experiment under which to launch the run.  backend Execution backend to use for run.  backend_config Path to JSON file which will be passed to the backend. For the Databricks backend, it should describe the cluster to use when launching a run on Databricks.  env_manager If specified, create an environment for the project using the specified environment manager. Available options are 'local', 'virtualenv', and 'conda'.  storage_dir Valid only when backend is local. MLflow downloads artifacts from distributed URIs passed to parameters of type path to subdirectories of storage_dir.      Value  The run associated with this run.   Examples  # This parametrized script trains a GBM model on the Iris dataset and can be run as an MLflow # project. You can run this script (assuming it's saved at /some/directory/params_example.R) # with custom parameters via: # mlflow_run(entry_point = ""params_example.R"", uri = ""/some/directory"", #   parameters = list(num_trees = 200, learning_rate = 0.1)) install.packages(""gbm"") library(mlflow) library(gbm) # define and read input parameters num_trees <- mlflow_param(name = ""num_trees"", default = 200, type = ""integer"") lr <- mlflow_param(name = ""learning_rate"", default = 0.1, type = ""numeric"") # use params to fit a model ir.adaboost <- gbm(Species ~., data=iris, n.trees=num_trees, shrinkage=lr)      mlflow_save_model.crate  Save Model for MLflow Saves model in MLflow format that can later be used for prediction and serving. This method is generic to allow package authors to save custom model types. list(list(""mlflow_save_model""), list(""crate""))(model, path, model_spec = list(), ...) mlflow_save_model(model, path, model_spec = list(), ...) list(list(""mlflow_save_model""), list(""H2OModel""))(model, path, model_spec = list(), conda_env = NULL, ...) list(list(""mlflow_save_model""), list(""keras.engine.training.Model""))(model, path, model_spec = list(), conda_env = NULL, ...) list(list(""mlflow_save_model""), list(""xgb.Booster""))(model, path, model_spec = list(), conda_env = NULL, ...)    Arguments        Argument Description    model The model that will perform a prediction.  path Destination path where this MLflow compatible model will be saved.  model_spec MLflow model config this model flavor is being added to.  ... Optional additional arguments.  conda_env Path to Conda dependencies file.       mlflow_search_experiments  Search Experiments Search for experiments that satisfy specified criteria. mlflow_search_experiments(   filter = NULL,   experiment_view_type = c(""ACTIVE_ONLY"", ""DELETED_ONLY"", ""ALL""),   max_results = 1000,   order_by = list(),   page_token = NULL,   client = NULL )    Arguments        Argument Description    filter A filter expression used to identify specific experiments. The syntax is a subset of SQL which allows only ANDing together binary operations. Examples: "attribute.name = 'MyExperiment'", "tags.problem_type = 'iris_regression'"  experiment_view_type Experiment view type. Only experiments matching this view type are returned.  max_results Maximum number of experiments to retrieve.  order_by List of properties to order by. Example: "attribute.name".  page_token Pagination token to go to the next page based on a previous query.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_search_registered_models  List registered models Retrieves a list of registered models. mlflow_search_registered_models(   filter = NULL,   max_results = 100,   order_by = list(),   page_token = NULL,   client = NULL )    Arguments        Argument Description    filter A filter expression used to identify specific registered models. The syntax is a subset of SQL which allows only ANDing together binary operations. Example: "name = 'my_model_name' and tag.key = 'value1'"  max_results Maximum number of registered models to retrieve.  order_by List of registered model properties to order by. Example: "name".  page_token Pagination token to go to the next page based on a previous query.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_search_runs  Search Runs Search for runs that satisfy expressions. Search expressions can use Metric and Param keys. mlflow_search_runs(   filter = NULL,   run_view_type = c(""ACTIVE_ONLY"", ""DELETED_ONLY"", ""ALL""),   experiment_ids = NULL,   order_by = list(),   client = NULL )    Arguments        Argument Description    filter A filter expression over params, metrics, and tags, allowing returning a subset of runs. The syntax is a subset of SQL which allows only ANDing together binary operations between a param/metric/tag and a constant.  run_view_type Run view type.  experiment_ids List of string experiment IDs (or a single string experiment ID) to search over. Attempts to use active experiment if not specified.  order_by List of properties to order by. Example: "metrics.acc DESC".  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_server  Run MLflow Tracking Server Wrapper for mlflow server. mlflow_server(   file_store = ""mlruns"",   default_artifact_root = NULL,   host = ""127.0.0.1"",   port = 5000,   workers = NULL,   static_prefix = NULL,   serve_artifacts = FALSE )    Arguments        Argument Description    file_store The root of the backing file store for experiment and run data.  default_artifact_root Local or S3 URI to store artifacts in, for newly created experiments.  host The network address to listen on (default: 127.0.0.1).  port The port to listen on (default: 5000).  workers Number of gunicorn worker processes to handle requests (default: 4).  static_prefix A prefix which will be prepended to the path of all static paths.  serve_artifacts A flag specifying whether or not to enable artifact serving (default: FALSE).       mlflow_set_experiment_tag  Set Experiment Tag Sets a tag on an experiment with the specified ID. Tags are experiment metadata that can be updated. mlflow_set_experiment_tag(key, value, experiment_id = NULL, client = NULL)    Arguments        Argument Description    key Name of the tag. All storage backends are guaranteed to support key values up to 250 bytes in size. This field is required.  value String value of the tag being logged. All storage backends are guaranteed to support key values up to 5000 bytes in size. This field is required.  experiment_id ID of the experiment.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_set_experiment  Set Experiment Sets an experiment as the active experiment. Either the name or ID of the experiment can be provided. If the a name is provided but the experiment does not exist, this function creates an experiment with provided name. Returns the ID of the active experiment. mlflow_set_experiment(   experiment_name = NULL,   experiment_id = NULL,   artifact_location = NULL )    Arguments        Argument Description    experiment_name Name of experiment to be activated.  experiment_id ID of experiment to be activated.  artifact_location Location where all artifacts for this experiment are stored. If not provided, the remote server will select an appropriate default.       mlflow_set_model_version_tag  Set Model version tag Set a tag for the model version. When stage is set, tag will be set for latest model version of the stage. Setting both version and stage parameter will result in error. mlflow_set_model_version_tag(   name,   version = NULL,   key = NULL,   value = NULL,   stage = NULL,   client = NULL )    Arguments        Argument Description    name Registered model name.  version Registered model version.  key Tag key to log. key is required.  value Tag value to log. value is required.  stage Registered model stage.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_set_tag  Set Tag Sets a tag on a run. Tags are run metadata that can be updated during a run and after a run completes. mlflow_set_tag(key, value, run_id = NULL, client = NULL)    Arguments        Argument Description    key Name of the tag. Maximum size is 255 bytes. This field is required.  value String value of the tag being logged. Maximum size is 500 bytes. This field is required.  run_id Run ID.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_set_tracking_uri  Set Remote Tracking URI Specifies the URI to the remote MLflow server that will be used to track experiments. mlflow_set_tracking_uri(uri)    Arguments        Argument Description    uri The URI to the remote MLflow server.       mlflow_source  Source a Script with MLflow Params This function should not be used interactively. It is designed to be called via Rscript from the terminal or through the MLflow CLI. mlflow_source(uri)    Arguments        Argument Description    uri Path to an R script, can be a quoted or unquoted string.       mlflow_start_run  Start Run Starts a new run. If client is not provided, this function infers contextual information such as source name and version, and also registers the created run as the active run. If client is provided, no inference is done, and additional arguments such as start_time can be provided. mlflow_start_run(   run_id = NULL,   experiment_id = NULL,   start_time = NULL,   tags = NULL,   client = NULL,   nested = FALSE )    Arguments        Argument Description    run_id If specified, get the run with the specified UUID and log metrics and params under that run. The run's end time is unset and its status is set to running, but the run's other attributes remain unchanged.  experiment_id Used only when run_id is unspecified. ID of the experiment under which to create the current run. If unspecified, the run is created under a new experiment with a randomly generated name.  start_time Unix timestamp of when the run started in milliseconds. Only used when client is specified.  tags Additional metadata for run in key-value pairs. Only used when client is specified.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.  nested Controls whether the run to be started is nested in a parent run. TRUE creates a nest run.      Examples  with(mlflow_start_run(), { mlflow_log_metric(""test"", 10) })      mlflow_transition_model_version_stage  Transition ModelVersion Stage Transition a model version to a different stage. mlflow_transition_model_version_stage(   name,   version,   stage,   archive_existing_versions = FALSE,   client = NULL )    Arguments        Argument Description    name Name of the registered model.  version Model version number.  stage Transition model_version to this stage.  archive_existing_versions (Optional)  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_ui  Run MLflow User Interface Launches the MLflow user interface. mlflow_ui(client, ...)    Arguments        Argument Description    client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.  ... Optional arguments passed to mlflow_server() when x is a path to a file store.      Examples  library(mlflow)  # launch mlflow ui locally mlflow_ui()  # launch mlflow ui for existing mlflow server mlflow_set_tracking_uri(""http://tracking-server:5000"") mlflow_ui()      mlflow_update_model_version  Update model version Updates a model version mlflow_update_model_version(name, version, description, client = NULL)    Arguments        Argument Description    name Name of the registered model.  version Model version number.  description Description of this model version.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.       mlflow_update_registered_model  Update a registered model Updates a model in the Model Registry. mlflow_update_registered_model(name, description, client = NULL)    Arguments        Argument Description    name The name of the registered model.  description The updated description for this registered model.  client (Optional) An MLflow client object returned from mlflow_client . If specified, MLflow will use the tracking server associated with the passed-in client. If unspecified (the common case), MLflow will use the tracking server associated with the current tracking URI.            Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
rest-api.html,"   Documentation  REST API       REST API  The MLflow REST API allows you to create, list, and get experiments and runs, and log parameters, metrics, and artifacts. The API is hosted under the /api route on the MLflow tracking server. For example, to search for experiments on a tracking server hosted at http://localhost:5000, make a POST request to http://localhost:5000/api/2.0/mlflow/experiments/search.  Table of Contents  Create Experiment Search Experiments Get Experiment Get Experiment By Name Delete Experiment Restore Experiment Update Experiment Create Run Delete Run Restore Run Get Run Log Metric Log Batch Log Model Log Inputs Set Experiment Tag Set Tag Delete Tag Log Param Get Metric History Search Runs List Artifacts Update Run Create RegisteredModel Get RegisteredModel Rename RegisteredModel Update RegisteredModel Delete RegisteredModel Get Latest ModelVersions Create ModelVersion Get ModelVersion Update ModelVersion Delete ModelVersion Search ModelVersions Get Download URI For ModelVersion Artifacts Transition ModelVersion Stage Search RegisteredModels Set Registered Model Tag Set Model Version Tag Delete Registered Model Tag Delete Model Version Tag Delete Registered Model Alias Get Model Version by Alias Set Registered Model Alias Data Structures     Create Experiment        Endpoint HTTP Method    2.0/mlflow/experiments/create POST    Create an experiment with a name. Returns the ID of the newly created experiment. Validates that another experiment with the same name does not already exist and fails if another experiment with the same name already exists. Throws RESOURCE_ALREADY_EXISTS if a experiment with the given name exists.  Request Structure         Field Name Type Description    name STRING Experiment name. This field is required.  artifact_location STRING Location where all artifacts for the experiment are stored. If not provided, the remote server will select an appropriate default.  tags An array of ExperimentTag A collection of tags to set on the experiment. Maximum tag size and number of tags per request depends on the storage backend. All storage backends are guaranteed to support tag keys up to 250 bytes in size and tag values up to 5000 bytes in size. All storage backends are also guaranteed to support up to 20 tags per request.      Response Structure         Field Name Type Description    experiment_id STRING Unique identifier for the experiment.        Search Experiments        Endpoint HTTP Method    2.0/mlflow/experiments/search POST     Request Structure         Field Name Type Description    max_results INT64 Maximum number of experiments desired. Servers may select a desired default max_results value. All servers are guaranteed to support a max_results threshold of at least 1,000 but may support more. Callers of this endpoint are encouraged to pass max_results explicitly and leverage page_token to iterate through experiments.  page_token STRING Token indicating the page of experiments to fetch  filter STRING A filter expression over experiment attributes and tags that allows returning a subset of experiments. The syntax is a subset of SQL that supports ANDing together binary operations between an attribute or tag, and a constant. Example: name LIKE 'test-%' AND tags.key = 'value' You can select columns with special characters (hyphen, space, period, etc.) by using double quotes or backticks. Example: tags.""extra-key"" = 'value' or tags.`extra-key` = 'value' Supported operators are =, !=, LIKE, and ILIKE.   order_by An array of STRING List of columns for ordering search results, which can include experiment name and id with an optional "DESC" or "ASC" annotation, where "ASC" is the default. Tiebreaks are done by experiment id DESC.  view_type ViewType Qualifier for type of experiments to be returned. If unspecified, return only active experiments.      Response Structure         Field Name Type Description    experiments An array of Experiment Experiments that match the search criteria  next_page_token STRING Token that can be used to retrieve the next page of experiments. An empty token means that no more experiments are available for retrieval.        Get Experiment        Endpoint HTTP Method    2.0/mlflow/experiments/get GET    Get metadata for an experiment. This method works on deleted experiments.  Request Structure         Field Name Type Description    experiment_id STRING ID of the associated experiment. This field is required.      Response Structure         Field Name Type Description    experiment Experiment Experiment details.        Get Experiment By Name        Endpoint HTTP Method    2.0/mlflow/experiments/get-by-name GET    Get metadata for an experiment. This endpoint will return deleted experiments, but prefers the active experiment if an active and deleted experiment share the same name. If multiple deleted experiments share the same name, the API will return one of them. Throws RESOURCE_DOES_NOT_EXIST if no experiment with the specified name exists.  Request Structure         Field Name Type Description    experiment_name STRING Name of the associated experiment. This field is required.      Response Structure         Field Name Type Description    experiment Experiment Experiment details.        Delete Experiment        Endpoint HTTP Method    2.0/mlflow/experiments/delete POST    Mark an experiment and associated metadata, runs, metrics, params, and tags for deletion. If the experiment uses FileStore, artifacts associated with experiment are also deleted.  Request Structure         Field Name Type Description    experiment_id STRING ID of the associated experiment. This field is required.        Restore Experiment        Endpoint HTTP Method    2.0/mlflow/experiments/restore POST    Restore an experiment marked for deletion. This also restores associated metadata, runs, metrics, params, and tags. If experiment uses FileStore, underlying artifacts associated with experiment are also restored. Throws RESOURCE_DOES_NOT_EXIST if experiment was never created or was permanently deleted.  Request Structure         Field Name Type Description    experiment_id STRING ID of the associated experiment. This field is required.        Update Experiment        Endpoint HTTP Method    2.0/mlflow/experiments/update POST    Update experiment metadata.  Request Structure         Field Name Type Description    experiment_id STRING ID of the associated experiment. This field is required.  new_name STRING If provided, the experiment's name is changed to the new name. The new name must be unique.        Create Run        Endpoint HTTP Method    2.0/mlflow/runs/create POST    Create a new run within an experiment. A run is usually a single execution of a machine learning or data ETL pipeline. MLflow uses runs to track Param, Metric, and RunTag associated with a single execution.  Request Structure         Field Name Type Description    experiment_id STRING ID of the associated experiment.  user_id STRING ID of the user executing the run. This field is deprecated as of MLflow 1.0, and will be removed in a future MLflow release. Use 'mlflow.user' tag instead.  run_name STRING Name of the run.  start_time INT64 Unix timestamp in milliseconds of when the run started.  tags An array of RunTag Additional metadata for run.      Response Structure         Field Name Type Description    run Run The newly created run.        Delete Run        Endpoint HTTP Method    2.0/mlflow/runs/delete POST    Mark a run for deletion.  Request Structure         Field Name Type Description    run_id STRING ID of the run to delete. This field is required.        Restore Run        Endpoint HTTP Method    2.0/mlflow/runs/restore POST    Restore a deleted run.  Request Structure         Field Name Type Description    run_id STRING ID of the run to restore. This field is required.        Get Run        Endpoint HTTP Method    2.0/mlflow/runs/get GET    Get metadata, metrics, params, and tags for a run. In the case where multiple metrics with the same key are logged for a run, return only the value with the latest timestamp. If there are multiple values with the latest timestamp, return the maximum of these values.  Request Structure         Field Name Type Description    run_id STRING ID of the run to fetch. Must be provided.  run_uuid STRING [Deprecated, use run_id instead] ID of the run to fetch. This field will be removed in a future MLflow version.      Response Structure         Field Name Type Description    run Run Run metadata (name, start time, etc) and data (metrics, params, and tags).        Log Metric        Endpoint HTTP Method    2.0/mlflow/runs/log-metric POST    Log a metric for a run. A metric is a key-value pair (string key, float value) with an associated timestamp. Examples include the various metrics that represent ML model accuracy. A metric can be logged multiple times.  Request Structure         Field Name Type Description    run_id STRING ID of the run under which to log the metric. Must be provided.  run_uuid STRING [Deprecated, use run_id instead] ID of the run under which to log the metric. This field will be removed in a future MLflow version.  key STRING Name of the metric. This field is required.  value DOUBLE Double value of the metric being logged. This field is required.  timestamp INT64 Unix timestamp in milliseconds at the time metric was logged. This field is required.  step INT64 Step at which to log the metric        Log Batch        Endpoint HTTP Method    2.0/mlflow/runs/log-batch POST    Log a batch of metrics, params, and tags for a run. If any data failed to be persisted, the server will respond with an error (non-200 status code). In case of error (due to internal server error or an invalid request), partial data may be written. You can write metrics, params, and tags in interleaving fashion, but within a given entity type are guaranteed to follow the order specified in the request body. That is, for an API request like {    ""run_id"": ""2a14ed5c6a87499199e0106c3501eab8"",    ""metrics"": [      {""key"": ""mae"", ""value"": 2.5, ""timestamp"": 1552550804},      {""key"": ""rmse"", ""value"": 2.7, ""timestamp"": 1552550804},    ],    ""params"": [      {""key"": ""model_class"", ""value"": ""LogisticRegression""},    ] }   the server is guaranteed to write metric "rmse" after "mae", though it may write param "model_class" before both metrics, after "mae", or after both metrics. The overwrite behavior for metrics, params, and tags is as follows:  Metrics: metric values are never overwritten. Logging a metric (key, value, timestamp) appends to the set of values for the metric with the provided key. Tags: tag values can be overwritten by successive writes to the same tag key. That is, if multiple tag values with the same key are provided in the same API request, the last-provided tag value is written. Logging the same tag (key, value) is permitted - that is, logging a tag is idempotent. Params: once written, param values cannot be changed (attempting to overwrite a param value will result in an error). However, logging the same param (key, value) is permitted - that is, logging a param is idempotent.   Request Limits  A single JSON-serialized API request may be up to 1 MB in size and contain:  No more than 1000 metrics, params, and tags in total Up to 1000 metrics Up to 100 params Up to 100 tags  For example, a valid request might contain 900 metrics, 50 params, and 50 tags, but logging 900 metrics, 50 params, and 51 tags is invalid. The following limits also apply to metric, param, and tag keys and values:  Metric, param, and tag keys can be up to 250 characters in length Param and tag values can be up to 250 characters in length    Request Structure         Field Name Type Description    run_id STRING ID of the run to log under  metrics An array of Metric Metrics to log. A single request can contain up to 1000 metrics, and up to 1000 metrics, params, and tags in total.  params An array of Param Params to log. A single request can contain up to 100 params, and up to 1000 metrics, params, and tags in total.  tags An array of RunTag Tags to log. A single request can contain up to 100 tags, and up to 1000 metrics, params, and tags in total.        Log Model        Endpoint HTTP Method    2.0/mlflow/runs/log-model POST     Note Experimental: This API may change or be removed in a future release without warning.   Request Structure         Field Name Type Description    run_id STRING ID of the run to log under  model_json STRING MLmodel file in json format.        Log Inputs        Endpoint HTTP Method    2.0/mlflow/runs/log-inputs POST     Note Experimental: This API may change or be removed in a future release without warning.   Request Structure   Note Experimental: This API may change or be removed in a future release without warning.         Field Name Type Description    run_id STRING ID of the run to log under This field is required.  datasets An array of DatasetInput Dataset inputs        Set Experiment Tag        Endpoint HTTP Method    2.0/mlflow/experiments/set-experiment-tag POST    Set a tag on an experiment. Experiment tags are metadata that can be updated.  Request Structure         Field Name Type Description    experiment_id STRING ID of the experiment under which to log the tag. Must be provided. This field is required.  key STRING Name of the tag. Maximum size depends on storage backend. All storage backends are guaranteed to support key values up to 250 bytes in size. This field is required.  value STRING String value of the tag being logged. Maximum size depends on storage backend. All storage backends are guaranteed to support key values up to 5000 bytes in size. This field is required.        Set Tag        Endpoint HTTP Method    2.0/mlflow/runs/set-tag POST    Set a tag on a run. Tags are run metadata that can be updated during a run and after a run completes.  Request Structure         Field Name Type Description    run_id STRING ID of the run under which to log the tag. Must be provided.  run_uuid STRING [Deprecated, use run_id instead] ID of the run under which to log the tag. This field will be removed in a future MLflow version.  key STRING Name of the tag. Maximum size depends on storage backend. All storage backends are guaranteed to support key values up to 250 bytes in size. This field is required.  value STRING String value of the tag being logged. Maximum size depends on storage backend. All storage backends are guaranteed to support key values up to 5000 bytes in size. This field is required.        Delete Tag        Endpoint HTTP Method    2.0/mlflow/runs/delete-tag POST    Delete a tag on a run. Tags are run metadata that can be updated during a run and after a run completes.  Request Structure         Field Name Type Description    run_id STRING ID of the run that the tag was logged under. Must be provided. This field is required.  key STRING Name of the tag. Maximum size is 255 bytes. Must be provided. This field is required.        Log Param        Endpoint HTTP Method    2.0/mlflow/runs/log-parameter POST    Log a param used for a run. A param is a key-value pair (string key, string value). Examples include hyperparameters used for ML model training and constant dates and values used in an ETL pipeline. A param can be logged only once for a run.  Request Structure         Field Name Type Description    run_id STRING ID of the run under which to log the param. Must be provided.  run_uuid STRING [Deprecated, use run_id instead] ID of the run under which to log the param. This field will be removed in a future MLflow version.  key STRING Name of the param. Maximum size is 255 bytes. This field is required.  value STRING String value of the param being logged. Maximum size is 6000 bytes. This field is required.        Get Metric History        Endpoint HTTP Method    2.0/mlflow/metrics/get-history GET    Get a list of all values for the specified metric for a given run.  Request Structure         Field Name Type Description    run_id STRING ID of the run from which to fetch metric values. Must be provided.  run_uuid STRING [Deprecated, use run_id instead] ID of the run from which to fetch metric values. This field will be removed in a future MLflow version.  metric_key STRING Name of the metric. This field is required.  page_token STRING Token indicating the page of metric history to fetch  max_results INT32 Maximum number of logged instances of a metric for a run to return per call. Backend servers may restrict the value of max_results depending on performance requirements. Requests that do not specify this value will behave as non-paginated queries where all metric history values for a given metric within a run are returned in a single response.      Response Structure         Field Name Type Description    metrics An array of Metric All logged values for this metric.  next_page_token STRING Token that can be used to issue a query for the next page of metric history values. A missing token indicates that no additional metrics are available to fetch.        Search Runs        Endpoint HTTP Method    2.0/mlflow/runs/search POST    Search for runs that satisfy expressions. Search expressions can use Metric and Param keys.  Request Structure         Field Name Type Description    experiment_ids An array of STRING List of experiment IDs to search over.  filter STRING A filter expression over params, metrics, and tags, that allows returning a subset of runs. The syntax is a subset of SQL that supports ANDing together binary operations between a param, metric, or tag and a constant. Example: metrics.rmse < 1 and params.model_class = 'LogisticRegression' You can select columns with special characters (hyphen, space, period, etc.) by using double quotes: metrics.""model class"" = 'LinearRegression' and tags.""user-name"" = 'Tomas' Supported operators are =, !=, >, >=, <, and <=.   run_view_type ViewType Whether to display only active, only deleted, or all runs. Defaults to only active runs.  max_results INT32 Maximum number of runs desired. If unspecified, defaults to 1000. All servers are guaranteed to support a max_results threshold of at least 50,000 but may support more. Callers of this endpoint are encouraged to pass max_results explicitly and leverage page_token to iterate through experiments.  order_by An array of STRING List of columns to be ordered by, including attributes, params, metrics, and tags with an optional "DESC" or "ASC" annotation, where "ASC" is the default. Example: ["params.input DESC", "metrics.alpha ASC", "metrics.rmse"] Tiebreaks are done by start_time DESC followed by run_id for runs with the same start time (and this is the default ordering criterion if order_by is not provided).  page_token STRING       Response Structure         Field Name Type Description    runs An array of Run Runs that match the search criteria.  next_page_token STRING         List Artifacts        Endpoint HTTP Method    2.0/mlflow/artifacts/list GET    List artifacts for a run. Takes an optional artifact_path prefix which if specified, the response contains only artifacts with the specified prefix.  Request Structure         Field Name Type Description    run_id STRING ID of the run whose artifacts to list. Must be provided.  run_uuid STRING [Deprecated, use run_id instead] ID of the run whose artifacts to list. This field will be removed in a future MLflow version.  path STRING Filter artifacts matching this path (a relative path from the root artifact directory).  page_token STRING Token indicating the page of artifact results to fetch      Response Structure         Field Name Type Description    root_uri STRING Root artifact directory for the run.  files An array of FileInfo File location and metadata for artifacts.  next_page_token STRING Token that can be used to retrieve the next page of artifact results        Update Run        Endpoint HTTP Method    2.0/mlflow/runs/update POST    Update run metadata.  Request Structure         Field Name Type Description    run_id STRING ID of the run to update. Must be provided.  run_uuid STRING [Deprecated, use run_id instead] ID of the run to update.. This field will be removed in a future MLflow version.  status RunStatus Updated status of the run.  end_time INT64 Unix timestamp in milliseconds of when the run ended.  run_name STRING Updated name of the run.      Response Structure         Field Name Type Description    run_info RunInfo Updated metadata of the run.        Create RegisteredModel        Endpoint HTTP Method    2.0/mlflow/registered-models/create POST    Throws RESOURCE_ALREADY_EXISTS if a registered model with the given name exists.  Request Structure         Field Name Type Description    name STRING Register models under this name This field is required.  tags An array of RegisteredModelTag Additional metadata for registered model.  description STRING Optional description for registered model.      Response Structure         Field Name Type Description    registered_model RegisteredModel         Get RegisteredModel        Endpoint HTTP Method    2.0/mlflow/registered-models/get GET     Request Structure         Field Name Type Description    name STRING Registered model unique name identifier. This field is required.      Response Structure         Field Name Type Description    registered_model RegisteredModel         Rename RegisteredModel        Endpoint HTTP Method    2.0/mlflow/registered-models/rename POST     Request Structure         Field Name Type Description    name STRING Registered model unique name identifier. This field is required.  new_name STRING If provided, updates the name for this registered_model.      Response Structure         Field Name Type Description    registered_model RegisteredModel         Update RegisteredModel        Endpoint HTTP Method    2.0/mlflow/registered-models/update PATCH     Request Structure         Field Name Type Description    name STRING Registered model unique name identifier. This field is required.  description STRING If provided, updates the description for this registered_model.      Response Structure         Field Name Type Description    registered_model RegisteredModel         Delete RegisteredModel        Endpoint HTTP Method    2.0/mlflow/registered-models/delete DELETE     Request Structure         Field Name Type Description    name STRING Registered model unique name identifier. This field is required.        Get Latest ModelVersions        Endpoint HTTP Method    2.0/mlflow/registered-models/get-latest-versions POST     Request Structure         Field Name Type Description    name STRING Registered model unique name identifier. This field is required.  stages An array of STRING List of stages.      Response Structure         Field Name Type Description    model_versions An array of ModelVersion Latest version models for each requests stage. Only return models with current READY status. If no stages provided, returns the latest version for each stage, including ""None"".        Create ModelVersion        Endpoint HTTP Method    2.0/mlflow/model-versions/create POST     Request Structure         Field Name Type Description    name STRING Register model under this name This field is required.  source STRING URI indicating the location of the model artifacts. This field is required.  run_id STRING MLflow run ID for correlation, if source was generated by an experiment run in MLflow tracking server  tags An array of ModelVersionTag Additional metadata for model version.  run_link STRING MLflow run link - this is the exact link of the run that generated this model version, potentially hosted at another instance of MLflow.  description STRING Optional description for model version.      Response Structure         Field Name Type Description    model_version ModelVersion Return new version number generated for this model in registry.        Get ModelVersion        Endpoint HTTP Method    2.0/mlflow/model-versions/get GET     Request Structure         Field Name Type Description    name STRING Name of the registered model This field is required.  version STRING Model version number This field is required.      Response Structure         Field Name Type Description    model_version ModelVersion         Update ModelVersion        Endpoint HTTP Method    2.0/mlflow/model-versions/update PATCH     Request Structure         Field Name Type Description    name STRING Name of the registered model This field is required.  version STRING Model version number This field is required.  description STRING If provided, updates the description for this registered_model.      Response Structure         Field Name Type Description    model_version ModelVersion Return new version number generated for this model in registry.        Delete ModelVersion        Endpoint HTTP Method    2.0/mlflow/model-versions/delete DELETE     Request Structure         Field Name Type Description    name STRING Name of the registered model This field is required.  version STRING Model version number This field is required.        Search ModelVersions        Endpoint HTTP Method    2.0/mlflow/model-versions/search GET     Request Structure         Field Name Type Description    filter STRING String filter condition, like "name='my-model-name'". Must be a single boolean condition, with string values wrapped in single quotes.  max_results INT64 Maximum number of models desired. Max threshold is 200K. Backends may choose a lower default value and maximum threshold.  order_by An array of STRING List of columns to be ordered by including model name, version, stage with an optional "DESC" or "ASC" annotation, where "ASC" is the default. Tiebreaks are done by latest stage transition timestamp, followed by name ASC, followed by version DESC.  page_token STRING Pagination token to go to next page based on previous search query.      Response Structure         Field Name Type Description    model_versions An array of ModelVersion Models that match the search criteria  next_page_token STRING Pagination token to request next page of models for the same search query.        Get Download URI For ModelVersion Artifacts        Endpoint HTTP Method    2.0/mlflow/model-versions/get-download-uri GET     Request Structure         Field Name Type Description    name STRING Name of the registered model This field is required.  version STRING Model version number This field is required.      Response Structure         Field Name Type Description    artifact_uri STRING URI corresponding to where artifacts for this model version are stored.        Transition ModelVersion Stage        Endpoint HTTP Method    2.0/mlflow/model-versions/transition-stage POST     Request Structure         Field Name Type Description    name STRING Name of the registered model This field is required.  version STRING Model version number This field is required.  stage STRING Transition model_version to new stage. This field is required.  archive_existing_versions BOOL When transitioning a model version to a particular stage, this flag dictates whether all existing model versions in that stage should be atomically moved to the "archived" stage. This ensures that at-most-one model version exists in the target stage. This field is required when transitioning a model versions's stage This field is required.      Response Structure         Field Name Type Description    model_version ModelVersion Updated model version        Search RegisteredModels        Endpoint HTTP Method    2.0/mlflow/registered-models/search GET     Request Structure         Field Name Type Description    filter STRING String filter condition, like "name LIKE 'my-model-name'". Interpreted in the backend automatically as "name LIKE '%my-model-name%'". Single boolean condition, with string values wrapped in single quotes.  max_results INT64 Maximum number of models desired. Default is 100. Max threshold is 1000.  order_by An array of STRING List of columns for ordering search results, which can include model name and last updated timestamp with an optional "DESC" or "ASC" annotation, where "ASC" is the default. Tiebreaks are done by model name ASC.  page_token STRING Pagination token to go to the next page based on a previous search query.      Response Structure         Field Name Type Description    registered_models An array of RegisteredModel Registered Models that match the search criteria.  next_page_token STRING Pagination token to request the next page of models.        Set Registered Model Tag        Endpoint HTTP Method    2.0/mlflow/registered-models/set-tag POST     Request Structure         Field Name Type Description    name STRING Unique name of the model. This field is required.  key STRING Name of the tag. Maximum size depends on storage backend. If a tag with this name already exists, its preexisting value will be replaced by the specified value. All storage backends are guaranteed to support key values up to 250 bytes in size. This field is required.  value STRING String value of the tag being logged. Maximum size depends on storage backend. This field is required.        Set Model Version Tag        Endpoint HTTP Method    2.0/mlflow/model-versions/set-tag POST     Request Structure         Field Name Type Description    name STRING Unique name of the model. This field is required.  version STRING Model version number. This field is required.  key STRING Name of the tag. Maximum size depends on storage backend. If a tag with this name already exists, its preexisting value will be replaced by the specified value. All storage backends are guaranteed to support key values up to 250 bytes in size. This field is required.  value STRING String value of the tag being logged. Maximum size depends on storage backend. This field is required.        Delete Registered Model Tag        Endpoint HTTP Method    2.0/mlflow/registered-models/delete-tag DELETE     Request Structure         Field Name Type Description    name STRING Name of the registered model that the tag was logged under. This field is required.  key STRING Name of the tag. The name must be an exact match; wild-card deletion is not supported. Maximum size is 250 bytes. This field is required.        Delete Model Version Tag        Endpoint HTTP Method    2.0/mlflow/model-versions/delete-tag DELETE     Request Structure         Field Name Type Description    name STRING Name of the registered model that the tag was logged under. This field is required.  version STRING Model version number that the tag was logged under. This field is required.  key STRING Name of the tag. The name must be an exact match; wild-card deletion is not supported. Maximum size is 250 bytes. This field is required.        Delete Registered Model Alias        Endpoint HTTP Method    2.0/mlflow/registered-models/alias DELETE     Request Structure         Field Name Type Description    name STRING Name of the registered model. This field is required.  alias STRING Name of the alias. The name must be an exact match; wild-card deletion is not supported. Maximum size is 256 bytes. This field is required.        Get Model Version by Alias        Endpoint HTTP Method    2.0/mlflow/registered-models/alias GET     Request Structure         Field Name Type Description    name STRING Name of the registered model. This field is required.  alias STRING Name of the alias. Maximum size is 256 bytes. This field is required.      Response Structure         Field Name Type Description    model_version ModelVersion         Set Registered Model Alias        Endpoint HTTP Method    2.0/mlflow/registered-models/alias POST     Request Structure         Field Name Type Description    name STRING Name of the registered model. This field is required.  alias STRING Name of the alias. Maximum size depends on storage backend. If an alias with this name already exists, its preexisting value will be replaced by the specified version. All storage backends are guaranteed to support alias name values up to 256 bytes in size. This field is required.  version STRING Model version number. This field is required.       Data Structures   Dataset   Note Experimental: This API may change or be removed in a future release without warning.  Dataset. Represents a reference to data used for training, testing, or evaluation during the model development process.        Field Name Type Description    name STRING The name of the dataset. E.g. ?my.uc.table@2? ?nyc-taxi-dataset?, ?fantastic-elk-3? This field is required.  digest STRING Dataset digest, e.g. an md5 hash of the dataset that uniquely identifies it within datasets of the same name. This field is required.  source_type STRING Source information for the dataset. Note that the source may not exactly reproduce the dataset if it was transformed / modified before use with MLflow. This field is required.  source STRING The type of the dataset source, e.g. ?databricks-uc-table?, ?DBFS?, ?S3?, … This field is required.  schema STRING The schema of the dataset. E.g., MLflow ColSpec JSON for a dataframe, MLflow TensorSpec JSON for an ndarray, or another schema format.  profile STRING The profile of the dataset. Summary statistics for the dataset, such as the number of rows in a table, the mean / std / mode of each column in a table, or the number of elements in an array.      DatasetInput   Note Experimental: This API may change or be removed in a future release without warning.  DatasetInput. Represents a dataset and input tags.        Field Name Type Description    tags An array of InputTag A list of tags for the dataset input, e.g. a ?context? tag with value ?training?  dataset Dataset The dataset being used as a Run input. This field is required.      Experiment  Experiment        Field Name Type Description    experiment_id STRING Unique identifier for the experiment.  name STRING Human readable name that identifies the experiment.  artifact_location STRING Location where artifacts for the experiment are stored.  lifecycle_stage STRING Current life cycle stage of the experiment: "active" or "deleted". Deleted experiments are not returned by APIs.  last_update_time INT64 Last update time  creation_time INT64 Creation time  tags An array of ExperimentTag Tags: Additional metadata key-value pairs.      ExperimentTag  Tag for an experiment.        Field Name Type Description    key STRING The tag key.  value STRING The tag value.      FileInfo         Field Name Type Description    path STRING Path relative to the root artifact directory run.  is_dir BOOL Whether the path is a directory.  file_size INT64 Size in bytes. Unset for directories.      InputTag   Note Experimental: This API may change or be removed in a future release without warning.  Tag for an input.        Field Name Type Description    key STRING The tag key. This field is required.  value STRING The tag value. This field is required.      Metric  Metric associated with a run, represented as a key-value pair.        Field Name Type Description    key STRING Key identifying this metric.  value DOUBLE Value associated with this metric.  timestamp INT64 The timestamp at which this metric was recorded.  step INT64 Step at which to log the metric.      ModelVersion         Field Name Type Description    name STRING Unique name of the model  version STRING Model's version number.  creation_timestamp INT64 Timestamp recorded when this model_version was created.  last_updated_timestamp INT64 Timestamp recorded when metadata for this model_version was last updated.  user_id STRING User that created this model_version.  current_stage STRING Current stage for this model_version.  description STRING Description of this model_version.  source STRING URI indicating the location of the source model artifacts, used when creating model_version  run_id STRING MLflow run ID used when creating model_version, if source was generated by an experiment run stored in MLflow tracking server.  status ModelVersionStatus Current status of model_version  status_message STRING Details on current status, if it is pending or failed.  tags An array of ModelVersionTag Tags: Additional metadata key-value pairs for this model_version.  run_link STRING Run Link: Direct link to the run that generated this version. This field is set at model version creation time only for model versions whose source run is from a tracking server that is different from the registry server.  aliases An array of STRING Aliases pointing to this model_version.      ModelVersionTag  Tag for a model version.        Field Name Type Description    key STRING The tag key.  value STRING The tag value.      Param  Param associated with a run.        Field Name Type Description    key STRING Key identifying this param.  value STRING Value associated with this param.      RegisteredModel         Field Name Type Description    name STRING Unique name for the model.  creation_timestamp INT64 Timestamp recorded when this registered_model was created.  last_updated_timestamp INT64 Timestamp recorded when metadata for this registered_model was last updated.  user_id STRING User that created this registered_model NOTE: this field is not currently returned.  description STRING Description of this registered_model.  latest_versions An array of ModelVersion Collection of latest model versions for each stage. Only contains models with current READY status.  tags An array of RegisteredModelTag Tags: Additional metadata key-value pairs for this registered_model.  aliases An array of RegisteredModelAlias Aliases pointing to model versions associated with this registered_model.      RegisteredModelAlias  Alias for a registered model        Field Name Type Description    alias STRING The name of the alias.  version STRING The model version number that the alias points to.      RegisteredModelTag  Tag for a registered model        Field Name Type Description    key STRING The tag key.  value STRING The tag value.      Run  A single run.        Field Name Type Description    info RunInfo Run metadata.  data RunData Run data.  inputs RunInputs Run inputs.      RunData  Run data (metrics, params, and tags).        Field Name Type Description    metrics An array of Metric Run metrics.  params An array of Param Run parameters.  tags An array of RunTag Additional metadata key-value pairs.      RunInfo  Metadata of a single run.        Field Name Type Description    run_id STRING Unique identifier for the run.  run_uuid STRING [Deprecated, use run_id instead] Unique identifier for the run. This field will be removed in a future MLflow version.  run_name STRING The name of the run.  experiment_id STRING The experiment ID.  user_id STRING User who initiated the run. This field is deprecated as of MLflow 1.0, and will be removed in a future MLflow release. Use 'mlflow.user' tag instead.  status RunStatus Current status of the run.  start_time INT64 Unix timestamp of when the run started in milliseconds.  end_time INT64 Unix timestamp of when the run ended in milliseconds.  artifact_uri STRING URI of the directory where artifacts should be uploaded. This can be a local path (starting with "/"), or a distributed file system (DFS) path, like s3://bucket/directory or dbfs:/my/directory. If not set, the local ./mlruns directory is  chosen.  lifecycle_stage STRING Current life cycle stage of the experiment : OneOf("active", "deleted")      RunInputs   Note Experimental: This API may change or be removed in a future release without warning.  Run inputs.        Field Name Type Description    dataset_inputs An array of DatasetInput Dataset inputs to the Run.      RunTag  Tag for a run.        Field Name Type Description    key STRING The tag key.  value STRING The tag value.      ModelVersionStatus        Name Description    PENDING_REGISTRATION Request to register a new model version is pending as server performs background tasks.  FAILED_REGISTRATION Request to register a new model version has failed.  READY Model version is ready for use.      RunStatus  Status of a run.       Name Description    RUNNING Run has been initiated.  SCHEDULED Run is scheduled to run at a later time.  FINISHED Run has completed.  FAILED Run execution failed.  KILLED Run killed by user.      ViewType  View type for ListExperiments query.       Name Description    ACTIVE_ONLY Default. Return only active experiments.  DELETED_ONLY Return only deleted experiments.  ALL Get all experiments.            Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
docker.html,"   Documentation  Official MLflow Docker Image       Official MLflow Docker Image  The official MLflow Docker image is available on GitHub Container Registry at https://ghcr.io/mlflow/mlflow. export CR_PAT=YOUR_TOKEN echo $CR_PAT | docker login ghcr.io -u USERNAME --password-stdin # Pull the latest version docker pull ghcr.io/mlflow/mlflow # Pull 2.0.1 docker pull ghcr.io/mlflow/mlflow:v2.0.1         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
community-model-flavors.html,"   Documentation  Community Model Flavors       Community Model Flavors  Other useful MLflow flavors are developed and maintained by the MLflow community, enabling you to use MLflow Models with an even broader ecosystem of machine learning libraries. For more information, check out the description of each community-developed flavor below.   MLflow VizMod BigML (bigmlflow) Sktime MLflavors    MLflow VizMod  The mlflow-vizmod project allows data scientists to be more productive with their visualizations. We treat visualizations as models - just like ML models - thus being able to use the same infrastructure as MLflow to track, create projects, register, and deploy visualizations. Installation: pip install mlflow-vizmod   Example: from sklearn.datasets import load_iris import altair as alt import mlflow_vismod  df_iris = load_iris(as_frame=True)  viz_iris = (     alt.Chart(df_iris)     .mark_circle(size=60)     .encode(x=""x"", y=""y"", color=""z:N"")     .properties(height=375, width=575)     .interactive() )  mlflow_vismod.log_model(     model=viz_iris,     artifact_path=""viz"",     style=""vegalite"",     input_example=df_iris.head(5), )     BigML (bigmlflow)  The bigmlflow library implements the bigml model flavor. It enables using BigML supervised models and offers the save_model(), log_model() and load_model() methods.  Installing bigmlflow  BigMLFlow can be installed from PyPI as follows: pip install bigmlflow     BigMLFlow usage  The bigmlflow module defines the flavor that implements the save_model() and log_model() methods. They can be used to save BigML models and their related information in MLflow Model format. import json import mlflow import bigmlflow  MODEL_FILE = ""logistic_regression.json"" with mlflow.start_run():     with open(MODEL_FILE) as handler:         model = json.load(handler)         bigmlflow.log_model(             model, artifact_path=""model"", registered_model_name=""my_model""         )   These methods also add the python_function flavor to the MLflow Models that they produce, allowing the models to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can only be scored with DataFrame inputs. # saving the model save_model(model, path=model_path) # retrieving model pyfunc_model = pyfunc.load_model(model_path) pyfunc_predictions = pyfunc_model.predict(dataframe)   You can also use the bigmlflow.load_model() method to load MLflow Models with the bigmlflow model flavor as a BigML SupervisedModel. For more information, see the BigMLFlow documentation and BigML's blog.    Sktime  The sktime custom model flavor enables logging of sktime models in MLflow format via the save_model() and log_model() methods. These methods also add the python_function flavor to the MLflow Models that they produce, allowing the model to be interpreted as generic Python functions for inference via mlflow.pyfunc.load_model(). This loaded PyFunc model can only be scored with a DataFrame input. You can also use the load_model() method to load MLflow Models with the sktime model flavor in native sktime formats.  Installing Sktime  Install sktime with mlflow dependency: pip install sktime[mlflow]     Usage example  Refer to the sktime mlflow documentation for details on the interface for utilizing sktime models loaded as a pyfunc type and an example notebook for extended code usage examples. import pandas as pd  from sktime.datasets import load_airline from sktime.forecasting.arima import AutoARIMA from sktime.utils import mlflow_sktime  airline = load_airline() model_path = ""model""   auto_arima_model = AutoARIMA(sp=12, d=0, max_p=2, max_q=2, suppress_warnings=True).fit(     airline, fh=[1, 2, 3] )  mlflow_sktime.save_model(     sktime_model=auto_arima_model,     path=model_path, )  loaded_model = mlflow_sktime.load_model(     model_uri=model_path, ) loaded_pyfunc = mlflow_sktime.pyfunc.load_model(     model_uri=model_path, )  print(loaded_model.predict()) print(loaded_pyfunc.predict(pd.DataFrame()))      MLflavors  The MLflavors package adds MLflow support for some popular machine learning frameworks currently not considered for inclusion as MLflow built-in flavors. Similar to the built-in flavors, you can use this package to save your model as an MLflow artifact, load your model from MLflow for batch inference, and deploy your model to a serving endpoint using MLflow deployment tools. The following open-source libraries are currently supported:         Framework Tutorials Category  Orbit MLflow-Orbit Time Series Forecasting  Sktime MLflow-Sktime Time Series Forecasting  StatsForecast MLflow-StatsForecast Time Series Forecasting  PyOD MLflow-PyOD Anomaly Detection  SDV MLflow-SDV Synthetic Data Generation     The interface design for the supported frameworks is similar to many of the existing built-in flavors. Particularly, the interface for utilizing the custom model loaded as a pyfunc flavor for generating predictions uses a single-row Pandas DataFrame configuration argument to expose the parameters of the flavor's inference API.  Documentation  Usage examples for all flavors and the API reference can be found in the package documenation.   Installation  Installing from PyPI: $ pip install mlflavors     Quickstart  This example trains a PyOD KNN outlier detection model using a synthetic dataset. A new MLflow experiment is created to log the evaluation metrics and the trained model as an artifact and anomaly scores are computed loading the trained model in native flavor and pyfunc flavor. Finally, the model is served for real-time inference using a local endpoint.  Saving the model as an MLflow artifact  import json  import mlflow import pandas as pd from pyod.models.knn import KNN from pyod.utils.data import generate_data from sklearn.metrics import roc_auc_score  import mlflavors  ARTIFACT_PATH = ""model""  with mlflow.start_run() as run:     contamination = 0.1  # percentage of outliers     n_train = 200  # number of training points     n_test = 100  # number of testing points      X_train, X_test, _, y_test = generate_data(         n_train=n_train, n_test=n_test, contamination=contamination     )      # Train kNN detector     clf = KNN()     clf.fit(X_train)      # Evaluate model     y_test_scores = clf.decision_function(X_test)      metrics = {         ""roc"": roc_auc_score(y_test, y_test_scores),     }      print(f""Metrics: \n{json.dumps(metrics, indent=2)}"")      # Log metrics     mlflow.log_metrics(metrics)      # Log model using pickle serialization (default).     mlflavors.pyod.log_model(         pyod_model=clf,         artifact_path=ARTIFACT_PATH,         serialization_format=""pickle"",     )     model_uri = mlflow.get_artifact_uri(ARTIFACT_PATH)  # Print the run id wich is used below for serving the model to a local REST API endpoint print(f""\nMLflow run id:\n{run.info.run_id}"")     Loading the model from MLflow  Make a prediction loading the model from MLflow in native format: loaded_model = mlflavors.pyod.load_model(model_uri=model_uri) print(loaded_model.decision_function(X_test))   Make a prediction loading the model from MLflow in pyfunc format: loaded_pyfunc = mlflavors.pyod.pyfunc.load_model(model_uri=model_uri)  # Create configuration DataFrame predict_conf = pd.DataFrame(     [         {             ""X"": X_test,             ""predict_method"": ""decision_function"",         }     ] )  print(loaded_pyfunc.predict(predict_conf)[0])     Serving the model using an endpoint  To serve the model using a local REST API endpoint run the command below where you substitute the run id printed above: mlflow models serve -m runs:/<run_id>/model --env-manager local --host 127.0.0.1   Similarly, you could serve the model using an endpoint in the cloud (e.g. Azure ML, AWS SageMaker, etc.) using MLflow deployment tools. Open a new terminal and run the below model scoring script to request a prediction from the served model: import pandas as pd import requests from pyod.utils.data import generate_data  contamination = 0.1  # percentage of outliers n_train = 200  # number of training points n_test = 100  # number of testing points  _, X_test, _, _ = generate_data(     n_train=n_train, n_test=n_test, contamination=contamination )  # Define local host and endpoint url host = ""127.0.0.1"" url = f""http://{host}:5000/invocations""  # Convert to list for JSON serialization X_test_list = X_test.tolist()  # Create configuration DataFrame predict_conf = pd.DataFrame(     [         {             ""X"": X_test_list,             ""predict_method"": ""decision_function"",         }     ] )  # Create dictionary with pandas DataFrame in the split orientation json_data = {""dataframe_split"": predict_conf.to_dict(orient=""split"")}  # Score model response = requests.post(url, json=json_data) print(response.json())            Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
tutorials-and-examples/index.html,"   Documentation  Tutorials and Examples       Tutorials and Examples  Below, you can find a number of tutorials and examples for various MLflow use cases.  Hyperparameter Tuning Orchestrating Multistep Workflows Using the MLflow REST API Directly Reproducibly run & share ML code  Packaging Training Code in a Docker Environment Python Package Anti-Tampering Packaging Training Code in a Conda Environment   Write & Use MLflow Plugins Instrument ML training code with MLflow  Gluon H2O Keras Prophet PyTorch XGBoost LightGBM Statsmodels Glmnet (R) SpaCy Fastai SHAP Prophet Pmdarima Diviner Transformers LangChain OpenAI Tensorflow SynapseML scikit-learn  Diabetes example Elastic Net example Logistic Regression example   RAPIDS  Random Forest Classifier            Previous               © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
getting-started/index.html,"   Documentation  Getting Started with MLflow       Getting Started with MLflow  For those new to MLflow or seeking a refresher on its core functionalities, the quickstart tutorials here are the perfect starting point. They will guide you step-by-step through fundamental concepts, focusing purely on a task that will maximize your understanding of how to use MLflow to solve a particular task.  5-minute Quickstart - MLflow Tracking  In this brief introductory quickstart on MLflow Tracking, you will learn how to leverage MLflow to:  Log training statistics (loss, accuracy, etc.) and hyperparameters for a model Log (save) a model for later retrieval Register a model to enable state transitions for deployment Load the model and use it for inference  In the process of learning these key concepts, you will be exposed to the MLflow fluent API, the MLflow Tracking UI, and learn how to add metadata associated with a model training event to an MLflow run.   If you would like to get started immediately by interactively running the notebook, you can:  Download the Notebook Quickstart elements  You can read through the quickstart as a guide, or navigate directly to the notebook example to get started.                          MLflow Tracking Quickstart Guide                                   Learn the basics of MLflow Tracking in a fast-paced guide with a focus on seeing your first model in the MLflow UI                                           MLflow Tracking Quickstart Notebook                                       See an example of using the MLflow fluent API to log, load, and register a model for inference. Great for code-focused learners!                        Logging your first MLflow Model  In this lengthy tutorial, you will walk through the basics of MLflow in a sequential and guided manner. With each subsequent step, you will increase your familiarity with the primary functionality around MLflow Tracking and how to navigate the MLflow UI. If you would like to get started immediately by interactively running the notebook, you can:  Download the Notebook Guide sections  Interested in navigating directly to the content that you're curious about? Select the section from each tutorial below!                          Setting up the MLflow Tracking Server                                       Learn how to start an MLflow Tracking Server and the MLflow UI Server locally                                           Using the MLflow Client                                       Connect to the Tracking Server with the MLflow Client and learn how to search for experiments                                           Create your first Experiment                                       Explore the MLflow UI and create your first MLflow experiment with a unique name and identifying tags                                           Search Experiments by tags                                       Learn how to use tags for MLflow experiments and how to leverage them for searching                                           Creating a Dataset for Testing                                       Build a synthetic dataset to use while exploring the features of MLflow                                           Logging your first MLflow run                                       Train a model using the synthetic dataset and log the trained model, metrics, and parameters                                           View the full Notebook                                       See the tutorial notebook in its entirety. If you prefer just reading code, this is the best place to look.                          15 minute Quickstart - Autologging in MLflow  In this rapid-pace quickstart, you will be exposed to the autologging feature in MLflow to simplify the logging of models, metrics, and parameters. After training and viewing the logged run data, we'll load the logged model to perform inference, showing core features of MLflow Tracking in the most time-efficient manner possible.                          Introduction to autologging                                       Train a model and use MLflow autologging to automatically record model artifacts, metrics, and parameters                                           View autologged data in the MLflow UI                                       See what autologging will autonomously log for you during model training with only a single line of code                                           Loading a model for inference                                       Load the autologged model in its native format and use it to generate predictions                         15 minute Quickstart - Comparing Runs and Deploying your Best Model  This quickstart tutorial focuses on the MLflow UI's run comparison feature, provides a brief overview of MLflow Projects, and shows how to register a model. After locally serving the registered model, a brief example of preparing a model for remote deployment via containerizing the model via Docker is covered.                          Generate runs                                       Run an MLflow Project that will perform hyperparameter tuning to generate a large volume of runs                                           Run comparison                                       Use the MLflow UI Runs Compare functionality to evaluate the hyperparameter tuning run and select the best model                                           Register the best model                                       Learn to register models with the MLflow UI and perform stage transitions from within the UI                                           Start a local ML inference server                                       Use the integrated inference server in MLflow to serve your registered model locally                                           Build a deployable container for your model                                       Learn how to generate a docker container that houses your model for deployment to external services                         5 Minute Tracking Server Overview  This quickstart tutorial walks through different types of MLflow Tracking Servers and how to use them to log your MLflow experiments.                          5 Minute Tracking Server Overview                                    Learn how to log MLflow experiments with different tracking servers                         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
getting-started/intro-quickstart/index.html,"   Documentation  Getting Started with MLflow  MLflow Tracking Quickstart       MLflow Tracking Quickstart  Welcome to MLflow! The purpose of this quickstart is to provide a quick guide to the most essential core APIs of MLflow Tracking. Specifically, those that enable the logging, registering, and loading of a model for inference.  Note For a more in-depth and tutorial-based approach (if that is your style), please see the Getting Started with MLflow tutorial. We recommend that you start here first, though, as this quickstart uses the most common and frequently-used APIs for MLflow Tracking and serves as a good foundation for the other tutorials in the documentation.   What you will learn  In just a few minutes of following along with this quickstart, you will learn:  How to log parameters, metrics, and a model The basics of the MLflow fluent API How to register a model during logging How to navigate to a model in the MLflow UI How to load a logged model for inference  If you would like to see this quickstart in a purely notebook format, we have a downloadable and viewable notebook-only version of this quickstart: View the Notebook    Step 1 - Get MLflow  MLflow is available on PyPI. If you don't already have it installed on your system, you can install it with:  pip install mlflow      Step 2 - Start a Tracking Server  We're going to start a local MLflow Tracking Server, which we will connect to for logging our data for this quickstart. From a terminal, run:  mlflow server --host 127.0.0.1 --port 8080     Note You can choose any port that you would like, provided that it's not already in use.    Step 3 - Train a model and prepare metadata for logging  In this section, we're going to log a model with MLflow. A quick overview of the steps are:  Load and prepare the Iris dataset for modeling. Train a Logistic Regression model and evaluate its performance. Prepare the model hyperparameters and calculate metrics for logging.   import mlflow from mlflow.models import infer_signature  import pandas as pd from sklearn import datasets from sklearn.model_selection import train_test_split from sklearn.linear_model import LogisticRegression from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score   # Load the Iris dataset X, y = datasets.load_iris(return_X_y=True)  # Split the data into training and test sets X_train, X_test, y_train, y_test = train_test_split(     X, y, test_size=0.2, random_state=42 )  # Define the model hyperparameters params = {     ""solver"": ""lbfgs"",     ""max_iter"": 1000,     ""multi_class"": ""auto"",     ""random_state"": 8888, }  # Train the model lr = LogisticRegression(**params) lr.fit(X_train, y_train)  # Predict on the test set y_pred = lr.predict(X_test)  # Calculate metrics accuracy = accuracy_score(y_test, y_pred)      Step 4 - Log the model and its metadata to MLflow  In this next step, we're going to use the model that we trained, the hyperparameters that we specified for the model's fit, and the loss metrics that were calculated by evaluating the model's performance on the test data to log to MLflow. The steps that we will take are:  Initiate an MLflow run context to start a new run that we will log the model and metadata to. Log model parameters and performance metrics. Tag the run for easy retrieval. Register the model in the MLflow Model Registry while logging (saving) the model.   Note While it can be valid to wrap the entire code within the start_run block, this is not recommended. If there as in issue with the training of the model or any other portion of code that is unrelated to MLflow-related actions, an empty or partially-logged run will be created, which will necessitate manual cleanup of the invalid run. It is best to keep the training execution outside of the run context block to ensure that the loggable content (parameters, metrics, artifacts, and the model) are fully materialized prior to logging.   # Set our tracking server uri for logging mlflow.set_tracking_uri(uri=""http://127.0.0.1:8080"")  # Create a new MLflow Experiment mlflow.set_experiment(""MLflow Quickstart"")  # Start an MLflow run with mlflow.start_run():     # Log the hyperparameters     mlflow.log_params(params)      # Log the loss metric     mlflow.log_metric(""accuracy"", accuracy)      # Set a tag that we can use to remind ourselves what this run was for     mlflow.set_tag(""Training Info"", ""Basic LR model for iris data"")      # Infer the model signature     signature = infer_signature(X_train, lr.predict(X_train))      # Log the model     model_info = mlflow.sklearn.log_model(         sk_model=lr,         artifact_path=""iris_model"",         signature=signature,         input_example=X_train,         registered_model_name=""tracking-quickstart"",     )      Step 5 - Load the model as a Python Function (pyfunc) and use it for inference  After logging the model, we can perform inference by:  Loading the model using MLflow's pyfunc flavor. Running Predict on new data using the loaded model.   Note The iris training data that we used was a numpy array structure. However, we can submit a Pandas DataFrame as well to the predict method, as shown below.   # Load the model back for predictions as a generic Python Function model loaded_model = mlflow.pyfunc.load_model(model_info.model_uri)  predictions = loaded_model.predict(X_test)  iris_feature_names = datasets.load_iris().feature_names  result = pd.DataFrame(X_test, columns=iris_feature_names) result[""actual_class""] = y_test result[""predicted_class""] = predictions  result[:4]    The output of this code will look something like this:           sepal length (cm) sepal width (cm) petal length (cm) petal width (cm) actual_class predicted_class    6.1 2.8 4.7 1.2 1 1  5.7 3.8 1.7 0.3 0 0  7.7 2.6 6.9 2.3 2 2  6.0 2.9 4.5 1.5 1 1      Step 6 - View the Run in the MLflow UI  In order to see the results of our run, we can navigate to the MLflow UI. Since we have already started the Tracking Server at http://localhost:8080, we can simply navigate to that URL in our browser. When opening the site, you will see a screen similar to the following:   The main MLflow Tracking page, showing Experiments that have been created   Clicking on the name of the Experiment that we created ("MLflow Quickstart") will give us a list of runs associated with the Experiment. You should see a random name that has been generated for the run and nothing else show up in the Table list view to the right. Clicking on the name of the run will take you to the Run page, where the details of what we've logged will be shown. The elements have been highlighted below to show how and where this data is recorded within the UI.   The run view page for our run     Conclusion  Congratulations on working through the MLflow Tracking Quickstart! You should now have a basic understanding of how to use the MLflow Tracking API to log models. If you are interested in a more in-depth tutorial, please see the Getting Started with MLflow tutorial as a good next step in increasing your knowledge about MLflow!        Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
getting-started/logging-first-model/index.html,"   Documentation  Getting Started with MLflow  Tutorial Overview       Tutorial Overview  In this entry point tutorial to MLflow, we'll be covering the essential basics of core MLflow functionality associated with tracking training event data. We'll start by learning how to start a local MLflow Tracking server, how to access and view the MLflow UI, and move on to our first interactions with the Tracking server through the use of the MLflow Client. The tutorial content builds upon itself, culminating in successfully logging your first MLflow model. The topics in this tutorial cover:  Starting an MLflow Tracking Server Exploring the MlflowClient API (briefly) Understanding the Default Experiment Searching for Experiments with the MLflow client API Understanding the uses of tags and how to leverage them for model organization Creating an Experiment that will contain our run (and our model) Learning how to log metrics, parameters, and a model artifact to a run Viewing our Experiment and our first run within the MLflow UI  To get started with the tutorial, click NEXT below.       Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
getting-started/quickstart-1/index.html,"   Documentation  Getting Started with MLflow  Quickstart: Install MLflow, instrument code & view results in minutes       Quickstart: Install MLflow, instrument code & view results in minutes  In less than 15 minutes, you will:  Install MLflow Add MLflow tracking to your code View runs and experiments in the MLflow tracking UI (Optional) Run a tracking server to share results with others (Optional) Use Databricks to store your results Store the models produced by your runs Load a model from a previous run for inference  As a data scientist, your explorations involve running your evolving training code many times. MLflow Tracking allows you to record important information your run, review and compare it with other runs, and share results with others. As an ML Engineer or MLOps professional, it allows you to compare, share, and deploy the best models produced by the team.  MLflow is available for Python, R, and Java, but this quickstart shows Python only. For Java, see Java API. For R, see R API.  Install MLflow  Install MLflow from PyPI using pip:  pip install mlflow    For more options, see Customize and troubleshoot MLflow installation.   Add MLflow tracking to your code  For many popular ML libraries, you make a single function call: mlflow.autolog(). If you are using one of the supported libraries, this will automatically log the parameters, metrics, and artifacts of your run (see list at Automatic Logging). For instance, the following autologs a scikit-learn run:  import mlflow  from sklearn.model_selection import train_test_split from sklearn.datasets import load_diabetes from sklearn.ensemble import RandomForestRegressor  mlflow.autolog()  db = load_diabetes() X_train, X_test, y_train, y_test = train_test_split(db.data, db.target)  # Create and train models. rf = RandomForestRegressor(n_estimators=100, max_depth=6, max_features=3) rf.fit(X_train, y_train)  # Use the model to make predictions on the test dataset. predictions = rf.predict(X_test)    In addition, or if you are using a library for which autolog is not yet supported, you may use key-value pairs to track:        Name Used for Function call    Parameters Constant values (for instance, configuration parameters) mlflow.log_param, mlflow.log_params  Metrics Values updated during the run (for instance, accuracy) mlflow.log_metric  Artifacts Files produced by the run (for instance, model weights) mlflow.log_artifacts, mlflow.log_image, mlflow.log_text    This example demonstrates the use of these functions:  import os from random import random, randint from mlflow import log_metric, log_param, log_params, log_artifacts  if __name__ == ""__main__"":     # Log a parameter (key-value pair)     log_param(""config_value"", randint(0, 100))      # Log a dictionary of parameters     log_params({""param1"": randint(0, 100), ""param2"": randint(0, 100)})      # Log a metric; metrics can be updated throughout the run     log_metric(""accuracy"", random() / 2.0)     log_metric(""accuracy"", random() + 0.1)     log_metric(""accuracy"", random() + 0.2)      # Log an artifact (output file)     if not os.path.exists(""outputs""):         os.makedirs(""outputs"")     with open(""outputs/test.txt"", ""w"") as f:         f.write(""hello world!"")     log_artifacts(""outputs"")    If you are using a library that supports autologging, but wish to disable it, you may do so by calling mlflow.autolog(disable=True).  For more details on automatic logging, see Automatic Logging. For more details on the explicit logging API, see Logging functions.    View MLflow runs and experiments  Once you've run your code, you may view the results with MLflow's tracking UI. To start the UI, run:  mlflow ui    And then navigate to http://localhost:5000 in your browser. You will see a page similar to:  You are in the Default experiment, which now contains the tracking data for your run. An experiment is a collection of related runs. The MLflow UI opens to the Table view. The main portion of the window shows a table of runs, with each row representing a single run. The columns show the run name, how long ago it was created, its running time, and so forth. If you select a run name, you will open details for the run, which shows the parameters, metrics, and artifacts of the run.  You can view the history of a metric by opening Metrics and selecting the metric name. For instance, the following image shows a run's Mean Average Precision over time:  From the main page, you can switch between Table view and Chart view. Chart view allows you to compare runs at a glance. For instance, the following image shows the Mean Average Precision of the highest-scoring runs in this experiment:  For more details on the tracking UI, see MLflow Tracking.   Share MLflow runs and experiments  For getting started, the last example stored the tracking data locally. Generally, you will want to use shared storage. Locally, MLflow stores tracking data and artifacts in an mlruns/ subdirectory of where you ran the code. The tracking UI, when run locally, visualizes this. You may also store your data remotely. You can track your runs with a tracking server, on a shared filesystem, with a SQLAlchemy-compatible database, or in a Databricks workspace. To do so:  Call mlflow.set_tracking_uri in your code; or Set the MLFLOW_TRACKING_URI environment variable  A tracking server is a lightweight HTTP server built in to MLflow. You can run a tracking server on a network-accessible server by running:  mlflow server    For instance, if you've run the above command on a machine with IP address 192.168.0.1 and port 5000, you can add tracking data to it either by:  mlflow.set_tracking_uri(""http://192.168.0.1:5000"") mlflow.autolog()  # Or other tracking functions    Or, on your development machine, you can set the MLFLOW_TRACKING_URI environment variable to the URL of that server:  export MLFLOW_TRACKING_URI=http://192.168.0.1:5000    Now, when you run your code, it will send tracking data to the tracking server. You can view the tracking data by navigating to the URI with a browser. There are many options available for the tracking backend. For more details, see MLflow Tracking Servers.  Use MLflow with a Databricks workspace  You need to configure MLflow to use your Databricks workspace (To get started with Databricks, see: Get started: Account and Workspace setup). You will need to know the URL of your Databricks workspace. You can find the URL in the Configuration page of the workspace:  At the command-line, run the following command to configure your experiment:  databricks configure    Set the Databricks Host to the URL of your Databricks workspace, and set the Username and Password to the credentials you use to access the workspace. If you've created an authentication token for your Databricks workspace (databricks tokens create), you can use it instead of your password. Call databricks configure with the -t, \--token option. In your training code, modify the call to mlflow.set_tracking_uri to use Databricks and set the experiment to the path of your experiment in Databricks, replacing user_name and experiment_name with the appropriate values:  mlflow.set_tracking_uri(""databricks"") mlflow.set_experiment(f""/Users/{user_name}/{experiment_name}"")    If the specified experiment does not exist, it will be created. For more on using MLflow with Databricks, see Databricks' documentation on MLflow.    Store a model in MLflow  An MLflow Model is a directory that packages machine learning models and support files in a standard format. The directory contains:  An MLModel file in YAML format specifying the model's flavor (or flavors), dependencies, signature (if supplied), and important metadata; The various files required by the model's flavor(s) to instantiate the model. This will often be a serialized Python object; Files necessary for recreating the model's runtime environment (for instance, a conda.yaml file); and Optionally, an input example  When using autologging, MLflow will automatically log whatever model or models the run creates. You can also log a model manually by calling mlflow.{library_module_name}.log_model. In addition, if you wish to load the model soon, it may be convenient to output the run's ID directly to the console. For that, you'll need the object of type mlflow.ActiveRun for the current run. You get that object by wrapping all of your logging code in a with mlflow.start_run() as run: block. (mlflow.start_run() API reference) For example:  import mlflow from mlflow.models import infer_signature  from sklearn.model_selection import train_test_split from sklearn.datasets import load_diabetes from sklearn.ensemble import RandomForestRegressor  with mlflow.start_run() as run:     # Load the diabetes dataset.     db = load_diabetes()     X_train, X_test, y_train, y_test = train_test_split(db.data, db.target)      # Create and train models.     rf = RandomForestRegressor(n_estimators=100, max_depth=6, max_features=3)     rf.fit(X_train, y_train)      # Use the model to make predictions on the test dataset.     predictions = rf.predict(X_test)     print(predictions)      signature = infer_signature(X_test, predictions)     mlflow.sklearn.log_model(rf, ""model"", signature=signature)      print(f""Run ID: {run.info.run_id}"")    In the case of the sklearn flavor, log_model stores the following files in the artifacts directory of the run's directory on the tracking server:  model/ |-- MLmodel |-- conda.yaml |-- model.pkl |-- python_env.yaml |-- requirements.txt    If you've not called set_tracking_uri or set the MLFLOW_TRACKING_URI environment variable to point to a remote tracking server, this model directory will be under the mlruns directory. For more information, including a list of supported model flavors and storing your own flavor, see Built-In Model Flavors.   Load a model from a specific training run for inference  To load and run a model stored in a previous run, you can use the mlflow.{library_module_name}.load_model function. You'll need the run ID of the run that logged the model. You can find the run ID in the tracking UI:   import mlflow  from sklearn.model_selection import train_test_split from sklearn.datasets import load_diabetes  db = load_diabetes() X_train, X_test, y_train, y_test = train_test_split(db.data, db.target)  model = mlflow.sklearn.load_model(""runs:/d7ade5106ee341e0b4c63a53a9776231"") predictions = model.predict(X_test) print(predictions)    Note that while log_model saves environment-specifying files such as conda.yaml and requirements.txt, load_model does not automatically recreate that environment. To do so, you need to use your preferred method (conda, virtualenv, pip, etc.), using the artifacts saved by log_model. If you serve your model with mlflow models serve, MLflow will automatically recreate the environment. Those commands also accept an --env-manager option for even more control. This is described in detail in Environment Management Tools. In the case of mlflow.pyfunc.spark_udf(), you can use the --env-manager flag to recreate the environment during Spark batch inference.        Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
getting-started/quickstart-2/index.html,"   Documentation  Getting Started with MLflow  Quickstart: Compare runs, choose a model, and deploy it to a REST API       Quickstart: Compare runs, choose a model, and deploy it to a REST API  In this quickstart, you will:  Run a hyperparameter sweep on a training script Compare the results of the runs in the MLflow UI Choose the best run and register it as a model Deploy the model to a REST API Build a container image suitable for deployment to a cloud platform  As an ML Engineer or MLOps professional, you can use MLflow to compare, share, and deploy the best models produced by the team. In this quickstart, you will use the MLflow Tracking UI to compare the results of a hyperparameter sweep, choose the best run, and register it as a model. Then, you will deploy the model to a REST API. Finally, you will create a Docker container image suitable for deployment to a cloud platform.   Set up   Install MLflow. See the introductory quickstart for instructions Run the tracking server: mlflow server    Run a hyperparameter sweep  This example tries to optimize the RMSE metric of a Keras deep learning model on a wine quality dataset. It has two hyperparameters that it tries to optimize: learning-rate and momentum. We will use the Hyperopt library to run a hyperparameter sweep across different values of learning-rate and momentum and record the results in MLflow. Run the hyperparameter sweep, setting the MLFLOW_TRACKING_URI environment variable to the URI of the MLflow tracking server: export MLFLOW_TRACKING_URI=http://localhost:5000   Import the following packages import numpy as np import pandas as pd from hyperopt import STATUS_OK, Trials, fmin, hp, tpe from sklearn.metrics import mean_squared_error from sklearn.model_selection import train_test_split from tensorflow.keras.layers import Dense, Lambda from tensorflow.keras.models import Sequential from tensorflow.keras.optimizers import SGD  import mlflow from mlflow.models import infer_signature   Now load the dataset and split it into training, validation, and test sets. # Load dataset data = pd.read_csv(     ""https://raw.githubusercontent.com/mlflow/mlflow/master/tests/datasets/winequality-white.csv"",     sep="";"", )  # Split the data into training, validation, and test sets train, test = train_test_split(data, test_size=0.25, random_state=42) train_x = train.drop([""quality""], axis=1).values train_y = train[[""quality""]].values.ravel() test_x = test.drop([""quality""], axis=1).values test_y = test[[""quality""]].values.ravel() train_x, valid_x, train_y, valid_y = train_test_split(     train_x, train_y, test_size=0.2, random_state=42 ) signature = infer_signature(train_x, train_y)   Then, define the model architecture and train the model. The train_model function uses MLflow to track the parameters, results, and model itself of each trial as a child run. def train_model(params, train_x, train_y, valid_x, valid_y, test_x, test_y, epochs):     # Define model architecture     model = Sequential()     model.add(         Lambda(lambda x: (x - np.mean(train_x, axis=0)) / np.std(train_x, axis=0))     )     model.add(Dense(64, activation=""relu"", input_shape=(train_x.shape[1],)))     model.add(Dense(1))      # Compile model     model.compile(         optimizer=SGD(lr=params[""lr""], momentum=params[""momentum""]),         loss=""mean_squared_error"",     )      # Train model with MLflow tracking     with mlflow.start_run(nested=True):         # Fit model         model.fit(             train_x,             train_y,             validation_data=(valid_x, valid_y),             epochs=epochs,             verbose=0,         )          # Evaluate the model         predicted_qualities = model.predict(test_x)         rmse = np.sqrt(mean_squared_error(test_y, predicted_qualities))          # Log parameters and results         mlflow.log_params(params)         mlflow.log_metric(""rmse"", rmse)          # Log model         mlflow.tensorflow.log_model(model, ""model"", signature=signature)          return {""loss"": rmse, ""status"": STATUS_OK, ""model"": model}   The objective function takes in the hyperparameters and returns the results of the train_model function for that set of hyperparameters. def objective(params):     # MLflow will track the parameters and results for each run     result = train_model(         params,         train_x=train_x,         train_y=train_y,         valid_x=valid_x,         valid_y=valid_y,         test_x=test_x,         test_y=test_y,         epochs=32,  # Or any other number of epochs     )     return result   Next, we will define the search space for Hyperopt. In this case, we want to try different values of learning-rate and momentum. space = {     ""lr"": hp.loguniform(""lr"", np.log(1e-5), np.log(1e-1)),     ""momentum"": hp.uniform(""momentum"", 0.0, 1.0), }   Finally, we will run the hyperparameter sweep using Hyperopt, passing in the objective function and search space. Hyperopt will try different hyperparameter combinations and return the results of the best one. We will store the best parameters, model, and rmse in MLflow. with mlflow.start_run():     # Conduct the hyperparameter search using Hyperopt     trials = Trials()     best = fmin(         fn=objective,         space=space,         algo=tpe.suggest,         max_evals=12,  # Set to a higher number to explore more hyperparameter configurations         trials=trials,     )      # Fetch the details of the best run     best_run = sorted(trials.results, key=lambda x: x[""loss""])[0]      # Log the best parameters, loss, and model     mlflow.log_params(best)     mlflow.log_metric(""rmse"", best_run[""loss""])     mlflow.tensorflow.log_model(best[""model""], ""model"", signature=signature)      # Print out the best parameters and corresponding loss     print(f""Best parameters: {best}"")     print(f""Best rmse: {best_run['loss']}"")     Compare the results  Open the MLflow UI in your browser at the MLFLOW_TRACKING_URI. You should see a nested list of runs. In the default Table view, choose the Columns button and add the Metrics | test_rmse column and the Parameters | lr and Parameters | momentum column. To sort by RMSE ascending, click the test_rmse column header. The best run typically has an RMSE on the test dataset of ~0.70. You can see the parameters of the best run in the Parameters column.  Choose Chart view. Choose the Parallel coordinates graph and configure it to show the lr and momentum coordinates and the test_rmse metric. Each line in this graph represents a run and associates each hyperparameter evaluation run's parameters to the evaluated error metric for the run. The red graphs on this graph are runs that fared poorly. The lowest one is a baseline run with both lr and momentum set to 0.0. That baseline run has an RMSE of ~0.89. The other red lines show that high momentum can also lead to poor results with this problem and architecture. The graphs shading towards blue are runs that fared better. Hover your mouse over individual runs to see their details.   Register your best model  Choose the best run and register it as a model. In the Table view, choose the best run. In the Run Detail page, open the Artifacts section and select the Register Model button. In the Register Model dialog, enter a name for the model, such as wine-quality, and click Register. Now, your model is available for deployment. You can see it in the Models page of the MLflow UI. Open the page for the model you just registered. You can add a description for the model, add tags, and easily navigate back to the source run that generated this model. You can also transition the model to different stages. For example, you can transition the model to Staging to indicate that it is ready for testing. You can transition it to Production to indicate that it is ready for deployment. Transition the model to Staging by choosing the Stage dropdown:    Serve the model locally  MLflow allows you to easily serve models produced by any run or model version. You can serve the model you just registered by running: mlflow models serve -m ""models:/wine-quality/Staging"" --port 5002   (Note that specifying the port as above will be necessary if you are running the tracking server on the same machine at the default port of 5000.) You could also have used a runs:/<run_id> URI to serve a model, or any supported URI described in Artifact Stores. To test the model, you can send a request to the REST API using the curl command: curl -d '{""dataframe_split"": { ""columns"": [""fixed acidity"",""volatile acidity"",""citric acid"",""residual sugar"",""chlorides"",""free sulfur dioxide"",""total sulfur dioxide"",""density"",""pH"",""sulphates"",""alcohol""], ""data"": [[7,0.27,0.36,20.7,0.045,45,170,1.001,3,0.45,8.8]]}}' \ -H 'Content-Type: application/json' -X POST localhost:5002/invocations   Inferencing is done with a JSON POST request to the invocations path on localhost at the specified port. The columns key specifies the names of the columns in the input data. The data value is a list of lists, where each inner list is a row of data. For brevity, the above only requests one prediction of wine quality (on a scale of 3-8). The response is a JSON object with a predictions key that contains a list of predictions, one for each row of data. In this case, the response is: {""predictions"": [{""0"": 5.310967445373535}]}   The schema for input and output is available in the MLflow UI in the Artifacts | Model description. The schema is available because the train.py script used the mlflow.infer_signature method and passed the result to the mlflow.log_model method. Passing the signature to the log_model method is highly recommended, as it provides clear error messages if the input request is malformed.   Build a container image for your model  Most routes toward deployment will use a container to package your model, its dependencies, and relevant portions of the runtime environment. You can use MLflow to build a Docker image for your model. mlflow models build-docker --model-uri ""models:/wine-quality/1"" --name ""qs_mlops""   This command builds a Docker image named qs_mlops that contains your model and its dependencies. The model-uri in this case specifies a version number (/1) rather than a lifecycle stage (/staging), but you can use whichever integrates best with your workflow. It will take several minutes to build the image. Once it completes, you can run the image to provide real-time inferencing locally, on-prem, on a bespoke Internet server, or cloud platform. You can run it locally with: docker run -p 5002:8080 qs_mlops   This Docker run command runs the image you just built and maps port 5002 on your local machine to port 8080 in the container. You can now send requests to the model using the same curl command as before: curl -d '{""dataframe_split"": {""columns"": [""fixed acidity"",""volatile acidity"",""citric acid"",""residual sugar"",""chlorides"",""free sulfur dioxide"",""total sulfur dioxide"",""density"",""pH"",""sulphates"",""alcohol""], ""data"": [[7,0.27,0.36,20.7,0.045,45,170,1.001,3,0.45,8.8]]}}' -H 'Content-Type: application/json' -X POST localhost:5002/invocations     Deploying to a cloud platform  Virtually all cloud platforms allow you to deploy a Docker image. The process varies considerably, so you will have to consult your cloud provider's documentation for details. In addition, some cloud providers have built-in support for MLflow. For instance:  Azure ML Databricks Amazon SageMaker Google Cloud  all support MLflow. Cloud platforms generally support multiple workflows for deployment: command-line, SDK-based, and Web-based. You can use MLflow in any of these workflows, although the details will vary between platforms and versions. Again, you will need to consult your cloud provider's documentation for details.        Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
llms/index.html,"   Documentation  LLMs       LLMs  LLMs, or Large Language Models, have rapidly become a cornerstone in the machine learning domain, offering immense capabilities ranging from natural language understanding to code generation and more. However, harnessing the full potential of LLMs often involves intricate processes, from interfacing with multiple providers to fine-tuning specific models to achieve desired outcomes. Such complexities can easily become a bottleneck for developers and data scientists aiming to integrate LLM capabilities into their applications. MLflow's Support for LLMs aims to alleviate these challenges by introducing a suite of features and tools designed with the end-user in mind:  MLflow AI Gateway    Serving as a unified interface, the MLflow AI Gateway simplifies interactions with multiple LLM providers, such as OpenAI, MosaicML, Cohere, Anthropic, PaLM 2, AWS Bedrock, and AI21 Labs. In addition to supporting the most popular SaaS LLM providers, the AI Gateway provides an integration to MLflow model serving, allowing you to serve your own LLM or a fine-tuned foundation model within your own serving infrastructure.  Note The MLflow AI Gateway is in active development and has been marked as Experimental. APIs may change as this new feature is refined and its functionality is expanded based on feedback.   Benefits of the MLflow AI Gateway   Unified Endpoint: No more juggling between multiple provider APIs. Simplified Integrations: One-time setup, no repeated complex integrations. Secure Credential Management:  Centralized storage prevents scattered API keys. No hardcoding or user-handled keys.   Consistent API Experience:  Uniform API across all providers. Easy-to-use REST endpoints and Client API.   Seamless Provider Swapping:  Swap providers without touching your code. Zero downtime provider, model, or route swapping.      Explore the Native Provider integrations  The MLflow AI Gateway supports a large range of foundational models from popular SaaS model vendors, as well as providing a means of self-hosting your own open source model via an integration with MLflow model serving. To learn more about how to get started using the MLflow AI Gateway to simplify the configuration and management of your LLM serving needs, select the provider that you're interested in below:                                                   Getting Started Examples for each Provider  If you're interested in learning about how to set up the MLflow AI Gateway for a specific provider, follow the links below for our up-to-date documentation on GitHub. Each link will take you to a README file that will explain how to set up a route for the provider. In the same directory as the README, you will find a runnable example of how to query the routes that the example creates, providing you with a quick reference for getting started with your favorite provider!  OpenAI quickstart MosaicML quickstart Anthropic quickstart Cohere quickstart MLflow quickstart AWS Bedrock quickstart AI21 Labs quickstart PaLM 2 quickstart Azure OpenAI quickstart Hugging Face Text Generation Interface (TGI) quickstart   Note The MLflow and Hugging Face TGI providers are for self-hosted LLM serving of either foundation open-source LLM models, fine-tuned open-source LLM models, or your own custom LLM. The example documentation for these providers will show you how to get started with these, using free-to-use open-source models from the Hugging Face Hub.     LLM Evaluation    Navigating the vast landscape of Large Language Models (LLMs) can be daunting. Determining the right model, prompt, or service that aligns with a project's needs is no small feat. Traditional machine learning evaluation metrics often fall short when it comes to assessing the nuanced performance of generative models. Enter MLflow LLM Evaluation. This feature is designed to simplify the evaluation process, offering a streamlined approach to compare foundational models, providers, and prompts.  Benefits of MLflow's LLM Evaluation   Simplified Evaluation: Navigate the LLM space with ease, ensuring the best fit for your project with standard metrics that can be used to compare generated text. Use-Case Specific Metrics: Leverage MLflow's mlflow.evaluate() API for a high-level, frictionless evaluation experience. Customizable Metrics: Beyond the provided metrics, MLflow supports a plugin-style for custom scoring, enhancing the evaluation's flexibility. Comparative Analysis: Effortlessly compare foundational models, providers, and prompts to make informed decisions. Deep Insights: Dive into the intricacies of generative models with a comprehensive suite of LLM-relevant metrics.  MLflow's LLM Evaluation is designed to bridge the gap between traditional machine learning evaluation and the unique challenges posed by LLMs.    Prompt Engineering UI    Effective utilization of LLMs often hinges on crafting the right prompts. The development of a high-quality prompt is an iterative process of trial and error, where subsequent experimentation is not guaranteed to result in cumulative quality improvements. With the volume and speed of iteration through prompt experimentation, it can quickly become very overwhelming to remember or keep a history of the state of different prompts that were tried. Serving as a powerful tool for prompt engineering, the MLflow Prompt Engineering UI revolutionizes the way developers interact with and refine LLM prompts.  Benefits of the MLflow Prompt Engineering UI   Iterative Development: Streamlined process for trial and error without the overwhelming complexity. UI-Based Prototyping: Prototype, iterate, and refine prompts without diving deep into code. Accessible Engineering: Makes prompt engineering more user-friendly, speeding up experimentation. Optimized Configurations: Quickly hone in on the best model configurations for tasks like question answering or document summarization. Transparent Tracking:  Every model iteration and configuration is meticulously tracked. Ensures reproducibility and transparency in your development process.     Note The MLflow Prompt Engineering UI is in active development and has been marked as Experimental. Features and interfaces may evolve as feedback is gathered and the tool is refined.     Native MLflow Flavors for LLMs  Harnessing the power of LLMs becomes effortless with flavors designed specifically for working with LLM libraries and frameworks.  Benefits of MLflow's Native Flavors for LLMs   Support for Popular Packages:  Native integration with packages like transformers, sentence-transformers, open-ai , and langchain. Standardized interfaces for tasks like saving, logging, and managing inference configurations.   PyFunc Compatibility:  Load models as PyFuncs for broad compatibility across serving infrastructures. Strengthens the MLOps process for LLMs, ensuring smooth deployments.   Cohesive Ecosystem:  All essential tools and functionalities consolidated under MLflow. Focus on deriving value from LLMs without getting bogged down by interfacing and optimization intricacies.      Explore the Native LLM Flavors  Select the integration below to read the documentation on how to leverage MLflow's native integration with these popular libraries:                          Native Integration Examples  If you'd like to directly explore code examples for how to get started with using our official library integrations, you can navigate directly to our up-to-date examples on GitHub below:  transformers   Simple Text Generation Example A Conversational Model Example Component Logging with Transformers Audio Transcription with Whisper Fine-tuning a Text Classification Model    sentence-transformers   Text Encoding Example    langchain   Logging and using a Chain Logging and using an Agent Logging and using a Retriever Chain 1 Logging and using a Retrieval QA Chain 1     1 Demonstrates the use of Retrieval Augmented Generation (RAG) using a Vector Store  openai   Using a Completions endpoint Using a Chat endpoint Performing Embeddings Generation Using OpenAI on a Spark DataFrame for Batch Processing Using Azure OpenAI        LLM Tracking in MLflow    Empowering developers with advanced tracking capabilities, the MLflow LLM Tracking System stands out as the premier solution for managing and analyzing interactions with Large Language Models (LLMs).  Benefits of the MLflow LLM Tracking System   Robust Interaction Management: Comprehensive tracking of every LLM interaction for maximum insight. Tailor-Made for LLMs:  Unique features specifically designed for LLMs. From logging prompts to tracking dynamic data, MLflow has it covered.   Deep Model Insight:  Introduces 'predictions' as a core entity, alongside the existing artifacts, parameters, and metrics. Gain unparalleled understanding of text-generating model behavior and performance.   Clarity and Repeatability:  Ensures consistent and transparent tracking across all LLM interactions. Facilitates informed decision-making and optimization in LLM deployment and utilization.       Tutorials and Use Case Guides for LLMs in MLflow  Interested in learning how to leverage MLflow for your LLM projects? Look in the tutorials and guides below to learn more about interesting use cases that could help to make your journey into leveraging LLMs a bit easier!                            Evaluating LLMs                                       Learn how to evaluate LLMs with MLflow.                                           Using Custom PyFunc with LLMs                                       Explore the nuances of packaging, customizing, and deploying advanced LLMs in MLflow using custom PyFuncs.                                           Question Generation for RAG                                       Learn how to leverage LLMs to generate a question dataset for use in Retrieval Augmented Generation applications.                            Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
llms/prompt-engineering/index.html,"   Documentation  LLMs  Prompt Engineering UI (Experimental)       Prompt Engineering UI (Experimental)  Starting in MLflow 2.7, the MLflow Tracking UI provides a best-in-class experience for prompt engineering. With no code required, you can try out multiple LLMs from the AI Gateway, parameter configurations, and prompts to build a variety of models for question answering, document summarization, and beyond. Using the embedded Evaluation UI, you can also evaluate multiple models on a set of inputs and compare the responses to select the best one. Every model created with the prompt engineering UI is stored in the MLflow Model format and can be deployed for batch or real time inference. All configurations (prompt templates, choice of LLM, parameters, etc.) are tracked as MLflow Runs.  Quickstart  The following guide will get you started with MLflow's UI for prompt engineering.  Step 1: Create an AI Gateway Completions or Chat Route  To use the prompt engineering UI, you need to create one or more AI Gateway completions or chat Routes. Follow the AI Gateway Quickstart guide to easily create a Route in less than five minutes. If you already have access to an AI Gateway Route of type llm/v1/completions or llm/v1/chat, you can skip this step. mlflow gateway start --config-path config.yaml --port 7000     Step 2: Connect the AI Gateway to your MLflow Tracking Server  The prompt engineering UI also requires a connection between the AI Gateway and the MLflow Tracking Server. To connect the AI Gateway with the MLflow Tracking Server, simply set the MLFLOW_GATEWAY_URI environment variable in the environment where the server is running and restart the server. For example, if the AI Gateway is running at http://localhost:7000, you can start an MLflow Tracking Server in a shell on your local machine and connect it to the AI Gateway using the mlflow server command as follows: export MLFLOW_GATEWAY_URI=""http://localhost:7000"" mlflow server --port 5000     Step 3: Create or find an MLflow Experiment  Next, open an existing MLflow Experiment in the MLflow UI, or create a new experiment.      Step 4: Create a run with prompt engineering  Once you have opened the Experiment, click the New Run button and select using Prompt Engineering. This will open the prompt engineering playground where you can try out different LLMs, parameters, and prompts.     Step 5: Select your Route and evaluate the example prompt  Next, click the Select route dropdown and select the AI Gateway completions Route you created in Step 1. Then, click the Evaluate button to test out an example prompt engineering use case for generating product advertisements. MLflow will embed the specified stock_type input variable value - ""books"" - into the specified prompt  template and send it to the LLM associated with the AI Gateway route with the configured temperature (currently 0.01) and max_tokens (currently 1000). The LLM response will appear in the Output section.      Step 6: Try a prompt of your choosing  Replace the prompt template from the previous step with a prompt template of your choosing. Prompts can define multiple variables. For example, you can use the following prompt template to instruct the LLM to answer questions about the MLflow documentation: Read the following article from the MLflow documentation that appears between triple backticks. Then, answer the question about the documentation that appears between triple quotes. Include relevant links and code examples in your answer.  ```{{article}}```  """""" {{question}} """"""   Then, fill in the input variables. For example, in the MLflow documentation use case, the article input variable can be set to the contents of https://mlflow.org/docs/latest/tracking.html#logging-data-to-runs and the question input variable can be set to ""How do I create a new MLflow Run using the Python API?"". Finally, click the Evaluate button to see the new output. You can also try choosing a larger value of temperature to observe how the LLM's output changes.      Step 7: Capture your choice of LLM, prompt template, and parameters as an MLflow Run  Once you're satisfied with your chosen prompt template and parameters, click the Create Run button to store this information, along with your choice of LLM, as an MLflow Run. This will create a new Run with the prompt template, parameters, and choice of LLM stored as Run params. It will also automatically create an MLflow Model with this information that can be used for batch or real-time inference.  To view this information, click the Run name to open the Run page:       You can also see the parameters and compare them with other configurations by opening the Table view tab:       After your Run is created, MLflow will open the Evaluation tab where you can see your latest playground input & output and try out additional inputs:          Step 8: Try new inputs  To test the behavior of your chosen LLM, prompt template, and parameters on a new inputs:  Click the Add Row button and fill in a value(s) your prompt template's input variable(s). For example, in the MLflow documentation use case, you can try asking a question unrelated to MLflow to see how the LLM responds. This is important to ensure that the application is robust to irrelevant inputs.      Then, click the Evaluate button to see the output.       Finally, click the Save button to store the new inputs and output.          Step 9: Adjust your prompt template and create a new Run  As you try additional inputs, you might discover scenarios where your choice of LLM, prompt template, and parameters doesn't perform as well as you would like. For example, in the MLflow documentation use case, the LLM still attempts to answer irrelevant questions about MLflow Projects even if the answer does not appear in the specified article.  To improve performance, create a new Run by selecting the Duplicate run option from the context menu. For example, in the MLflow documentation use case, adding the following text to the prompt template helps improve robustness to irrelevant questions: If the question does not relate to the article, respond exactly with the phrase ""I do not know how to answer that question."" Do not include any additional text in your response.       Then, from the prompt engineering playground, adjust the prompt template (and / or choice of LLM and parameters), evaluate an input, and click the Create Run button to create a new Run.          Step 10: Evaluate the new prompt template on previous inputs  Now that you've made an adjustment to your prompt template, it's important to make sure that the new template performs well on the previous inputs and compare the outputs with older configurations.  From the Evaluation tab, click the Evaluate all button next to the new Run to evaluate all of the previous inputs.     Click the Save button to store the results.        Step 11: Load evaluation data programmatically  All of the inputs and outputs produced by the MLflow prompt engineering UI and Evaluation UI are stored as artifacts in MLflow Runs. They can be accessed programmatically using the mlflow.load_table() API as follows:  import mlflow  mlflow.set_experiment(""/Path/to/your/prompt/engineering/experiment"")  # Load input and output data across all Runs (configurations) as a Pandas DataFrame inputs_outputs_pdf = mlflow.load_table(     # All inputs and outputs created from the MLflow UI are stored in an artifact called     # ""eval_results_table.json""     artifact_file=""eval_results_table.json"",     # Include the run ID as a column in the table to distinguish inputs and outputs     # produced by different runs     extra_columns=[""run_id""], ) # Optionally convert the Pandas DataFrame to Spark where it can be stored as a Delta # table or joined with existing Delta tables inputs_outputs_sdf = spark.createDataFrame(inputs_outputs_pdf)      Step 12: Generate predictions programmatically  Once you have found a configuration of LLM, prompt template, and parameters that performs well, you can generate predictions using the corresponding MLflow Model in a Python environment of your choosing, or you can deploy it for real-time serving.  To load the MLflow Model in a notebook for batch inference, click on the Run's name to open the Run Page and select the model directory in the Artifact Viewer. Then, copy the first few lines of code from the Predict on a Pandas DataFrame section and run them in a Python environment of your choosing, for example:    import mlflow  logged_model = ""runs:/8451075c46964f82b85fe16c3d2b7ea0/model""  # Load model as a PyFuncModel. loaded_model = mlflow.pyfunc.load_model(logged_model)    Then, to generate predictions, call the predict() method and pass in a dictionary of input variables. For example: article_text = """""" An MLflow Project is a format for packaging data science code in a reusable and reproducible way. The MLflow Projects component includes an API and command-line tools for running projects, which also integrate with the Tracking component to automatically record the parameters and git commit of your source code for reproducibility.  This article describes the format of an MLflow Project and how to run an MLflow project remotely using the MLflow CLI, which makes it easy to vertically scale your data science code. """""" question = ""What is an MLflow project?""  loaded_model.predict({""article"": article_text, ""question"": question})   For more information about deployment for real-time serving with MLflow, see the instructions below.     Step 13: Perform metric-based evaluation of your model's outputs  If you'd like to assess your model's performance on specific metrics, MLflow provides the mlflow.evaluate() API. Let's evaluate our model on some pre-defined metrics for text summarization:  import mlflow import pandas as pd  logged_model = ""runs:/840a5c43f3fb46f2a2059b761557c1d0/model""  article_text = """""" An MLflow Project is a format for packaging data science code in a reusable and reproducible way. The MLflow Projects component includes an API and command-line tools for running projects, which also integrate with the Tracking component to automatically record the parameters and git commit of your source code for reproducibility.  This article describes the format of an MLflow Project and how to run an MLflow project remotely using the MLflow CLI, which makes it easy to vertically scale your data science code. """""" question = ""What is an MLflow project?""  data = pd.DataFrame(     {         ""article"": [article_text],         ""question"": [question],         ""ground_truth"": [             article_text         ],  # used for certain evaluation metrics, such as ROUGE score     } )  with mlflow.start_run():     results = mlflow.evaluate(         model=logged_model,         data=data,         targets=""ground_truth"",         model_type=""text-summarization"",     )  eval_table = results.tables[""eval_results_table""] print(f""See evaluation table below: \n{eval_table}"")    The evaluation results can also be viewed in the MLflow Evaluation UI:      The mlflow.evaluate() API also supports custom metrics, static dataset evaluation, and much more. For a more in-depth guide, see MLflow LLM Evaluate.    Deployment for real-time serving  Once you have found a configuration of LLM, prompt template, and parameters that performs well, you can deploy the corresponding MLflow Model for real-time serving as follows:  Register your model with the MLflow Model Registry. The following example registers an MLflow Model created from the Quickstart as Version 1 of the Registered Model named ""mlflow_docs_qa_model"". mlflow.register_model(     model_uri=""runs:/8451075c46964f82b85fe16c3d2b7ea0/model"",     name=""mlflow_docs_qa_model"", )    Define the following environment variables in the environment where you will run your MLflow Model Server, such as a shell on your local machine:  MLFLOW_GATEWAY_URI: The URL of the MLflow AI Gateway   Use the mlflow models serve command to start the MLflow Model Server. For example, running the following command from a shell on your local machine will serve the model on port 8000: mlflow models serve --model-uri models:/mlflow_docs_qa_model/1 --port 8000    Once the server has been started, it can be queried via REST API call. For example:  input=' {     ""dataframe_records"": [         {             ""article"": ""An MLflow Project is a format for packaging data science code..."",             ""question"": ""What is an MLflow Project?""         }     ] }'  echo $input | curl \   -s \   -X POST \   https://localhost:8000/invocations   -H 'Content-Type: application/json' \   -d @-   where article and question are replaced with the input variable(s) from your prompt template.           Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
llms/gateway/index.html,"   Documentation  LLMs  MLflow AI Gateway (Experimental)       MLflow AI Gateway (Experimental)   Warning The MLflow AI Gateway is a new, experimental feature. It is subject to modification, feature improvements, or feature removal without advance notice.  The MLflow AI Gateway service is a powerful tool designed to streamline the usage and management of various large language model (LLM) providers, such as OpenAI and Anthropic, within an organization. It offers a high-level interface that simplifies the interaction with these services by providing a unified endpoint to handle specific LLM related requests. A major advantage of using the MLflow AI Gateway service is its centralized management of API keys. By storing these keys in one secure location, organizations can significantly enhance their security posture by minimizing the exposure of sensitive API keys throughout the system. It also helps to prevent exposing these keys within code or requiring end-users to manage keys safely. The gateway is designed to be flexible and adaptable, capable of easily defining and managing routes by updating the configuration file. This enables the easy incorporation of new LLM providers or provider LLM types into the system without necessitating changes to applications that interface with the gateway. This level of adaptability makes the MLflow AI Gateway Service an invaluable tool in environments that require agility and quick response to changes. This simplification and centralization of language model interactions, coupled with the added layer of security for API key management, make the MLflow AI Gateway service an ideal choice for organizations that use LLMs on a regular basis.    Tutorials and Guides  If you're interested in diving right in to a step by step guide that will get you up and running with the MLflow AI Gateway as fast as possible, the guides below will be your best first stop. View the AI Gateway Getting Started Guide  Quickstart  The following guide will assist you in getting up and running, using a 3-route configuration to OpenAI services for chat, completions, and embeddings.  Step 1: Install the MLflow AI Gateway service  First, you need to install the MLflow AI Gateway service on your machine. You can do this using pip from PyPI or from the MLflow repository.  Installing from PyPI  pip install 'mlflow[gateway]'      Step 2: Set the OpenAI API Key(s) for each provider  The Gateway service needs to communicate with the OpenAI API. To do this, it requires an API key. You can create an API key from the OpenAI dashboard. For this example, we're only connecting with OpenAI. If there are additional providers within the configuration, these keys will need to be set as well. Once you have the key, you can set it as an environment variable in your terminal: export OPENAI_API_KEY=your_api_key_here   This sets a temporary session-based environment variable. For production use cases, it is advisable to store this key in the .bashrc or .zshrc files so that the key doesn't have to be re-entered upon system restart.   Step 3: Create a Gateway Configuration File  Next, you need to create a Gateway configuration file. This is a YAML file where you specify the routes that the Gateway service should expose. Let's create a file with three routes using OpenAI as a provider: completions, chat, and embeddings. For details about the configuration file's parameters (including parameters for other providers besides OpenAI), see the AI Gateway Configuration Details section below. routes:   - name: completions     route_type: llm/v1/completions     model:       provider: openai       name: gpt-3.5-turbo       config:         openai_api_key: $OPENAI_API_KEY    - name: chat     route_type: llm/v1/chat     model:       provider: openai       name: gpt-3.5-turbo       config:         openai_api_key: $OPENAI_API_KEY    - name: embeddings     route_type: llm/v1/embeddings     model:       provider: openai       name: text-embedding-ada-002       config:         openai_api_key: $OPENAI_API_KEY   Save this file to a location on the system that is going to be running the MLflow AI Gateway server.   Step 4: Start the Gateway Service  You're now ready to start the Gateway service! Use the MLflow AI Gateway start command and specify the path to your configuration file: mlflow gateway start --config-path config.yaml --port {port} --host {host} --workers {worker count}   The configuration file can also be set using the MLFLOW_GATEWAY_CONFIG_PATH environment variable: export MLFLOW_GATEWAY_CONFIG_PATH=/path/to/config.yaml   If you do not specify the host, a localhost address will be used. If you do not specify the port, port 5000 will be used. The worker count for gunicorn defaults to 2 workers.   Step 5: Access the Interactive API Documentation  The MLflow AI Gateway service provides an interactive API documentation endpoint that you can use to explore and test the exposed routes. Navigate to http://{host}:{port}/ (or http://{host}:{port}/docs) in your browser to access it. The docs endpoint allow for direct interaction with the routes and permits submitting actual requests to the provider services by click on the "try it now" option within the endpoint definition entry.   Step 6: Send Requests Using the Fluent API  For information on formatting requirements and how to pass parameters, see Querying the AI Gateway. Here's an example of how to send a chat request using the Fluent API : from mlflow.gateway import query, set_gateway_uri  set_gateway_uri(gateway_uri=""http://localhost:5000"")  response = query(     ""chat"",     {""messages"": [{""role"": ""user"", ""content"": ""What is the best day of the week?""}]}, )  print(response)   Note: Remember to change the uri definition to the actual uri of your Gateway server. The returned response will be in this data structure (the actual content and token values will likely be different): {     ""candidates"": [         {             ""message"": {                 ""role"": ""assistant"",                 ""content"": ""\n\nIt's hard to say what the best day of the week is."",             },             ""metadata"": {""finish_reason"": ""stop""},         }     ],     ""metadata"": {         ""input_tokens"": 13,         ""output_tokens"": 15,         ""total_tokens"": 28,         ""model"": ""gpt-3.5-turbo-0301"",         ""route_type"": ""llm/v1/chat"",     }, }     Step 7: Send Requests Using the Client API  See the Client API section for further information.   Step 8: Send Requests to Routes via REST API  You can now send requests to the exposed routes. See the REST examples for guidance on request formatting.   Step 9: Compare Provider Models  Here's an example of adding a new model from a provider to determine which model instance is better for a given use case. Firstly, update the MLflow AI Gateway config YAML file with the additional route definition to test: routes:   - name: completions     route_type: llm/v1/completions     model:       provider: openai       name: gpt-3.5-turbo       config:         openai_api_key: $OPENAI_API_KEY   - name: completions-gpt4     route_type: llm/v1/completions     model:       provider: openai       name: gpt-4       config:         openai_api_key: $OPENAI_API_KEY   This updated configuration adds a new completions route completions-gpt4 while still preserving the original completions route that was configured with the gpt-3.5-turbo  model. Once the configuration file is updated, simply save your changes. The Gateway will automatically create the new route with zero downtime. At this point, you may use the Fluent API to query both routes with similar prompts to decide which model performs best for your use case. If you no longer need a route, you can delete it from the configuration YAML and save your changes. The AI Gateway will automatically remove the route.   Step 10: Use AI Gateway routes for model development  Now that you have created several AI Gateway routes, you can create MLflow Models that query these routes to build application-specific logic using techniques like prompt engineering. For more information, see AI Gateway and MLflow Models.    Concepts  There are several concepts that are referred to within the MLflow AI Gateway APIs, the configuration definitions, examples, and documentation. Becoming familiar with these terms will help in configuring new endpoints (routes) and ease the use of the interface APIs for the AI Gateway.  Providers  The MLflow AI Gateway is designed to support a variety of model providers. A provider represents the source of the machine learning models, such as OpenAI, Anthropic, and so on. Each provider has its specific characteristics and configurations that are encapsulated within the model part of a route in the MLflow AI Gateway.  Supported Provider Models  The table below presents a non-exhaustive list of models and a corresponding route type within the MLflow AI Gateway. With the rapid development of LLMs, there is no guarantee that this list will be up to date at all times. However, the associations listed below can be used as a helpful guide when configuring a given route for any newly released model types as they become available with a given provider. N/A means that the provider or the AI Gateway implementation currently doesn't support the route type.         Provider Routes   llm/v1/completions llm/v1/chat llm/v1/embeddings    OpenAI  gpt-3.5-turbo gpt-4    gpt-3.5-turbo gpt-4    text-embedding-ada-002    MosaicML  mpt-7b-instruct mpt-30b-instruct llama2-70b-chat†    llama2-70b-chat†    instructor-large instructor-xl    Anthropic  claude-1 claude-1.3-100k claude-2   N/A N/A  Cohere  command command-light-nightly   N/A  embed-english-v2.0 embed-multilingual-v2.0    Azure OpenAI  text-davinci-003 gpt-35-turbo    gpt-35-turbo gpt-4    text-embedding-ada-002    PaLM  text-bison-001    chat-bison-001    embedding-gecko-001    MLflow  MLflow served models*    MLflow served models*    MLflow served models**    HuggingFace TGI N/A  HF TGI Models   N/A  AI21 Labs  j2-ultra j2-mid j2-light   N/A N/A  AWS Bedrock  Amazon Titan Third-party providers   N/A N/A    † Llama 2 is licensed under the LLAMA 2 Community License, Copyright © Meta Platforms, Inc. All Rights Reserved. Within each model block in the configuration file, the provider field is used to specify the name of the provider for that model. This is a string value that needs to correspond to a provider the MLflow AI Gateway supports.  Note * MLflow Model Serving will only work for chat or completions if the output return is in a route-compatible format. The response must conform to either an output of {""predictions"": str} or {""predictions"": {""candidates"": str}}. Any complex return type from a model that does not conform to these structures will raise an exception at query time. ** Embeddings support is only available for models whose response signatures conform to the structured format of {""predictions"": List[float]} or {""predictions"": List[List[float]]}. Any other return type will raise an exception at query time. FeatureExtractionPipeline in transformers and models using the sentence_transformers flavor will return the correct data structures for the embeddings route.  Here's an example of a provider configuration within a route: routes:   - name: chat     route_type: llm/v1/chat     model:       provider: openai       name: gpt-4       config:         openai_api_key: $OPENAI_API_KEY   In the above configuration, openai is the provider for the model. As of now, the MLflow AI Gateway supports the following providers:  mosaicml: This is used for models offered by MosaicML. openai: This is used for models offered by OpenAI and the Azure integrations for Azure OpenAI and Azure OpenAI with AAD. anthropic: This is used for models offered by Anthropic. cohere: This is used for models offered by Cohere. palm: This is used for models offered by PaLM. huggingface text generation inference: This is used for models deployed using Huggingface Text Generation Inference. ai21labs: This is used for models offered by AI21 Labs. bedrock: This is used for models offered by AWS Bedrock.  More providers are being added continually. Check the latest version of the MLflow AI Gateway Docs for the most up-to-date list of supported providers. Remember, the provider you specify must be one that the MLflow AI Gateway supports. If the provider is not supported, the Gateway will return an error when trying to route requests to that provider.    Routes  Routes are central to how the MLflow AI Gateway functions. Each route acts as a proxy endpoint for the user, forwarding requests to the underlying Models and Providers specified in the configuration file. A route in the MLflow AI Gateway consists of the following fields:  name: This is the unique identifier for the route. This will be part of the URL when making API calls via the MLflow AI Gateway. type: The type of the route corresponds to the type of language model interaction you desire. For instance, llm/v1/completions for text completion operations, llm/v1/embeddings for text embeddings, and llm/v1/chat for chat operations. model: Defines the model to which this route will forward requests. The model contains the following details:   provider: Specifies the name of the provider for this model. For example, openai for OpenAI's GPT-3.5 models. name: The name of the model to use. For example, gpt-3.5-turbo for OpenAI's GPT-3.5-Turbo model. config: Contains any additional configuration details required for the model. This includes specifying the API base URL and the API key.     Here's an example of a route configuration: routes:   - name: completions     type: chat/completions     model:       provider: openai       name: gpt-3.5-turbo       config:         openai_api_key: $OPENAI_API_KEY   In the example above, a request sent to the completions route would be forwarded to the gpt-3.5-turbo model provided by openai. The routes in the configuration file can be updated at any time, and the MLflow AI Gateway will automatically update its available routes without requiring a restart. This feature provides you with the flexibility to add, remove, or modify routes as your needs change. It enables 'hot-swapping' of routes, providing a seamless experience for any applications or services that interact with the MLflow AI Gateway. When defining routes in the configuration file, ensure that each name is unique to prevent conflicts. Duplicate route names will raise an MlflowException.   Models  The model section within a route specifies which model to use for generating responses. This configuration block needs to contain a name field which is used to specify the exact model instance to be used. Additionally, a provider needs to be specified, one that you have an authenticated access api key for. Different endpoint types are often associated with specific models. For instance, the llm/v1/chat and llm/v1/completions endpoints are generally associated with conversational models, while llm/v1/embeddings endpoints would typically be associated with embedding or transformer models. The model you choose should be appropriate for the type of endpoint specified. Here's an example of a model name configuration within a route: routes:   - name: embeddings     route_type: llm/v1/embeddings     model:       provider: openai       name: text-embedding-ada-002       config:         openai_api_key: $OPENAI_API_KEY   In the above configuration, text-embedding-ada-002 is the model used for the embeddings endpoint. When specifying a model, it is critical that the provider supports the model you are requesting. For instance, openai as a provider supports models like text-embedding-ada-002, but other providers may not. If the model is not supported by the provider, the MLflow AI Gateway will return an HTTP 4xx error when trying to route requests to that model.  Important Always check the latest documentation of the specified provider to ensure that the model you want to use is supported for the type of endpoint you're configuring.  Remember, the model you choose directly affects the results of the responses you'll get from the API calls. Therefore, choose a model that fits your use-case requirements. For instance, for generating conversational responses, you would typically choose a chat model. Conversely, for generating embeddings of text, you would choose an embedding model.    Configuring the AI Gateway  The MLflow AI Gateway service relies on a user-provided configuration file, written in YAML, that defines the routes and providers available to the service. The configuration file dictates how the gateway interacts with various language model providers and determines the end-points that users can access.  AI Gateway Configuration  The configuration file includes a series of sections, each representing a unique route. Each route section has a name, a type, and a model specification, which includes the model provider, name, and configuration details. The configuration section typically contains the base URL for the API and an environment variable for the API key. Here is an example of a single-route configuration: routes:   - name: chat     route_type: llm/v1/chat     model:       provider: openai       name: gpt-3.5-turbo       config:         openai_api_key: $OPENAI_API_KEY   In this example, we define a route named chat that corresponds to the llm/v1/chat type, which will use the gpt-3.5-turbo model from OpenAI to return query responses from the OpenAI service. The Gateway configuration is very easy to update. Simply edit the configuration file and save your changes, and the MLflow AI Gateway service will automatically update the routes with zero disruption or down time. This allows you to try out new providers or model types while keeping your applications steady and reliable. In order to define an API key for a given provider, there are three primary options:  Directly include it in the YAML configuration file. Use an environment variable to store the API key and reference it in the YAML configuration file. Define your API key in a file and reference the location of that key-bearing file within the YAML configuration file.  If you choose to include the API key directly, replace $OPENAI_API_KEY in the YAML file with your actual API key.  Warning The MLflow AI Gateway service provides direct access to billed external LLM services. It is strongly recommended to restrict access to this server. See the section on security for guidance.  If you prefer to use an environment variable (recommended), you can define it in your shell environment. For example: export OPENAI_API_KEY=""your_openai_api_key""   Note: Replace "your_openai_api_key" with your actual OpenAI API key.  AI Gateway Configuration Details  The MLflow AI Gateway service relies on a user-provided configuration file. It defines how the gateway interacts with various language model providers and dictates the routes that users can access. The configuration file is written in YAML and includes a series of sections, each representing a unique route. Each route section has a name, a type, and a model specification, which includes the provider, model name, and provider-specific configuration details. Here are the details of each configuration parameter:  General Configuration Parameters   routes: This is a list of route configurations. Each route represents a unique endpoint that maps to a particular language model service.  Each route has the following configuration parameters:  name: This is the name of the route. It needs to be a unique name without spaces or any non-alphanumeric characters other than hyphen and underscore. route_type: This specifies the type of service offered by this route. This determines the interface for inputs to a route and the returned outputs. Current supported route types are:  "llm/v1/completions" "llm/v1/chat" "llm/v1/embeddings"   model: This defines the provider-specific details of the language model. It contains the following fields:  provider: This indicates the provider of the AI model. It accepts the following values:  "openai" "mosaicml" "anthropic" "cohere" "palm" "azure" / "azuread" "mlflow-model-serving" "huggingface-text-generation-inference" "ai21labs" "bedrock"   name: This is an optional field to specify the name of the model. config: This contains provider-specific configuration details.      Provider-Specific Configuration Parameters   OpenAI          Configuration Parameter Required Default Description    openai_api_key Yes  This is the API key for the OpenAI service.  openai_api_type No  This is an optional field to specify the type of OpenAI API to use.  openai_api_base No https://api.openai.com/v1 This is the base URL for the OpenAI API.  openai_api_version No  This is an optional field to specify the OpenAI API version.  openai_organization No  This is an optional field to specify the organization in OpenAI.      MosaicML          Configuration Parameter Required Default Description    mosaicml_api_key Yes N/A This is the API key for the MosaicML service.      Cohere          Configuration Parameter Required Default Description    cohere_api_key Yes N/A This is the API key for the Cohere service.      HuggingFace Text Generation Inference          Configuration Parameter Required Default Description    hf_server_url Yes N/A This is the url of the Huggingface TGI Server.      PaLM          Configuration Parameter Required Default Description    palm_api_key Yes N/A This is the API key for the PaLM service.      AI21 Labs          Configuration Parameter Required Default Description    ai21labs_api_key Yes N/A This is the API key for the AI21 Labs service.      Anthropic          Configuration Parameter Required Default Description    anthropic_api_key Yes N/A This is the API key for the Anthropic service.      AWS Bedrock  Top-level model configuration for AWS Bedrock routes must be one of the following two supported authentication modes: key-based or role-based.         Configuration Parameter Required Default Description    aws_config No  An object with either the key-based or role-based schema below.    To use key-based authentication, define an AWS Bedrock route with the required fields below. .. note: If using a configured route purely for development or testing, utilizing an IAM User role or a temporary short-lived standard IAM role are recommended; while for production deployments, a standard long-expiry IAM role is recommended to ensure that the route is capable of handling authentication for a long period. If the authentication expires and a new set of keys need to be supplied, the route must be recreated in order to persist the new keys.           Configuration Parameter Required Default Description    aws_region No AWS_REGION/AWS_DEFAULT_REGION The AWS Region to use for bedrock access.  aws_secret_access_key Yes  AWS secret access key for the IAM user/role authorized to use bedrock  aws_access_key_id Yes  AWS access key ID for the IAM user/role authorized to use Bedrock  aws_session_token No None Optional session token, if required    Alternatively, for role-based authentication, an AWS Bedrock route can be defined and initialized with an a IAM Role  ARN that is authorized to access Bedrock.  The MLflow AI Gateway will attempt to assume this role with using the standard credential provider chain and will renew the role credentials if they have expired.         Configuration Parameter Required Default Description    aws_region No AWS_REGION/AWS_DEFAULT_REGION The AWS Region to use for bedrock access.  aws_role_arn Yes  An AWS role authorized to use Bedrock.  The standard credential provider chain must be able to find credentials authorized to assume this role.  session_length_seconds No 900 The length of session to request.      MLflow Model Serving          Configuration Parameter Required Default Description    model_server_url Yes N/A This is the url of the MLflow Model Server.    Note that with MLflow model serving, the name parameter for the model definition is not used for validation and is only present for reference purposes. This alias can be useful for understanding a particular version or route definition that was used that can be referenced back to a deployed model. You may choose any name that you wish, provided that it is JSON serializable.   Azure OpenAI  Azure provides two different mechanisms for integrating with OpenAI, each corresponding to a different type of security validation. One relies on an access token for validation, referred to as azure, while the other uses Azure Active Directory (Azure AD) integration for authentication, termed as azuread. To match your user's interaction and security access requirements, adjust the openai_api_type parameter to represent the preferred security validation model. This will ensure seamless interaction and reliable security for your Azure-OpenAI integration.         Configuration Parameter Required Default Description    openai_api_key Yes  This is the API key for the Azure OpenAI service.  openai_api_type Yes  This field must be either azure or azuread depending on the security access protocol.  openai_api_base Yes  This is the base URL for the Azure OpenAI API service provided by Azure.  openai_api_version Yes  The version of the Azure OpenAI service to utilize, specified by a date.  openai_deployment_name Yes  This is the name of the deployment resource for the Azure OpenAI service.  openai_organization No  This is an optional field to specify the organization in OpenAI.    An example configuration for Azure OpenAI is: routes:   - name: completions     route_type: llm/v1/completions     model:       provider: openai       name: gpt-35-turbo       config:         openai_api_type: ""azuread""         openai_api_key: $AZURE_AAD_TOKEN         openai_deployment_name: ""{your_deployment_name}""         openai_api_base: ""https://{your_resource_name}-azureopenai.openai.azure.com/""         openai_api_version: ""2023-05-15""    Note Azure OpenAI has distinct features as compared with the direct OpenAI service. For an overview, please see the comparison documentation.  For specifying an API key, there are three options:  (Preferred) Use an environment variable to store the API key and reference it in the YAML configuration file. This is denoted by a $ symbol before the name of the environment variable. (Preferred) Define the API key in a file and reference the location of that key-bearing file within the YAML configuration file. Directly include it in the YAML configuration file.   Important The use of environment variables or file-based keys is recommended for better security practices. If the API key is directly included in the configuration file, it should be ensured that the file is securely stored and appropriately access controlled. Please ensure that the configuration file is stored in a secure location as it contains sensitive API keys.        Querying the AI Gateway  Once the MLflow AI Gateway server has been configured and started, it is ready to receive traffic from users.  Standard Query Parameters  The MLflow AI Gateway defines standard parameters for chat, completions, and embeddings that can be used when querying any route regardless of its provider. Each parameter has a standard range and default value. When querying a route with a particular provider, the MLflow AI Gateway automatically scales parameter values according to the provider's value ranges for that parameter.  Completions  The standard parameters for completions routes with type llm/v1/completions are:          Query Parameter Type Required Default Description    prompt string Yes N/A The prompt for which to generate completions.  candidate_count integer No 1 The number of completions to generate for the specified prompt, between 1 and 5.  temperature float No 0.0 The sampling temperature to use, between 0 and 1. Higher values will make the output more random, and lower values will make the output more deterministic.  max_tokens integer No None The maximum completion length, between 1 and infinity (unlimited).  stop array[string] No None Sequences where the model should stop generating tokens and return the completion.      Chat  The standard parameters for chat routes with type llm/v1/chat are:          Query Parameter Type Required Default Description    messages array[message] Yes N/A A list of messages in a conversation from which to a new message (chat completion). For information about the message structure, see Messages.  candidate_count integer No 1 The number of chat completions to generate for the specified prompt, between 1 and 5.  temperature float No 0.0 The sampling temperature to use, between 0 and 1. Higher values will make the output more random, and lower values will make the output more deterministic.  max_tokens integer No None The maximum completion length, between 1 and infinity (unlimited).  stop array[string] No None Sequences where the model should stop generating tokens and return the chat completion.     Messages  Each chat message is a string dictionary containing the following fields:         Field Name Required Default Description    role Yes N/A The role of the conversation participant who sent the message. Must be one of: ""system"", ""user"", or ""assistant"".  content Yes N/A The message content.       Embeddings  The standard parameters for completions routes with type llm/v1/embeddings are:          Query Parameter Type Required Default Description    text string or array[string] Yes N/A A string or list of strings for which to generate embeddings.       Additional Query Parameters  In addition to the Standard Query Parameters, you can pass any additional parameters supported by the route's provider as part of your query. For example:  logit_bias (supported by OpenAI, Cohere) top_k (supported by MosaicML, Anthropic, PaLM, Cohere) frequency_penalty (supported by OpenAI, Cohere, AI21 Labs) presence_penalty (supported by OpenAI, Cohere, AI21 Labs)  The following parameters are not allowed:  stream is not supported. Setting this parameter on any provider will not work currently.  Below is an example of submitting a query request to an MLflow AI Gateway route using additional parameters: data = {     ""prompt"": (         ""What would happen if an asteroid the size of ""         ""a basketball encountered the Earth traveling at 0.5c? ""         ""Please provide your answer in .rst format for the purposes of documentation.""     ),     ""temperature"": 0.5,     ""max_tokens"": 1000,     ""candidate_count"": 1,     ""frequency_penalty"": 0.2,     ""presence_penalty"": 0.2, }  query(route=""completions-gpt4"", data=data)   The results of the query are: {   ""candidates"": [     {       ""text"": ""If an asteroid the size of a basketball (roughly 24 cm in       diameter) were to hit the Earth at 0.5 times the speed of light       (approximately 150,000 kilometers per second), the energy released       on impact would be enormous. The kinetic energy of an object moving       at relativistic speeds is given by the formula: KE = (\\gamma - 1)       mc^2 where \\gamma is the Lorentz factor given by..."",       ""metadata"": {         ""finish_reason"": ""stop""       }     }   ],   ""metadata"": {     ""input_tokens"": 40,     ""output_tokens"": 622,     ""total_tokens"": 662,     ""model"": ""gpt-4-0613"",     ""route_type"": ""llm/v1/completions""   } }     FastAPI Documentation ("/docs")  FastAPI, the framework used for building the MLflow AI Gateway, provides an automatic interactive API documentation interface, which is accessible at the "/docs" endpoint (e.g., http://my.gateway:9000/docs). This interactive interface is very handy for exploring and testing the available API endpoints. As a convenience, accessing the root URL (e.g., http://my.gateway:9000) redirects to this "/docs" endpoint.   MLflow Python Client APIs  MlflowGatewayClient is the user-facing client API that is used to interact with the MLflow AI Gateway. It abstracts the HTTP requests to the Gateway via a simple, easy-to-use Python API. The fluent API is a higher-level interface that supports setting the Gateway URI once and using simple functions to interact with the AI Gateway Server.  Fluent API  For the fluent API, here are some examples:  Set the Gateway URI: Before using the Fluent API, the gateway URI must be set via set_gateway_uri(). Alternatively to directly calling the set_gateway_uri function, the environment variable MLFLOW_GATEWAY_URI can be set directly, achieving the same session-level persistence for all fluent API usages. from mlflow.gateway import set_gateway_uri  set_gateway_uri(gateway_uri=""http://my.gateway:7000"")    Query a route: The query() function queries the specified route and returns the response from the provider in a standardized format. The data structure you send in the query depends on the route. from mlflow.gateway import query  response = query(     ""embeddings"", {""text"": [""It was the best of times"", ""It was the worst of times""]} ) print(response)       Client API  To use the MlflowGatewayClient API, see the below examples for the available API methods:  Create an MlflowGatewayClient from mlflow.gateway import MlflowGatewayClient  gateway_client = MlflowGatewayClient(""http://my.gateway:8888"")    List all routes: The search_routes() method returns a list of all routes. routes = gateway_client.search_routes() for route in routes:     print(route)    Query a route: The query() method submits a query to a configured provider route. The data structure you send in the query depends on the route. response = gateway_client.query(     ""chat"", {""messages"": [{""role"": ""user"", ""content"": ""Tell me a joke about rabbits""}]} ) print(response)       LangChain Integration  LangChain supports an integration for MLflow AI Gateway. This integration enable users to use prompt engineering, retrieval augmented generation, and other techniques with LLMs in the gateway.  Example  import mlflow from langchain import LLMChain, PromptTemplate from langchain.llms import MlflowAIGateway  gateway = MlflowAIGateway(     gateway_uri=""http://127.0.0.1:5000"",     route=""completions"",     params={         ""temperature"": 0.0,         ""top_p"": 0.1,     }, )  llm_chain = LLMChain(     llm=gateway,     prompt=PromptTemplate(         input_variables=[""adjective""],         template=""Tell me a {adjective} joke"",     ), ) result = llm_chain.run(adjective=""funny"") print(result)  with mlflow.start_run():     model_info = mlflow.langchain.log_model(chain, ""model"")  model = mlflow.pyfunc.load_model(model_info.model_uri) print(model.predict([{""adjective"": ""funny""}]))      MLflow Models  Interfacing with MLflow Models can be done in two ways. With the use of a custom PyFunc Model, a query can be issued directly to an AI Gateway endpoint and used in a broader context within a model. Data may be augmented, manipulated, or used in a mixture of experts paradigm. The other means of utilizing the AI Gateway along with MLflow Models is to define a served MLflow model directly as a route within the AI Gateway.  Using the AI Gateway to Query a served MLflow Model  For a full walkthrough and example of using the MLflow serving integration to query a model directly through the MLflow AI Gateway, please see the full example. Within the guide, you will see the entire end-to-end process of serving multiple models from different servers and configuring an MLflow AI Gateway server instance to provide a single unified point to handle queries from.   Using an MLflow Model to Query the AI Gateway  You can also build and deploy MLflow Models that call the MLflow AI Gateway. The example below demonstrates how to use an AI Gateway server from within a custom pyfunc model.  Note The custom Model shown in the example below is utilizing environment variables for the AI Gateway server's uri. These values can also be set manually within the definition or can be applied via mlflow.gateway.get_gateway_uri() after the uri has been set. For the example below, the value for MLFLOW_GATEWAY_URI is http://127.0.0.1:5000/. For an actual deployment use case, this value would be set to the configured and production deployment server.  import os import pandas as pd import mlflow   def predict(data):     from mlflow.gateway import MlflowGatewayClient      client = MlflowGatewayClient(os.environ[""MLFLOW_GATEWAY_URI""])      payload = data.to_dict(orient=""records"")     return [         client.query(route=""completions-claude"", data=query)[""candidates""][0][""text""]         for query in payload     ]   input_example = pd.DataFrame.from_dict(     {""prompt"": [""Where is the moon?"", ""What is a comet made of?""]} ) signature = mlflow.models.infer_signature(     input_example, [""Above our heads."", ""It's mostly ice and rocks.""] )  with mlflow.start_run():     model_info = mlflow.pyfunc.log_model(         python_model=predict,         registered_model_name=""anthropic_completions"",         artifact_path=""anthropic_completions"",         input_example=input_example,         signature=signature,     )  df = pd.DataFrame.from_dict(     {         ""prompt"": [""Tell me about Jupiter"", ""Tell me about Saturn""],         ""temperature"": 0.6,         ""max_records"": 500,     } )  loaded_model = mlflow.pyfunc.load_model(model_info.model_uri)  print(loaded_model.predict(df))   This custom MLflow model can be used in the same way as any other MLflow model. It can be used within a spark_udf, used with mlflow.evaluate(), or deploy like any other model.    REST API  The REST API allows you to send HTTP requests directly to the MLflow AI Gateway server. This is useful if you're not using Python or if you prefer to interact with the Gateway using HTTP directly. Here are some examples for how you might use curl to interact with the Gateway:  Get information about a particular route: GET /api/2.0/gateway/routes/{name} This endpoint returns a serialized representation of the Route data structure. This provides information about the name and type, as well as the model details for the requested route endpoint. curl -X GET http://my.gateway:8888/api/2.0/gateway/routes/embeddings    List all routes: GET /api/2.0/gateway/routes/ This endpoint returns a list of all routes. curl -X GET http://my.gateway:8888/api/2.0/gateway/routes/    Query a route: POST /gateway/{route}/invocations This endpoint allows you to submit a query to a configured provider route. The data structure you send in the query depends on the route. Here are examples for the "completions", "chat", and "embeddings" routes:  Completions curl -X POST http://my.gateway:8888/gateway/completions/invocations \   -H ""Content-Type: application/json"" \   -d '{""prompt"": ""Describe the probability distribution of the decay chain of U-235""}'    Chat curl -X POST http://my.gateway:8888/gateway/chat/invocations \   -H ""Content-Type: application/json"" \   -d '{""messages"": [{""role"": ""user"", ""content"": ""Can you write a limerick about orange flavored popsicles?""}]}'    Embeddings curl -X POST http://my.gateway:8888/gateway/embeddings/invocations \   -H ""Content-Type: application/json"" \   -d '{""text"": [""I would like to return my shipment of beanie babies, please"", ""Can I please speak to a human now?""]}'       Note: Remember to replace http://my.gateway:8888 with the URL of your actual MLflow AI Gateway Server.     MLflow AI Gateway API Documentation  API documentation   AI Gateway Security Considerations  Remember to ensure secure access to the system that the MLflow AI Gateway service is running in to protect access to these keys. An effective way to secure your MLflow AI Gateway service is by placing it behind a reverse proxy. This will allow the reverse proxy to handle incoming requests and forward them to the MLflow AI Gateway. The reverse proxy effectively shields your application from direct exposure to Internet traffic. A popular choice for a reverse proxy is Nginx. In addition to handling the traffic to your application, Nginx can also serve static files and load balance the traffic if you have multiple instances of your application running. Furthermore, to ensure the integrity and confidentiality of data between the client and the server, it's highly recommended to enable HTTPS on your reverse proxy. In addition to the reverse proxy, it's also recommended to add an authentication layer before the requests reach the MLflow AI Gateway. This could be HTTP Basic Authentication, OAuth, or any other method that suits your needs. For example, here's a simple configuration for Nginx with Basic Authentication: http {     server {         listen 80;          location / {             auth_basic ""Restricted Content"";             auth_basic_user_file /etc/nginx/.htpasswd;              proxy_pass http://localhost:5000;  # Replace with the MLflow AI Gateway service port         }     } }   In this example, /etc/nginx/.htpasswd is a file that contains the username and password for authentication. These measures, together with a proper network setup, can significantly improve the security of your system and ensure that only authorized users have access to submit requests to your LLM services.   LangChain Integration  LangChain supports an integration for MLflow AI Gateway. See https://python.langchain.com/docs/ecosystem/integrations/mlflow_ai_gateway for more information.        Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
llms/llm-tracking/index.html,"   Documentation  LLMs  MLflow's LLM Tracking Capabilities       MLflow's LLM Tracking Capabilities  MLflow's LLM Tracking system is an enhancement to the existing MLflow Tracking system, offerring additional capabilities for monitoring, managing, and interpreting interactions with Large Language Models (LLMs). At its core, MLflow's LLM suite builds upon the standard logging capabilities familiar to professionals working with traditional Machine Learning (ML) and Deep Learning (DL). However, it introduces distinct features tailored for the unique intricacies of LLMs. One such standout feature is the introduction of "prompts" – the queries or inputs directed towards an LLM – and the subsequent data the model generates in response. While MLflow's offerings for other model types typically exclude built-in mechanisms for preserving inference results, LLMs necessitate this due to their dynamic and generative nature. Recognizing this, MLflow introduces the term 'predictions' alongside the existing tracking components of artifacts, parameters, tags, and metrics, ensuring comprehensive lineage and quality tracking for text-generating models.  Introduction to LLM Tracking  The world of Large Language Models is vast, and as these models become more intricate and sophisticated, the need for a robust tracking system becomes paramount. MLflow's LLM Tracking is centered around the concept of runs. In essence, a run is a distinct execution or interaction with the LLM — whether it's a single query, a batch of prompts, or an entire fine-tuning session. Each run meticulously records:  Parameters: Key-value pairs that detail the input parameters for the LLM. These could range from model-specific parameters like top_k and temperature to more generic ones. They provide context and configuration for each run. Parameters can be logged using both mlflow.log_param() for individual entries and mlflow.log_params() for bulk logging. Metrics: These are quantitative measures, often numeric, that give insights into the performance, accuracy, or any other measurable aspect of the LLM interaction. Metrics are dynamic and can be updated as the run progresses, offering a real-time or post-process insight into the model's behavior. Logging of metrics is facilitated through mlflow.log_metric() and mlflow.log_metrics(). Predictions: To understand and evaluate LLM outputs, MLflow allows for the logging of predictions. This encompasses the prompts or inputs sent to the LLM and the outputs or responses received. For structured storage and easy retrieval, these predictions are stored as artifacts in CSV format, ensuring that each interaction is preserved in its entirety. This logging is achieved using the dedicated mlflow.log_table(). Artifacts: Beyond predictions, MLflow's LLM Tracking can store a myriad of output files, ranging from visualization images (e.g., PNGs), serialized models (e.g., an openai model), to structured data files (e.g., a Parquet file). The mlflow.log_artifact() function is at the heart of this, allowing users to log and organize their artifacts with ease.  Furthermore, to provide structured organization and comparative analysis capabilities, runs can be grouped into experiments. These experiments act as containers, grouping related runs, and providing a higher level of organization. This organization ensures that related runs can be compared, analyzed, and managed as a cohesive unit.   Detailed Logging of LLM Interactions  MLflow's LLM Tracking doesn't just record data — it offers structured logging mechanisms tailored to the needs of LLM interactions:  Parameters: Logging parameters is straightforward. Whether you're logging a single parameter using mlflow.log_param() or multiple parameters simultaneously with mlflow.log_params(), MLflow ensures that every detail is captured. Metrics: Quantitative insights are crucial. Whether it's tracking the accuracy of a fine-tuned LLM or understanding its response time, metrics provide this insight. They can be logged individually via mlflow.log_metric() or in bulk using mlflow.log_metrics(). Predictions: Every interaction with an LLM yields a result — a prediction. Capturing this prediction, along with the inputs that led to it, is crucial. The mlflow.log_table() function is specifically designed for this, ensuring that both inputs and outputs are logged cohesively. Artifacts: Artifacts act as the tangible outputs of an LLM run. They can be images, models, or any other form of data. Logging them is seamless with mlflow.log_artifact(), which ensures that every piece of data, regardless of its format, is stored and linked to its respective run.    Structured Storage of LLM Tracking Data  Every piece of data, every parameter, metric, prediction, and artifact is not just logged — it's structured and stored as part of an MLflow Experiment run. This organization ensures data integrity, easy retrieval, and a structured approach to analyzing and understanding LLM interactions in the grand scheme of machine learning workflows.        Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
llms/llm-evaluate/index.html,"   Documentation  LLMs  MLflow LLM Evaluate       MLflow LLM Evaluate  With the emerging of ChatGPT, LLMs have shown its power of text generation in various fields, such as question answering, translating and text summarization. Evaluating LLMs' performance is slightly different from traditional ML models, as very often there is no single ground truth to compare against. MLflow provides an API mlflow.evaluate() to help evaluate your LLMs. MLflow's LLM evaluation functionality consists of 3 main components:  A model to evaluate: it can be an MLflow pyfunc model, a URI pointing to one registered MLflow model, or any python callable that represents your model, e.g, a HuggingFace text summarization pipeline. Metrics: the metrics to compute, LLM evaluate will use LLM metrics. Evaluation data: the data your model is evaluated at, it can be a pandas Dataframe, a python list, a numpy array or an mlflow.data.dataset.Dataset() instance.   Full Notebook Guides and Examples  If you're interested in thorough use-case oriented guides that showcase the simplicity and power of MLflow's evaluate functionality for LLMs, please navigate to the notebook collection below: View the Notebook Guides  Quickstart  Below is a simple example that gives an quick overview of how MLflow LLM evaluation works. The example builds a simple question-answering model by wrapping "openai/gpt-4" with custom prompt. You can paste it to your IPython or local editor and execute it, and install missing dependencies as prompted. Running the code requires OpenAI API key, if you don't have an OpenAI key, you can set it up [here](https://platform.openai.com/account/api-keys). export OPENAI_API_KEY='your-api-key-here'   import mlflow import openai import os import pandas as pd from getpass import getpass  eval_data = pd.DataFrame(     {         ""inputs"": [             ""What is MLflow?"",             ""What is Spark?"",         ],         ""ground_truth"": [             ""MLflow is an open-source platform for managing the end-to-end machine learning (ML) ""             ""lifecycle. It was developed by Databricks, a company that specializes in big data and ""             ""machine learning solutions. MLflow is designed to address the challenges that data ""             ""scientists and machine learning engineers face when developing, training, and deploying ""             ""machine learning models."",             ""Apache Spark is an open-source, distributed computing system designed for big data ""             ""processing and analytics. It was developed in response to limitations of the Hadoop ""             ""MapReduce computing model, offering improvements in speed and ease of use. Spark ""             ""provides libraries for various tasks such as data ingestion, processing, and analysis ""             ""through its components like Spark SQL for structured data, Spark Streaming for ""             ""real-time data processing, and MLlib for machine learning tasks"",         ],     } )  with mlflow.start_run() as run:     system_prompt = ""Answer the following question in two sentences""     # Wrap ""gpt-4"" as an MLflow model.     logged_model_info = mlflow.openai.log_model(         model=""gpt-4"",         task=openai.ChatCompletion,         artifact_path=""model"",         messages=[             {""role"": ""system"", ""content"": system_prompt},             {""role"": ""user"", ""content"": ""{question}""},         ],     )      # Use predefined question-answering metrics to evaluate our model.     results = mlflow.evaluate(         logged_model_info.model_uri,         eval_data,         targets=""ground_truth"",         model_type=""question-answering"",     )     print(f""See aggregated evaluation results below: \n{results.metrics}"")      # Evaluation result for each data record is available in `results.tables`.     eval_table = results.tables[""eval_results_table""]     print(f""See evaluation table below: \n{eval_table}"")     LLM Evaluation Metrics  There are two types of LLM evaluation metrics in MLflow:  Metrics relying on SaaS model (e.g., OpenAI) for scoring, e.g., mlflow.metrics.genai.answer_relevance(). These metrics are created via mlflow.metrics.genai.make_genai_metric() method. For each data record, these metrics under the hood sends one prompt consisting of the following information to the SaaS model, and extract the score from model response:  Metrics definition. Metrics grading criteria. Reference examples. Input data/context. Model output. [optional] Ground truth.  More details of how these fields are set can be found in the section "Create your Custom LLM-evaluation Metrics".  Function-based per-row metrics. These metrics calculate a score for each data record (row in terms of Pandas/Spark dataframe), based on certain functions, like Rouge (mlflow.metrics.rougeL()) or Flesch Kincaid (mlflow.metrics.flesch_kincaid_grade_level()). These metrics are similar to traditional metrics.   Select Metrics to Evaluate  There are two ways to select metrics to evaluate your model:  Use default metrics for pre-defined model types. Use a custom list of metrics.    Use Default Metrics for Pre-defined Model Types  MLflow LLM evaluation includes default collections of metrics for pre-selected tasks, e.g, "question-answering". Depending on the LLM use case that you are evaluating, these pre-defined collections can greatly simplify the process of running evaluations. To use defaults metrics for pre-selected tasks, specify the model_type argument in mlflow.evaluate(), as shown by the example below: results = mlflow.evaluate(     model,     eval_data,     targets=""ground_truth"",     model_type=""question-answering"", )   The supported LLM model types and associated metrics are listed below:  question-answering: model_type=""question-answering"":   exact-match toxicity 1 ari_grade_level 2 flesch_kincaid_grade_level 2    text-summarization: model_type=""text-summarization"":   ROUGE 3 toxicity 1 ari_grade_level 2 flesch_kincaid_grade_level 2    text models: model_type=""text"":   toxicity 1 ari_grade_level 2 flesch_kincaid_grade_level 2     1 Requires package evaluate, torch, and transformers 2 Requires package textstat 3 Requires package evaluate, nltk, and rouge-score   Use a Custom List of Metrics  Using the pre-defined metrics associated with a given model type is not the only way to generate scoring metrics for LLM evaluation in MLflow. You can specify a custom list of metrics in the extra_metrics argument in mlflow.evaluate:  To add additional metrics to the default metrics list of pre-defined model type, keep the model_type and add your metrics to extra_metrics: results = mlflow.evaluate(     model,     eval_data,     targets=""ground_truth"",     model_type=""question-answering"",     extra_metrics=[mlflow.metrics.latency()], )   The above code will evaluate your model using all metrics for "question-answering" model plus mlflow.metrics.latency().  To disable default metric calculation and only calculate your selected metrics, remove the model_type argument and define the desired metrics.  results = mlflow.evaluate(     model,     eval_data,     targets=""ground_truth"",     extra_metrics=[mlflow.metrics.toxicity(), mlflow.metrics.latency()], )      The full reference for supported evaluation metrics can be found here.   Metrics with LLM as the Judge  MLflow offers a few pre-canned metrics which uses LLM as the judge. Despite the difference under the hood, the usage is the same - put these metrics in the extra_metrics argument in mlflow.evaluate(). Here is the list of pre-canned metrics:  mlflow.metrics.genai.answer_similarity(): Use this metric when you want to evaluate how similar the model generated output is compared to the information in the ground_truth. High scores mean that your model outputs contain similar information as the ground_truth, while low scores mean that outputs may disagree with the ground_truth. mlflow.metrics.genai.answer_correctness(): Use this metric when you want to evaluate how factually correct the model generated output is based on the information in the ground_truth. High scores mean that your model outputs contain similar information as the ground_truth and that this information is correct, while low scores mean that outputs may disagree with the ground_truth or that the information in the output is incorrect. Note that this builds onto answer_similarity. mlflow.metrics.genai.answer_relevance(): Use this metric when you want to evaluate how relevant the model generated output is to the input (context is ignored). High scores mean that your model outputs are about the same subject as the input, while low scores mean that outputs may be non-topical. mlflow.metrics.genai.relevance(): Use this metric when you want to evaluate how relevant the model generated output is with respect to both the input and the context. High scores mean that the model has understood the context and correct extracted relevant information from the context, while low score mean that output has completely ignored the question and the context and could be hallucinating. mlflow.metrics.genai.faithfulness(): Use this metric when you want to evaluate how faithful the model generated output is based on the context provided. High scores mean that the outputs contain information that is in line with the context, while low scores mean that outputs may disagree with the context (input is ignored).     Creating Custom LLM-evaluation Metrics   Create LLM-as-judge Evaluation Metrics (Category 1)  You can also create your own Saas LLM evaluation metrics with MLflow API mlflow.metrics.genai.make_genai_metric(), which needs the following information:  name: the name of your custom metric. definition: describe what's the metric doing. grading_prompt: describe the scoring critieria. examples: a few input/output examples with score, they are used as a reference for LLM judge. model: the identifier of LLM judge. parameters: the extra parameters to send to LLM judge, e.g., temperature for ""openai:/gpt-3.5-turbo-16k"". aggregations: The list of options to aggregate the per-row scores using numpy functions. greater_is_better: indicates if a higher score means your model is better.  Under the hood, definition, grading_prompt, examples together with evaluation data and model output will be composed into a long prompt and sent to LLM. If you are familiar with the concept of prompt engineering, SaaS LLM evaluation metric is basically trying to compose a "right" prompt containing instructions, data and model output so that LLM, e.g., GPT4 can output the information we want. Now let's create a custom GenAI metrics called "professionalism", which measures how professional our model output is. Let's first create a few examples with scores, these will be the reference samples LLM judge uses. To create such examples, we will use mlflow.metrics.genai.EvaluationExample() class, which has 4 fields:  input: input text. output: output text. score: the score for output in the context of input. justification: why do we give the score for the data.  professionalism_example_score_2 = mlflow.metrics.genai.EvaluationExample(     input=""What is MLflow?"",     output=(         ""MLflow is like your friendly neighborhood toolkit for managing your machine learning projects. It helps ""         ""you track experiments, package your code and models, and collaborate with your team, making the whole ML ""         ""workflow smoother. It's like your Swiss Army knife for machine learning!""     ),     score=2,     justification=(         ""The response is written in a casual tone. It uses contractions, filler words such as 'like', and ""         ""exclamation points, which make it sound less professional. ""     ), ) professionalism_example_score_4 = mlflow.metrics.genai.EvaluationExample(     input=""What is MLflow?"",     output=(         ""MLflow is an open-source platform for managing the end-to-end machine learning (ML) lifecycle. It was ""         ""developed by Databricks, a company that specializes in big data and machine learning solutions. MLflow is ""         ""designed to address the challenges that data scientists and machine learning engineers face when ""         ""developing, training, and deploying machine learning models."",     ),     score=4,     justification=(""The response is written in a formal language and a neutral tone. ""), )   Now let's define the professionalism metric, you will see how each field is set up. professionalism = mlflow.metrics.genai.make_genai_metric(     name=""professionalism"",     definition=(         ""Professionalism refers to the use of a formal, respectful, and appropriate style of communication that is ""         ""tailored to the context and audience. It often involves avoiding overly casual language, slang, or ""         ""colloquialisms, and instead using clear, concise, and respectful language.""     ),     grading_prompt=(         ""Professionalism: If the answer is written using a professional tone, below are the details for different scores: ""         ""- Score 0: Language is extremely casual, informal, and may include slang or colloquialisms. Not suitable for ""         ""professional contexts.""         ""- Score 1: Language is casual but generally respectful and avoids strong informality or slang. Acceptable in ""         ""some informal professional settings.""         ""- Score 2: Language is overall formal but still have casual words/phrases. Borderline for professional contexts.""         ""- Score 3: Language is balanced and avoids extreme informality or formality. Suitable for most professional contexts. ""         ""- Score 4: Language is noticeably formal, respectful, and avoids casual elements. Appropriate for formal ""         ""business or academic settings. ""     ),     examples=[professionalism_example_score_2, professionalism_example_score_4],     model=""openai:/gpt-3.5-turbo-16k"",     parameters={""temperature"": 0.0},     aggregations=[""mean"", ""variance""],     greater_is_better=True, )     Create heuristic-based LLM Evaluation Metrics (Category 2)  This is very similar to creating a custom traditional metrics, with the exception of returning a EvaluationResult instance. Basically you need to:  Implement a eval_fn to define your scoring logic, it must take in 3 args predictions, targets and metrics. eval_fn must return a mlflow.metrics.MetricValue() instance. Pass eval_fn and other arguments to mlflow.metrics.make_metric API to create the metric.  The following code creates a dummy per-row metric called ""over_10_chars"": if the model output is greater than 10, the score is 1 otherwise 0. def eval_fn(predictions, targets, metrics):     scores = []     for i in range(len(predictions)):         if len(predictions[i]) > 10:             scores.append(1)         else:             scores.append(0)     return MetricValue(         scores=scores,         aggregate_results=standard_aggregations(scores),     )   # Create an EvaluationMetric object. passing_code_metric = make_metric(     eval_fn=eval_fn, greater_is_better=False, name=""over_10_chars"" )      Prepare Your LLM for Evaluating  In order to evaluate your LLM with mlflow.evaluate(), your LLM has to be one of the following type:  A mlflow.pyfunc.PyFuncModel() instance or a URI pointing to a logged mlflow.pyfunc.PyFuncModel model. In general we call that MLflow model. The A python function that takes in string inputs and outputs a single string. Your callable must match the signature of mlflow.pyfunc.PyFuncModel.predict() (without params argument), briefly it should:  Has data as the only argument, which can be a pandas.Dataframe, numpy.ndarray, python list, dictionary or scipy matrix. Returns one of pandas.DataFrame, pandas.Series, numpy.ndarray or list.   Set model=None, and put model outputs in data. Only applicable when the data is a Pandas dataframe.   Evaluating with an MLflow Model  For detailed instruction on how to convert your model into a mlflow.pyfunc.PyFuncModel instance, please read this doc. But in short, to evaluate your model as an MLflow model, we recommend following the steps below:  Package your LLM as an MLflow model and log it to MLflow server by log_model. Each flavor (opeanai, pytorch, …) has its own log_model API, e.g., mlflow.openai.log_model(): with mlflow.start_run():     system_prompt = ""Answer the following question in two sentences""     # Wrap ""gpt-3.5-turbo"" as an MLflow model.     logged_model_info = mlflow.openai.log_model(         model=""gpt-3.5-turbo"",         task=openai.ChatCompletion,         artifact_path=""model"",         messages=[             {""role"": ""system"", ""content"": system_prompt},             {""role"": ""user"", ""content"": ""{question}""},         ],     )    Use the URI of logged model as the model instance in mlflow.evaluate(): results = mlflow.evaluate(     logged_model_info.model_uri,     eval_data,     targets=""ground_truth"",     model_type=""question-answering"", )       Evaluating with a Custom Function  As of MLflow 2.8.0, mlflow.evaluate() supports evaluating a python function without requiring logging the model to MLflow. This is useful when you don't want to log the model and just want to evaluate it. The following example uses mlflow.evaluate() to evaluate a function. You also need to set up OpenAI authentication to run the code below. eval_data = pd.DataFrame(     {         ""inputs"": [             ""What is MLflow?"",             ""What is Spark?"",         ],         ""ground_truth"": [             ""MLflow is an open-source platform for managing the end-to-end machine learning (ML) lifecycle. It was developed by Databricks, a company that specializes in big data and machine learning solutions. MLflow is designed to address the challenges that data scientists and machine learning engineers face when developing, training, and deploying machine learning models."",             ""Apache Spark is an open-source, distributed computing system designed for big data processing and analytics. It was developed in response to limitations of the Hadoop MapReduce computing model, offering improvements in speed and ease of use. Spark provides libraries for various tasks such as data ingestion, processing, and analysis through its components like Spark SQL for structured data, Spark Streaming for real-time data processing, and MLlib for machine learning tasks"",         ],     } )   def openai_qa(inputs):     answers = []     system_prompt = ""Please answer the following question in formal language.""     for index, row in inputs.iterrows():         completion = openai.ChatCompletion.create(             model=""gpt-3.5-turbo"",             messages=[                 {""role"": ""system"", ""content"": system_prompt},                 {""role"": ""user"", ""content"": ""{row}""},             ],         )         answers.append(completion.choices[0].message.content)      return answers   with mlflow.start_run() as run:     results = mlflow.evaluate(         openai_qa,         eval_data,         model_type=""question-answering"",     )     Evaluating with a Static Dataset  For MLflow >= 2.8.0, mlflow.evaluate() supports evaluating a static dataset without specifying a model. This is useful when you save the model output to a column in a Pandas DataFrame or an MLflow PandasDataset, and want to evaluate the static dataset without re-running the model. If you are using a Pandas DataFrame, you must specify the column name that contains the model output using the top-level predictions parameter in mlflow.evaluate(): import mlflow import pandas as pd  eval_data = pd.DataFrame(     {         ""inputs"": [             ""What is MLflow?"",             ""What is Spark?"",         ],         ""ground_truth"": [             ""MLflow is an open-source platform for managing the end-to-end machine learning (ML) lifecycle. ""             ""It was developed by Databricks, a company that specializes in big data and machine learning solutions. ""             ""MLflow is designed to address the challenges that data scientists and machine learning engineers ""             ""face when developing, training, and deploying machine learning models."",             ""Apache Spark is an open-source, distributed computing system designed for big data processing and ""             ""analytics. It was developed in response to limitations of the Hadoop MapReduce computing model, ""             ""offering improvements in speed and ease of use. Spark provides libraries for various tasks such as ""             ""data ingestion, processing, and analysis through its components like Spark SQL for structured data, ""             ""Spark Streaming for real-time data processing, and MLlib for machine learning tasks"",         ],         ""predictions"": [             ""MLflow is an open-source platform that provides handy tools to manage Machine Learning workflow ""             ""lifecycle in a simple way"",             ""Spark is a popular open-source distributed computing system designed for big data processing and analytics."",         ],     } )  with mlflow.start_run() as run:     results = mlflow.evaluate(         data=eval_data,         targets=""ground_truth"",         predictions=""predictions"",         extra_metrics=[mlflow.metrics.genai.answer_similarity()],         evaluators=""default"",     )     print(f""See aggregated evaluation results below: \n{results.metrics}"")      eval_table = results.tables[""eval_results_table""]     print(f""See evaluation table below: \n{eval_table}"")      Viewing Evaluation Results   View Evaluation Results via Code  mlflow.evaluate() returns the evaluation results as an mlflow.models.EvaluationResult() instace. To see the score on selected metrics, you can check:  metrics: stores the aggregated results, like average/variance across the evaluation dataset. Let's take a second pass on the code example above and focus on printing out the aggregated results. with mlflow.start_run() as run:     results = mlflow.evaluate(         data=eval_data,         targets=""ground_truth"",         predictions=""predictions"",         extra_metrics=[mlflow.metrics.genai.answer_similarity()],         evaluators=""default"",     )     print(f""See aggregated evaluation results below: \n{results.metrics}"")    tables[""eval_results_table""]: stores the per-row evaluation results. with mlflow.start_run() as run:     results = mlflow.evaluate(         data=eval_data,         targets=""ground_truth"",         predictions=""predictions"",         extra_metrics=[mlflow.metrics.genai.answer_similarity()],         evaluators=""default"",     )     print(         f""See per-data evaluation results below: \n{results.tables['eval_results_table']}""     )       View Evaluation Results via the MLflow UI  Your evaluation result is automatically logged into MLflow server, so you can view your evaluation results directly from the MLflow UI. To view the evaluation results on MLflow UI, please follow the steps below:  Go to the experiment view of your MLflow experiment. Select the "Evaluation" tab. Select the runs you want to check evaluation results. Select the metrics from the dropdown menu on the right side.  Please see the screenshot below for clarity:            Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
llms/custom-pyfunc-for-llms/index.html,"   Documentation  LLMs  Deploying Advanced LLMs with Custom PyFuncs in MLflow       Deploying Advanced LLMs with Custom PyFuncs in MLflow  Advanced Large Language Models (LLMs) such as the MPT-7B instruct transformer are intricate and have requirements that don't align with traditional MLflow flavors. This demands a deeper understanding and the need for custom solutions. What's in this tutorial? This guide is designed to provide insights into the deployment of advanced LLMs with MLflow, with a focus on using custom PyFuncs to address challenges:  The World of LLMs: An introduction to LLMs, particularly models like the MPT-7B instruct transformer. We'll delve into their intricacies, importance, and the challenges associated with their deployment. Why Custom PyFuncs for LLM Deployment?: We'll explore the reasons behind the need for custom PyFuncs in the context of LLMs. How do they provide a bridge, ensuring that LLMs can be seamlessly deployed while adhering to MLflow's standards?   Managing Complex Behaviors: How custom PyFuncs can help in handling intricate model behaviors and dependencies that aren't catered to by MLflow's default flavors. Interface Data Manipulation: Delve into how custom PyFuncs allow the manipulation of interface data to generate prompts, thereby simplifying end-user interactions in a RESTful environment.    Crafting Custom PyFuncs for LLM Deployment: A step-by-step walkthrough on how to define, manage, and deploy an LLM using a custom PyFunc. We'll look at how to design a pyfunc to address LLM requirements and behaviors, and then how to deploy it using MLflow. Challenges with Traditional LLM Deployment: Recognize the issues and limitations when trying to deploy an advanced LLM using MLflow's built-in capabilities. Understand why custom PyFuncs become essential in such scenarios.  By the conclusion of this guide, you'll possess a deep understanding of how to deploy advanced LLMs in MLflow using custom PyFuncs. You'll appreciate the role of custom PyFuncs in making complex deployments streamlined and efficient.  Explore the Tutorial  View the Custom Pyfunc for LLMs Tutorial         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
llms/rag/index.html,"   Documentation  LLMs  Retrieval Augmented Generation (RAG)       Retrieval Augmented Generation (RAG)  Retrieval Augmented Generation (RAG) is a powerful and efficient approach to natural language processing that combines the strength of both pre-trained foundation models and retrieval mechanisms. It allows the generative model to access a dataset of documents through a retrieval mechanism, which enhances generated responses to be more contextually relevant and factually accurate. This improvement results in a cost-effective and accessible alternative to training custom models for specific use cases. The Retrieval mechanism works by embedding documents and questions in the same latent space, allowing a user to ask a question and get the most relevant document chunk as a response. This mechanism then passes the contextual chunk to the generative model, resulting in better quality responses with fewer hallucinations.  Benefits of RAG   Provides LLM access to external knowledge through documents, resulting in contextually accurate and factual responses. RAG is more cost-effective than fine-tuning, since it doesn't require the labeled data and computational resources that come with model training.    Understanding the Power of RAG  In the realm of artificial intelligence, particularly within natural language processing, the ability to generate coherent and contextually relevant responses is paramount. Large language models (LLMs) have shown immense promise in this area, but they often operate based on their internal knowledge, which can sometimes lead to inconsistencies or inaccuracies in their outputs. This is where RAG comes into play. RAG is a groundbreaking framework designed to enhance the capabilities of LLMs. Instead of solely relying on the vast but static knowledge embedded during their training, RAG empowers these models to actively retrieve and reference information from external knowledge bases. This dynamic approach ensures that the generated responses are not only rooted in the most current and reliable facts but also transparent in their sourcing. In essence, RAG transforms LLMs from closed-book learners, relying on memorized information, to open-book thinkers, capable of actively seeking out and referencing external knowledge. The implications of RAG are profound. By grounding responses in verifiable external sources, it significantly reduces the chances of LLMs producing misleading or incorrect information. Furthermore, it offers a more cost-effective solution for businesses, as there's less need for continuous retraining of the model. With RAG, LLMs can provide answers that are not only more accurate but also more trustworthy, paving the way for a new era of AI-driven insights and interactions.   Explore the Tutorial  View the RAG Question Generation Tutorial         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
model-evaluation/index.html,"   Documentation  Model Evaluation       Model Evaluation   Harnessing the Power of Automation  In the evolving landscape of machine learning, the evaluation phase of model development is just as important as ever. Ensuring the accuracy, reliability, and efficiency of models is paramount to ensure that the model that has been trained has been as thoroughly validated as it can be prior to promoting it beyond the development phase. However, manual evaluation can be tedious, error-prone, and time-consuming. MLflow addresses these challenges head-on, offering a suite of automated tools that streamline the evaluation process, saving time and enhancing accuracy, helping you to have confidence that the solution that you've spent so much time working on will meet the needs of the problem you're trying to solve.   LLM Model Evaluation  The rise of Large Language Models (LLMs) like ChatGPT has transformed the landscape of text generation, finding applications in question answering, translation, and text summarization. However, evaluating LLMs introduces unique challenges, primarily because there's often no single ground truth to compare against. MLflow's evaluation tools are tailored for LLMs, ensuring a streamlined and accurate evaluation process. Key Features:  Versatile Model Evaluation: MLflow supports evaluating various types of LLMs, whether it's an MLflow pyfunc model, a URI pointing to a registered MLflow model, or any python callable representing your model. Comprehensive Metrics: MLflow offers a range of metrics for LLM evaluation. From metrics that rely on SaaS models like OpenAI for scoring (e.g., mlflow.metrics.genai.answer_relevance()) to function-based per-row metrics such as Rouge (mlflow.metrics.rougeL()) or Flesch Kincaid (mlflow.metrics.flesch_kincaid_grade_level()). Predefined Metric Collections: Depending on your LLM use case, MLflow provides predefined metric collections, such as "question-answering" or "text-summarization", simplifying the evaluation process. Custom Metric Creation: Beyond the predefined metrics, MLflow allows users to create custom LLM evaluation metrics. Whether you're looking to evaluate the professionalism of a response or any other custom criteria, MLflow provides the tools to define and implement these metrics. Evaluation with Static Datasets: As of MLflow 2.8.0, you can evaluate a static dataset without specifying a model. This is especially useful when you've saved model outputs in a dataset and want a swift evaluation without rerunning the model. Integrated Results View: MLflow's mlflow.evaluate() returns comprehensive evaluation results, which can be viewed directly in the code or through the MLflow UI for a more visual representation.  Harnessing these features, MLflow's LLM evaluation tools eliminate the complexities and ambiguities associated with evaluating large language models. By automating these critical evaluation tasks, MLflow ensures that users can confidently assess the performance of their LLMs, leading to more informed decisions in the deployment and application of these models.  Guides and Tutorials for LLM Model Evaluation  To learn more about how you can leverage MLflow's evaluation features for your LLM-powered project work, see the tutorials below:                          RAG Evaluation                                       Learn how to evaluate a Retrieval Augmented Generation setup with MLflow Evaluate                                           Question-Answering Evaluation                                       See a working example of how to evaluate the quality of an LLM Question-Answering solution                                           RAG Question Generation Evaluation                                       See how to generate Questions for RAG generation and how to evaluate a RAG solution using MLflow                        Traditional ML Evaluation  Traditional machine learning techniques, from classification to regression, have been the bedrock of many industries. MLflow recognizes their significance and offers automated evaluation tools tailored for these classic techniques. Key Features:  Evaluating a Function: To get immediate results, you can evaluate a python function directly without logging the model. This is especially useful when you want a quick evaluation without the overhead of logging. Evaluating a Dataset: MLflow also supports evaluating a static dataset without specifying a model. This is invaluable when you've saved model outputs in a dataset and want a swift evaluation without having to rerun model inference. Evaluating a Model: With MLflow, you can set validation thresholds for your metrics. If a model doesn't meet these thresholds compared to a baseline, MLflow will alert you. This automated validation ensures that only high-quality models progress to the next stages. Common Metrics and Visualizations: MLflow automatically logs common metrics like accuracy, precision, recall, and more. Additionally, visual graphs such as the confusion matrix, lift_curve_plot, and others are auto-logged, providing a comprehensive view of your model's performance. SHAP Integration: MLflow is integrated with SHAP, allowing for the auto-logging of SHAP's summarization importances validation visualizations when using the evaluate APIs.         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
deep-learning/index.html,"   Documentation  Deep Learning       Deep Learning  The realm of deep learning has witnessed an unprecedented surge, revolutionizing numerous sectors with its ability to process vast amounts of data and capture intricate patterns. From the real-time object detection in autonomous vehicles to the generation of art through Generative Adversarial Networks, and from natural language processing applications in chatbots to predictive analytics in e-commerce, deep learning models are at the forefront of today's AI-driven innovations. MLflow acknowledges the profound impact and complexity of deep learning. With a keen focus on the unique challenges posed by deep learning workflows, such as iterative model training and hyperparameter tuning, MLflow introduces a robust suite of tools specifically designed for these advanced models. MLflow helps to facilitate seamless model development, ensuring reproducibility, and provides enhanced monitoring capabilities with the concept of 'steps' for recording metrics at various training iterations, with integrated UI features that enable you to easily visualize the iterative improvements of key metrics during training epochs.  Key Benefits:   Iterative Model Training: With the concept of 'steps', MLflow allows users to log metrics at various training iterations, offering a granular view of the model's progress. Reproducibility: Ensure that every model training run can be replicated with the exact same conditions. Scalability: Handle projects ranging from small-scale models to enterprise-level deployments with ease. Traceability: Keep track of every detail, from hyperparameters to the final model output.    Deep Autologging Integrations  One of the standout features of MLflow's deep learning support is its deep autologging integrations. These integrations automatically capture and log intricate details during the training of deep learning models, ensuring that every nuance, from model parameters to evaluation metrics, is meticulously recorded. This is especially prominent in frameworks like TensorFlow, PyTorch Lightning, base PyTorch, and Keras, making the iterative training process more insightful and manageable.   Native Library Support  Deep learning in MLflow is enriched by its native support for a number of the most popular libraries. The native integration with each of these libraries within MLflow help to streamline and simplify the training process, as well as saving, logging, loading, and representing models as generic Python functions for inference use anywhere. Opting for these native integrations brings forth a myriad of advantages:  Auto-logging Capabilities: Automatically capture details without manual intervention. Custom Serialization: Streamline the model saving and loading process with custom methods tailored for each library. Unified Interface: Regardless of the underlying library, interact with a consistent MLflow interface.  The officially supported integrations for deep learning libraries in MLflow encompass:                             Harness the power of these integrations and elevate your deep learning projects with MLflow's comprehensive support.   MLflow Tracking for Deep Learning  Tracking remains a cornerstone of the MLflow ecosystem, especially vital for the iterative nature of deep learning:  Experiments and Runs: Organize your deep learning projects into experiments, with each experiment containing multiple runs. Each run captures essential data like metrics at various training steps, hyperparameters, and the code state. Artifacts: Store vital outputs such as deep learning models, visualizations, or even tensorboard logs. This artifact repository ensures traceability and easy access. Metrics at Steps: With deep learning's iterative nature, MLflow allows logging metrics at various training steps, offering a granular view of the model's progress. Dependencies and Environment: Capture the computational environment, including deep learning frameworks' versions, ensuring reproducibility. Input Examples and Model Signatures: Define the expected format of the model's inputs, crucial for complex data like images or sequences. UI Integration: The enhanced UI provides a visual overview of deep learning runs, facilitating comparison and insights into training progress. Search Functionality: Efficiently navigate through your deep learning experiments using robust search capabilities. APIs: Interact with the tracking system programmatically, integrating deep learning workflows seamlessly.    Model Registry  A centralized repository for your deep learning models:  Versioning: Handle multiple iterations and versions of deep learning models, facilitating comparison or reversion. Annotations: Attach notes, training datasets, or other relevant metadata to models. Lifecycle Stages: Clearly define the stage of each model version, ensuring clarity in deployment and further fine-tuning.    Deployment for Deep Learning Models  Transition deep learning models from training to real-world applications:  Consistency: Ensure models, especially those with GPU dependencies, behave consistently across different deployment environments. Docker and GPU Support: Deploy in containerized environments, ensuring all dependencies, including GPU support, are encapsulated. Scalability: From deploying a single model to serving multiple distributed deep learning models, MLflow scales as per your requirements.         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
traditional-ml/index.html,"   Documentation  Traditional ML       Traditional ML  In the dynamic landscape of machine learning, traditional techniques remain foundational, playing pivotal roles across various industries and research institutions. From the precision of classification algorithms in healthcare diagnostics to the predictive prowess of regression models in finance, and from the forecasting capabilities of time-series analyses in supply chain management to the insights drawn from statistical modeling in social sciences, these core methodologies underscore many of the technological advancements we witness today. MLflow recognizes the enduring significance of traditional machine learning. Designed with precision and a deep understanding of the challenges and intricacies faced by data scientists and ML practitioners, MLflow offers a comprehensive suite of tools tailor-made for these classic techniques. This platform not only streamlines the model development and deployment processes but also ensures reproducibility, scalability, and traceability. As we delve further, we'll explore the multifaceted functionalities MLflow offers, showcasing how it enhances the efficacy, reliability, and insights derived from traditional ML models. Whether you're a seasoned expert looking to optimize workflows or a newcomer eager to make a mark, MLflow stands as an invaluable ally in your machine learning journey.  Native Library Support  There are a number of natively supported traditional ML libraries within MLflow. Throughout the documentation, you may see these referred to as "flavors", as they are specific implementations of native support for saving, logging, loading, and generic python function representation for the models that are produced from these libraries. There are distinct benefits to using the native versions of these implementations, as many have auto-logging functionality built in, as well as specific custom handling with serialization and deserialization that can greatly simplify your MLOps experiences when using these libraries. The officially supported integrations for traditional ML libraries include:                                         Tutorials and Guides                           Hyperparameter Tuning with MLflow and Optuna                                       Explore the integration of MLflow Tracking with Optuna for hyperparameter tuning. Dive into the capabilities of MLflow,                     understand parent-child run relationships, and compare different tuning runs to optimize model performance.                                           Custom Pyfunc Models with MLflow                                       Dive deep into the world of MLflow's Custom Pyfunc. Starting with basic model definitions, embark on a journey that                     showcases the versatility and power of Pyfunc. From simple mathematical curves to complex machine learning integrations,                     discover how Pyfunc offers standardized, reproducible, and efficient workflows for a variety of use cases.                         MLflow Tracking  Tracking is central to the MLflow ecosystem, facilitating the systematic organization of experiments and runs:  Experiments and Runs: Each experiment encapsulates a specific aspect of your research, and each experiment can house multiple runs. Runs document critical data like metrics, parameters, and the code state. Artifacts: Store crucial output from runs, be it models, visualizations, datasets, or other metadata. This repository of artifacts ensures traceability and easy access. Metrics and Parameters: By allowing users to log parameters and metrics, MLflow makes it straightforward to compare different runs, facilitating model optimization. Dependencies and Environment: The platform automatically captures the computational environment, ensuring that experiments are reproducible across different setups. Input Examples and Model Signatures: These features allow developers to define the expected format of the model's inputs, making validation and debugging more straightforward. UI Integration: The integrated UI provides a visual overview of all runs, enabling easy comparison and deeper insights. Search Functionality: Efficiently sift through your experiments using MLflow's robust search functionality. APIs: Comprehensive APIs are available, allowing users to interact with the tracking system programmatically, integrating it into existing workflows.    MLflow Recipes  Recipes in MLflow are predefined templates tailored for specific tasks:  Reduced Boilerplate: These templates help eliminate repetitive setup or initialization code, speeding up development. Best Practices: MLflow's recipes are crafted keeping best practices in mind, ensuring that users are aligned with industry standards right from the get-go. Customizability: While recipes provide a structured starting point, they're designed to be flexible, accommodating tweaks and modifications as needed.    MLflow Evaluate  Ensuring model quality is paramount:  Auto-generated Metrics: MLflow automatically evaluates models, providing key metrics for regression (like RMSE, MAE) and classification (such as F1-score, AUC-ROC). Visualization: Understand your model better with automatically generated plots. For instance, MLflow can produce confusion matrices, precision-recall curves, and more for classification tasks. Extensibility: While MLflow provides a rich set of evaluation tools out of the box, it's also designed to accommodate custom metrics and visualizations.    Model Registry  This feature acts as a catalog for models:  Versioning: As models evolve, keeping track of versions becomes crucial. The Model Registry handles versioning, ensuring that users can revert to older versions or compare different iterations. Annotations: Models in the registry can be annotated with descriptions, use-cases, or other relevant metadata. Lifecycle Stages: Track the stage of each model version, be it 'staging', 'production', or 'archived'. This ensures clarity in deployment and maintenance processes.    Deployment  MLflow simplifies the transition from development to production:  Consistency: By meticulously recording dependencies and the computational environment, MLflow ensures that models behave consistently across different deployment setups. Docker Support: Facilitate deployment in containerized environments using Docker, encapsulating all dependencies and ensuring a uniform runtime environment. Scalability: MLflow is designed to accommodate both small-scale deployments and large, distributed setups, ensuring that it scales with your needs.         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
traditional-ml/hyperparameter-tuning-with-child-runs/index.html,"   Documentation  Traditional ML  Hyperparameter Tuning with MLflow and Optuna       Hyperparameter Tuning with MLflow and Optuna  In this guide, we venture into a frequent use case of MLflow Tracking: hyperparameter tuning. Our journey will begin with a detailed notebook that showcases hyperparameter tuning using Optuna, and how each of these tuning runs are logged seamlessly with MLflow. Following this, we'll delve deeper, exploring alternative APIs and techniques that can be leveraged to further enhance our model tracking capabilities. Throughout this guide, our focal points will be:  Introducing the capabilities of MLflow for tracking hyperparameter tuning Understanding the distinction between parent and child runs in MLflow Delving into the components of MLflow and their relevance in our workflow Discovering and navigating experiments, parent runs, and child runs in the MLflow UI Conducting a comparative analysis of runs to understand the nuances of hyperparameter tuning   If you're keen on navigating directly to a specific topic within this guide, the links below will lead you right to the desired section:  Full Notebooks The Parent-Child relationship with runs Logging plots with MLflow   To get started with this guide, click NEXT below.       Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
traditional-ml/creating-custom-pyfunc/index.html,"   Documentation  Traditional ML  Building Custom Python Function Models with MLflow       Building Custom Python Function Models with MLflow  MLflow offers a wide range of pre-defined model flavors, but there are instances where you'd want to go beyond these and craft something tailored to your needs. That's where custom PyFuncs come in handy. What's in this tutorial? This guide aims to walk you through the intricacies of PyFuncs, explaining the why, the what, and the how:  Named Model Flavors: Before we dive into the custom territory, it's essential to understand the existing named flavors in MLflow. These pre-defined flavors simplify model tracking and deployment, but they might not cover every use case. Custom PyFuncs Demystified: What exactly is a custom PyFunc? How is it different from the named flavors, and when would you want to use one? We'll cover:   Pre/Post Processing: Integrate preprocessing or postprocessing steps as part of your model's prediction pipeline. Unsupported Libraries: Maybe you're using a niche or newly-released ML library that MLflow doesn't support yet. No worries, custom PyFuncs have you covered. External References: Avert serialization issues and simplify model deployment by externalizing references.    Getting Started with Custom PyFuncs: We'll begin with the simplest of examples, illuminating the core components and abstract methods essential for your custom PyFunc. Tackling Unsupported Libraries: Stepping up the complexity, we'll demonstrate how to integrate a model from an unsupported library into MLflow using custom PyFuncs. Overriding Default Inference Methods: Sometimes, the default isn't what you want. We'll show you how to override a model's inference method, for example, using predict_proba instead of predict.  By the end of this tutorial, you'll have a clear understanding of how to leverage custom PyFuncs in MLflow to cater to specialized needs, ensuring flexibility without compromising on the ease of use.   Models, Flavors, and PyFuncs in MLflow Understanding Pyfunc Components Full Notebooks         Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
deployment/index.html,"   Documentation  Deployment       Deployment  In the modern age of machine learning, deploying models effectively and consistently plays a pivotal role. The capability to serve predictions at scale, manage dependencies, and ensure reproducibility is paramount for businesses to derive actionable insights from their ML models. Whether it's for real-time predictions, batch processing, or interactive analyses, a robust model serving framework is essential. MLflow offers a comprehensive suite tailored for seamless model deployment. With its focus on ease of use, consistency, and adaptability, MLflow simplifies the model serving process, ensuring models are not just artifacts but actionable tools for decision-making.  The Power of MLflow in Model Serving   Dependency and Environment Management: MLflow ensures that the deployment environment mirrors the training environment, capturing all dependencies. This guarantees that models run consistently, regardless of where they're deployed. Packaging Models and Code: With MLflow, not just the model, but any supplementary code and configurations are packaged along with the deployment container. This ensures that the model can be executed seamlessly without any missing components.    Deployment Avenues  MLflow offers multiple ways to deploy your models based on your needs:  Local Flask Server: Quickly deploy your model in a containerized local environment. This server runs a model container with the dependencies defined during model saving. Local Flask Server with MLServer: Easily deploy a containerized model along with MLServer and KServe. This powerful alternative to a base Flask server leverages all of the benefits of Seldon's framework to enhance your inference capabilities. Remote Container Serving: Once a model container has been defined and built, it can be deployed to a remote serving environment. This is especially useful for cloud deployments where providers offer elastic container execution capabilities.   AzureML AWS Sagemaker    Kubernetes: For those invested in the Kubernetes ecosystem, MLflow supports model deployment to Kubernetes clusters, ensuring scalability and resilience. Databricks Model Serving: Directly deploy models in a Databricks environment, taking advantage of Databricks' performance optimizations and integrations.    Tutorials and Guides                           Deploying a Model to Kubernetes with MLflow                                       Explore an end-to-end guide on using MLflow to train a linear regression model, package it, and deploy it to a Kubernetes cluster.                     Understand how MLflow simplifies the entire process, from training to serving.                         Conclusion  Model serving is an intricate process, and MLflow is designed to make it as intuitive and reliable as possible. With its myriad deployment options and focus on consistency, MLflow ensures that models are ready for action, be it in a local environment, the cloud, or on a large-scale Kubernetes cluster. Dive into the provided tutorials, explore the functionalities, and streamline your model deployment journey with MLflow.        Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
deployment/deploy-model-to-kubernetes/index.html,"   Documentation  Deployment  Deploying a model to Kubernetes       Deploying a model to Kubernetes  This guide showcases how you can use MLflow end-to-end to:  Train a linear regression model Package the code that trains the model in a reusable and reproducible model format Deploy the model into a simple HTTP server that will enable you to score predictions  Note The modeling usecase shown here utilizes a common dataset to predict the quality of wine based on quantitative features like the wine's "fixed acidity", "pH", "residual sugar", and so on. The dataset is from UCI's machine learning repository. 1  Table of Contents  What You'll Need Training the Model Comparing the Models Packaging Training Code in a Conda Environment Specifying pip requirements using pip_requirements and extra_pip_requirements Serving the Model Deploy the Model to Seldon Core or KServe    What You'll Need  In order to execute the code in this guide locally, you'll need to:    Install MLflow and scikit-learn. There are two options for installing these dependencies:   Install MLflow with extra dependencies, including scikit-learn (via pip install mlflow[extras]) Install MLflow (via pip install mlflow) and install scikit-learn separately (via pip install scikit-learn)    Install conda Clone (download) the MLflow repository via git clone https://github.com/mlflow/mlflow cd into the examples directory within your clone of MLflow - we'll use this working directory for walking through the guide. We avoid running directly from our clone of MLflow as doing so would use MLflow from the source (master branch), rather than your PyPI installation of MLflow.     Install the MLflow package (via install.packages(""mlflow"")) Clone (download) the MLflow repository via git clone https://github.com/mlflow/mlflow setwd() into the examples directory within your clone of MLflow - we'll use this working directory for running the code in this guide. We avoid running directly from our clone of MLflow as doing so would cause the code to execute using MLflow from source, rather than your PyPI installation of MLflow.      Training the Model  First, train a linear regression model that takes two hyperparameters: alpha and l1_ratio.   The code is located at examples/sklearn_elasticnet_wine/train.py and is reproduced below. # The data set used in this example is from http://archive.ics.uci.edu/ml/datasets/Wine+Quality # P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis. # Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547-553, 2009.  import logging import sys import warnings from urllib.parse import urlparse  import numpy as np import pandas as pd from sklearn.linear_model import ElasticNet from sklearn.metrics import mean_absolute_error, mean_squared_error, r2_score from sklearn.model_selection import train_test_split  import mlflow import mlflow.sklearn from mlflow.models import infer_signature  logging.basicConfig(level=logging.WARN) logger = logging.getLogger(__name__)   def eval_metrics(actual, pred):     rmse = np.sqrt(mean_squared_error(actual, pred))     mae = mean_absolute_error(actual, pred)     r2 = r2_score(actual, pred)     return rmse, mae, r2   if __name__ == ""__main__"":     warnings.filterwarnings(""ignore"")     np.random.seed(40)      # Read the wine-quality csv file from the URL     csv_url = (         ""https://raw.githubusercontent.com/mlflow/mlflow/master/tests/datasets/winequality-red.csv""     )     try:         data = pd.read_csv(csv_url, sep="";"")     except Exception as e:         logger.exception(             ""Unable to download training & test CSV, check your internet connection. Error: %s"", e         )      # Split the data into training and test sets. (0.75, 0.25) split.     train, test = train_test_split(data)      # The predicted column is ""quality"" which is a scalar from [3, 9]     train_x = train.drop([""quality""], axis=1)     test_x = test.drop([""quality""], axis=1)     train_y = train[[""quality""]]     test_y = test[[""quality""]]      alpha = float(sys.argv[1]) if len(sys.argv) > 1 else 0.5     l1_ratio = float(sys.argv[2]) if len(sys.argv) > 2 else 0.5      with mlflow.start_run():         lr = ElasticNet(alpha=alpha, l1_ratio=l1_ratio, random_state=42)         lr.fit(train_x, train_y)          predicted_qualities = lr.predict(test_x)          (rmse, mae, r2) = eval_metrics(test_y, predicted_qualities)          print(f""Elasticnet model (alpha={alpha:f}, l1_ratio={l1_ratio:f}):"")         print(f""  RMSE: {rmse}"")         print(f""  MAE: {mae}"")         print(f""  R2: {r2}"")          mlflow.log_param(""alpha"", alpha)         mlflow.log_param(""l1_ratio"", l1_ratio)         mlflow.log_metric(""rmse"", rmse)         mlflow.log_metric(""r2"", r2)         mlflow.log_metric(""mae"", mae)          predictions = lr.predict(train_x)         signature = infer_signature(train_x, predictions)          tracking_url_type_store = urlparse(mlflow.get_tracking_uri()).scheme          # Model registry does not work with file store         if tracking_url_type_store != ""file"":             # Register the model             # There are other ways to use the Model Registry, which depends on the use case,             # please refer to the doc for more information:             # https://mlflow.org/docs/latest/model-registry.html#api-workflow             mlflow.sklearn.log_model(                 lr, ""model"", registered_model_name=""ElasticnetWineModel"", signature=signature             )         else:             mlflow.sklearn.log_model(lr, ""model"", signature=signature)   This example uses the familiar pandas, numpy, and sklearn APIs to create a simple machine learning model. The MLflow tracking APIs log information about each training run, like the hyperparameters alpha and l1_ratio, used to train the model and metrics, like the root mean square error, used to evaluate the model. The example also serializes the model in a format that MLflow knows how to deploy. You can run the example with default hyperparameters as follows: # Make sure the current working directory is 'examples' python sklearn_elasticnet_wine/train.py   Try out some other values for alpha and l1_ratio by passing them as arguments to train.py: # Make sure the current working directory is 'examples' python sklearn_elasticnet_wine/train.py <alpha> <l1_ratio>   Each time you run the example, MLflow logs information about your experiment runs in the directory mlruns.  Note If you would like to use the Jupyter notebook version of train.py, try out the example notebook at examples/sklearn_elasticnet_wine/train.ipynb.    The code is located at examples/r_wine/train.R and is reproduced below. # The data set used in this example is from http://archive.ics.uci.edu/ml/datasets/Wine+Quality # P. Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis. # Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547-553, 2009.  library(mlflow) library(glmnet) library(carrier)  set.seed(40)  # Read the wine-quality csv file data <- read.csv(""wine-quality.csv"")  # Split the data into training and test sets. (0.75, 0.25) split. sampled <- sample(1:nrow(data), 0.75 * nrow(data)) train <- data[sampled, ] test <- data[-sampled, ]  # The predicted column is ""quality"" which is a scalar from [3, 9] train_x <- as.matrix(train[, !(names(train) == ""quality"")]) test_x <- as.matrix(test[, !(names(train) == ""quality"")]) train_y <- train[, ""quality""] test_y <- test[, ""quality""]  alpha <- mlflow_param(""alpha"", 0.5, ""numeric"") lambda <- mlflow_param(""lambda"", 0.5, ""numeric"")  with(mlflow_start_run(), {     model <- glmnet(train_x, train_y, alpha = alpha, lambda = lambda, family= ""gaussian"", standardize = FALSE)     predictor <- crate(~ glmnet::predict.glmnet(!!model, as.matrix(.x)), !!model)     predicted <- predictor(test_x)      rmse <- sqrt(mean((predicted - test_y) ^ 2))     mae <- mean(abs(predicted - test_y))     r2 <- as.numeric(cor(predicted, test_y) ^ 2)      message(""Elasticnet model (alpha="", alpha, "", lambda="", lambda, ""):"")     message(""  RMSE: "", rmse)     message(""  MAE: "", mae)     message(""  R2: "", r2)      mlflow_log_param(""alpha"", alpha)     mlflow_log_param(""lambda"", lambda)     mlflow_log_metric(""rmse"", rmse)     mlflow_log_metric(""r2"", r2)     mlflow_log_metric(""mae"", mae)      mlflow_log_model(predictor, ""model"") })   This example uses the familiar glmnet package to create a simple machine learning model. The MLflow tracking APIs log information about each training run, like the hyperparameters alpha and lambda, used to train the model and metrics, like the root mean square error, used to evaluate the model. The example also serializes the model in a format that MLflow knows how to deploy. You can run the example with default hyperparameters as follows: # Make sure the current working directory is 'examples' mlflow_run(uri = ""r_wine"", entry_point = ""train.R"")   Try out some other values for alpha and lambda by passing them as arguments to train.R: # Make sure the current working directory is 'examples' mlflow_run(uri = ""r_wine"", entry_point = ""train.R"", parameters = list(alpha = 0.1, lambda = 0.5))   Each time you run the example, MLflow logs information about your experiment runs in the directory mlruns.  Note If you would like to use an R notebook version of train.R, try the example notebook at examples/r_wine/train.Rmd.      Comparing the Models  Next, use the MLflow UI to compare the models that you have produced. In the same current working directory as the one that contains the mlruns run:  mlflow ui   mlflow_ui()    and view it at http://localhost:5000. On this page, you can see a list of experiment runs with metrics you can use to compare the models.         You can  use the search feature to quickly filter out many models. For example, the query metrics.rmse < 0.8 returns all the models with root mean squared error less than 0.8. For more complex manipulations, you can download this table as a CSV and use your favorite data munging software to analyze it.   Packaging Training Code in a Conda Environment  Now that you have your training code, you can package it so that other data scientists can easily reuse the model, or so that you can run the training remotely, for example on Databricks.   You do this by using MLflow Projects conventions to specify the dependencies and entry points to your code. The sklearn_elasticnet_wine/MLproject file specifies that the project has the dependencies located in a Conda environment file called conda.yaml and has one entry point that takes two parameters: alpha and l1_ratio. name: tutorial  python_env: python_env.yaml  entry_points:   main:     parameters:       alpha: {type: float, default: 0.5}       l1_ratio: {type: float, default: 0.1}     command: ""python train.py {alpha} {l1_ratio}""   sklearn_elasticnet_wine/conda.yaml file lists the dependencies: name: tutorial channels:   - conda-forge dependencies:   - python=3.8   - pip   - pip:       - scikit-learn==1.2.0       - mlflow>=1.0       - pandas   To run this project, invoke mlflow run sklearn_elasticnet_wine -P alpha=0.42. After running this command, MLflow runs your training code in a new Conda environment with the dependencies specified in conda.yaml. If the repository has an MLproject file in the root you can also run a project directly from GitHub. This guide is duplicated in the https://github.com/mlflow/mlflow-example repository which you can run with mlflow run https://github.com/mlflow/mlflow-example.git -P alpha=5.0.   You do this by running mlflow_snapshot() to create an R dependencies packrat file called r-dependencies.txt. The R dependencies file lists the dependencies: # examples/r_wine/r-dependencies.txt  PackratFormat: 1.4 PackratVersion: 0.4.9.3 RVersion: 3.5.1 Repos: CRAN=https://cran.rstudio.com/  Package: BH Source: CRAN Version: 1.66.0-1 Hash: 4cc8883584b955ed01f38f68bc03af6d  Package: Matrix Source: CRAN Version: 1.2-14 Hash: 521aa8772a1941dfdb007bf532d19dde Requires: lattice  ...   To run this project, invoke: # Make sure the current working directory is 'examples' mlflow_run(""r_wine"", entry_point = ""train.R"", parameters = list(alpha = 0.2))   After running this command, MLflow runs your training code in a new R session. To restore the dependencies specified in r-dependencies.txt, you can run instead: mlflow_restore_snapshot() # Make sure the current working directory is 'examples' mlflow_run(""r_wine"", entry_point = ""train.R"", parameters = list(alpha = 0.2))   You can also run a project directly from GitHub. This guide is duplicated in the https://github.com/rstudio/mlflow-example repository which you can run with: mlflow_run(   ""train.R"",   ""https://github.com/rstudio/mlflow-example"",   parameters = list(alpha = 0.2) )       Specifying pip requirements using pip_requirements and extra_pip_requirements  """""" This example demonstrates how to specify pip requirements using `pip_requirements` and `extra_pip_requirements` when logging a model via `mlflow.*.log_model`. """"""  import tempfile  import sklearn import xgboost as xgb from sklearn.datasets import load_iris  import mlflow from mlflow.artifacts import download_artifacts from mlflow.models.signature import infer_signature   def read_lines(path):     with open(path) as f:         return f.read().splitlines()   def get_pip_requirements(run_id, artifact_path, return_constraints=False):     req_path = download_artifacts(run_id=run_id, artifact_path=f""{artifact_path}/requirements.txt"")     reqs = read_lines(req_path)      if return_constraints:         con_path = download_artifacts(             run_id=run_id, artifact_path=f""{artifact_path}/constraints.txt""         )         cons = read_lines(con_path)         return set(reqs), set(cons)      return set(reqs)   def main():     iris = load_iris()     dtrain = xgb.DMatrix(iris.data, iris.target)     model = xgb.train({}, dtrain)     predictions = model.predict(dtrain)     signature = infer_signature(dtrain.get_data(), predictions)      xgb_req = f""xgboost=={xgb.__version__}""     sklearn_req = f""scikit-learn=={sklearn.__version__}""      with mlflow.start_run() as run:         run_id = run.info.run_id          # Default (both `pip_requirements` and `extra_pip_requirements` are unspecified)         artifact_path = ""default""         mlflow.xgboost.log_model(model, artifact_path, signature=signature)         pip_reqs = get_pip_requirements(run_id, artifact_path)         assert xgb_req in pip_reqs, pip_reqs          # Overwrite the default set of pip requirements using `pip_requirements`         artifact_path = ""pip_requirements""         mlflow.xgboost.log_model(             model, artifact_path, pip_requirements=[sklearn_req], signature=signature         )         pip_reqs = get_pip_requirements(run_id, artifact_path)         assert sklearn_req in pip_reqs, pip_reqs          # Add extra pip requirements on top of the default set of pip requirements         # using `extra_pip_requirements`         artifact_path = ""extra_pip_requirements""         mlflow.xgboost.log_model(             model, artifact_path, extra_pip_requirements=[sklearn_req], signature=signature         )         pip_reqs = get_pip_requirements(run_id, artifact_path)         assert pip_reqs.issuperset({xgb_req, sklearn_req}), pip_reqs          # Specify pip requirements using a requirements file         with tempfile.NamedTemporaryFile(""w"", suffix="".requirements.txt"") as f:             f.write(sklearn_req)             f.flush()              # Path to a pip requirements file             artifact_path = ""requirements_file_path""             mlflow.xgboost.log_model(                 model, artifact_path, pip_requirements=f.name, signature=signature             )             pip_reqs = get_pip_requirements(run_id, artifact_path)             assert sklearn_req in pip_reqs, pip_reqs              # List of pip requirement strings             artifact_path = ""requirements_file_list""             mlflow.xgboost.log_model(                 model,                 artifact_path,                 pip_requirements=[xgb_req, f""-r {f.name}""],                 signature=signature,             )             pip_reqs = get_pip_requirements(run_id, artifact_path)             assert pip_reqs.issuperset({xgb_req, sklearn_req}), pip_reqs          # Using a constraints file         with tempfile.NamedTemporaryFile(""w"", suffix="".constraints.txt"") as f:             f.write(sklearn_req)             f.flush()              artifact_path = ""constraints_file""             mlflow.xgboost.log_model(                 model,                 artifact_path,                 pip_requirements=[xgb_req, f""-c {f.name}""],                 signature=signature,             )             pip_reqs, pip_cons = get_pip_requirements(                 run_id, artifact_path, return_constraints=True             )             assert pip_reqs.issuperset({xgb_req, ""-c constraints.txt""}), pip_reqs             assert pip_cons == {sklearn_req}, pip_cons   if __name__ == ""__main__"":     main()     Serving the Model  Now that you have packaged your model using the MLproject convention and have identified the best model, it is time to deploy the model using MLflow Models. An MLflow Model is a standard format for packaging machine learning models that can be used in a variety of downstream tools — for example, real-time serving through a REST API or batch inference on Apache Spark. In the example training code, after training the linear regression model, a function in MLflow saved the model as an artifact within the run.   mlflow.sklearn.log_model(lr, ""model"")   To view this artifact, you can use the UI again. When you click a date in the list of experiment runs you'll see this page.  At the bottom, you can see that the call to mlflow.sklearn.log_model produced two files in /Users/mlflow/mlflow-prototype/mlruns/0/7c1a0d5c42844dcdb8f5191146925174/artifacts/model. The first file, MLmodel, is a metadata file that tells MLflow how to load the model. The second file, model.pkl, is a serialized version of the linear regression model that you trained. In this example, you can use this MLmodel format with MLflow to deploy a local REST server that can serve predictions. To deploy the server, run (replace the path with your model's actual path): mlflow models serve -m /Users/mlflow/mlflow-prototype/mlruns/0/7c1a0d5c42844dcdb8f5191146925174/artifacts/model -p 1234    Note The version of Python used to create the model must be the same as the one running mlflow models serve. If this is not the case, you may see the error UnicodeDecodeError: 'ascii' codec can't decode byte 0x9f in position 1: ordinal not in range(128) or raise ValueError, ""unsupported pickle protocol: %d"".  Once you have deployed the server, you can pass it some sample data and see the predictions. The following example uses curl to send a JSON-serialized pandas DataFrame with the split orientation to the model server. For more information about the input data formats accepted by the model server, see the MLflow deployment tools documentation. # On Linux and macOS curl -X POST -H ""Content-Type:application/json"" --data '{""dataframe_split"": {""columns"":[""fixed acidity"", ""volatile acidity"", ""citric acid"", ""residual sugar"", ""chlorides"", ""free sulfur dioxide"", ""total sulfur dioxide"", ""density"", ""pH"", ""sulphates"", ""alcohol""],""data"":[[6.2, 0.66, 0.48, 1.2, 0.029, 29, 75, 0.98, 3.33, 0.39, 12.8]]}}' http://127.0.0.1:1234/invocations  # On Windows curl -X POST -H ""Content-Type:application/json"" --data ""{\""dataframe_split\"": {\""columns\"":[\""fixed acidity\"", \""volatile acidity\"", \""citric acid\"", \""residual sugar\"", \""chlorides\"", \""free sulfur dioxide\"", \""total sulfur dioxide\"", \""density\"", \""pH\"", \""sulphates\"", \""alcohol\""],\""data\"":[[6.2, 0.66, 0.48, 1.2, 0.029, 29, 75, 0.98, 3.33, 0.39, 12.8]]}}"" http://127.0.0.1:1234/invocations   the server should respond with output similar to: [6.379428821398614]     mlflow_log_model(predictor, ""model"")   To view this artifact, you can use the UI again. When you click a date in the list of experiment runs you'll see this page.  At the bottom, you can see that the call to mlflow_log_model() produced two files in mlruns/0/c2a7325210ef4242bd4631cec8f92351/artifacts/model/. The first file, MLmodel, is a metadata file that tells MLflow how to load the model. The second file, r_model.bin, is a serialized version of the linear regression model that you trained. In this example, you can use this MLmodel format with MLflow to deploy a local REST server that can serve predictions. To deploy the server, run: mlflow_rfunc_serve(model_uri=""mlruns/0/c2a7325210ef4242bd4631cec8f92351/artifacts/model"", port=8090)   This initializes a REST server and opens a Swagger interface to perform predictions against the REST API:   Note By default, a model is served using the R packages available. To ensure the environment serving the prediction function matches the model, set restore = TRUE when calling mlflow_rfunc_serve().  To serve a prediction, enter this in the Swagger UI: {   ""fixed acidity"": 6.2,   ""volatile acidity"": 0.66,   ""citric acid"": 0.48,   ""residual sugar"": 1.2,   ""chlorides"": 0.029,   ""free sulfur dioxide"": 29,   ""total sulfur dioxide"": 75,   ""density"": 0.98,   ""pH"": 3.33,   ""sulphates"": 0.39,   ""alcohol"": 12.8 }   which should return something like: [   [     6.4287492410792   ] ]   Or run: # On Linux and macOS curl -X POST ""http://127.0.0.1:8090/predict/"" -H ""accept: application/json"" -H ""Content-Type: application/json"" -d ""{""fixed acidity"": 6.2, ""volatile acidity"": 0.66, ""citric acid"": 0.48, ""residual sugar"": 1.2, ""chlorides"": 0.029, ""free sulfur dioxide"": 29, ""total sulfur dioxide"": 75, ""density"": 0.98, ""pH"": 3.33, ""sulphates"": 0.39, ""alcohol"": 12.8}""  # On Windows curl -X POST ""http://127.0.0.1:8090/predict/"" -H ""accept: application/json"" -H ""Content-Type: application/json"" -d ""{\""fixed acidity\"": 6.2, \""volatile acidity\"": 0.66, \""citric acid\"": 0.48, \""residual sugar\"": 1.2, \""chlorides\"": 0.029, \""free sulfur dioxide\"": 29, \""total sulfur dioxide\"": 75, \""density\"": 0.98, \""pH\"": 3.33, \""sulphates\"": 0.39, \""alcohol\"": 12.8}""   the server should respond with output similar to: [[6.4287492410792]]       Deploy the Model to Seldon Core or KServe  After training and testing our model, we are now ready to deploy it to production. MLflow allows you to serve your model using MLServer, which is already used as the core Python inference server in Kubernetes-native frameworks including Seldon Core and KServe (formerly known as KFServing). Therefore, we can leverage this support to build a Docker image compatible with these frameworks.  Note This an optional step, which is currently only available for Python models. This step also requires some basic Kubernetes knowledge, including familiarity with kubectl.  To build a Docker image containing our model, we can use the mlflow models build-docker subcommand, alongside the --enable-mlserver flag. For example, to build a image named my-docker-image, we could do: mlflow models build-docker \   -m /Users/mlflow/mlflow-prototype/mlruns/0/7c1a0d5c42844dcdb8f5191146925174/artifacts/model \   -n my-docker-image \   --enable-mlserver   Once we have our image built, the next step will be to deploy it to our cluster. One way to do this is by applying the respective Kubernetes manifests through the kubectl CLI: kubectl apply -f my-manifest.yaml     This step assumes that you've got kubectl access to a Kubernetes cluster already setup with Seldon Core. To read more on how to set this up, you can refer to the Seldon Core quickstart guide. apiVersion: machinelearning.seldon.io/v1 kind: SeldonDeployment metadata:   name: mlflow-model spec:   protocol: v2   predictors:     - name: default       graph:         name: mlflow-model         type: MODEL       componentSpecs:         - spec:             containers:               - name: mlflow-model                 image: my-docker-image     This step assumes that you've got kubectl access to a Kubernetes cluster already setup with KServe. To read more on how to set this up, you can refer to the KServe quickstart guide. apiVersion: serving.kserve.io/v1beta1 kind: InferenceService metadata:   name: mlflow-model spec:   predictor:     containers:       - name: mlflow-model         image: my-docker-image         ports:           - containerPort: 8080             protocol: TCP         env:           - name: PROTOCOL             value: v2     Congratulations on finishing the guide!  1  Cortez, A. Cerdeira, F. Almeida, T. Matos and J. Reis. Modeling wine preferences by data mining from physicochemical properties. In Decision Support Systems, Elsevier, 47(4):547-553, 2009.           Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "
introduction/index.html,"   Documentation  What is MLflow?       What is MLflow?  Stepping into the world of Machine Learning (ML) is an exciting journey, but it often comes with complexities that can hinder innovation and experimentation. MLflow is a solution to many of these issues in this dynamic landscape, offering tools and simplifying processes to streamline the ML lifecycle and foster collaboration among ML practitioners. Whether you're an individual researcher, a member of a large team, or somewhere in between, MLflow provides a unified platform to navigate the intricate maze of model development, deployment, and management. MLflow aims to enable innovation in ML solution development by streamlining otherwise cumbersome logging, organization, and lineage concerns that are unique to model development. This focus allows you to ensure that your ML projects are robust, transparent, and ready for real-world challenges. Read on to discover the core components of MLflow and understand the unique advantages it brings to the complex workflows associated with model development and management.  Core Components of MLflow  MLflow, at its core, provides a suite of tools aimed at simplifying the ML workflow. It is tailored to assist ML practitioners throughout the various stages of ML development and deployment. Despite its expansive offerings, MLflow's functionalities are rooted in several foundational components:  Tracking: MLflow Tracking provides both an API and UI dedicated to the logging of parameters, code versions, metrics, and artifacts during the ML process. This centralized repository captures details such as parameters, metrics, artifacts, data, and environment configurations, giving teams insight into their models' evolution over time. Whether working in standalone scripts, notebooks, or other environments, Tracking facilitates the logging of results either to local files or a server, making it easier to compare multiple runs across different users. Model Registry: A systematic approach to model management, the Model Registry assists in handling different versions of models, discerning their current state, and ensuring a smooth transition from development to production. It offers a centralized model store, APIs, and UI to collaboratively manage an MLflow Model's full lifecycle, including model lineage, versioning, stage transitions, and annotations. AI Gateway: This server, equipped with a set of standardized APIs, streamlines access to both SaaS and OSS LLM models. It serves as a unified interface, bolstering security through authenticated access, and offers a common set of APIs for prominent LLMs. Evaluate: Designed for in-depth model analysis, this set of tools facilitates objective model comparison, be it traditional ML algorithms or cutting-edge LLMs. Prompt Engineering UI: A dedicated environment for prompt engineering, this UI-centric component provides a space for prompt experimentation, refinement, evaluation, testing, and deployment. Recipes: Serving as a guide for structuring ML projects, Recipes, while offering recommendations, are focused on ensuring functional end results optimized for real-world deployment scenarios. Projects: MLflow Projects standardize the packaging of ML code, workflows, and artifacts, akin to an executable. Each project, be it a directory with code or a Git repository, employs a descriptor or convention to define its dependencies and execution method.  By integrating these core components, MLflow offers an end-to-end platform, ensuring efficiency, consistency, and traceability throughout the ML lifecycle.   Why Use MLflow?  The machine learning (ML) process is intricate, comprising various stages, from data preprocessing to model deployment and monitoring. Ensuring productivity and efficiency throughout this lifecycle poses several challenges:  Experiment Management: It's tough to keep track of the myriad experiments, especially when working with files or interactive notebooks. Determining which combination of data, code, and parameters led to a particular result can become a daunting task. Reproducibility: Ensuring consistent results across runs is not trivial. Beyond just tracking code versions and parameters, capturing the entire environment, including library dependencies, is critical. This becomes even more challenging when collaborating with other data scientists or when scaling the code to different platforms. Deployment Consistency: With the plethora of ML libraries available, there's often no standardized way to package and deploy models. Custom solutions can lead to inconsistencies, and the crucial link between a model and the code and parameters that produced it might be lost. Model Management: As data science teams produce numerous models, managing these models, their versions, and stage transitions becomes a significant hurdle. Without a centralized platform, managing model lifecycles, from development to staging to production, becomes unwieldy. Library Agnosticism: While individual ML libraries might offer solutions to some of the challenges, achieving the best results often involves experimenting across multiple libraries. A platform that offers compatibility with various libraries while ensuring models are usable as reproducible "black boxes" is essential.  MLflow addresses these challenges by offering a unified platform tailored for the entire ML lifecycle. Its benefits include:  Traceability: With tools like the Tracking Server, every experiment is logged, ensuring that teams can trace back and understand the evolution of models. Consistency: Be it accessing models through the AI Gateway or structuring projects with MLflow Recipes, MLflow promotes a consistent approach, reducing both the learning curve and potential errors. Flexibility: MLflow's library-agnostic design ensures compatibility with a wide range of machine learning libraries. It offers comprehensive support across different programming languages, backed by a robust REST API, CLI, and APIs for Python API, R API, and Java API.  By simplifying the complex landscape of ML workflows, MLflow empowers data scientists and developers to focus on building and refining models, ensuring a streamlined path from experimentation to production.   Who Uses MLflow?  Throughout the lifecycle of a particular project, there are components within MLflow that are designed to cater to different needs.    MLflow's versatility enhances workflows across various roles, from data scientists to prompt engineers, extending its impact beyond just the confines of a Data Science team.  Data Scientists leverage MLflow for:  Experiment tracking and hypothesis testing persistence. Code structuring for better reproducibility. Model packaging and dependency management. Evaluating hyperparameter tuning selection boundaries. Comparing the results of model retraining over time. Reviewing and selecting optimal models for deployment.  MLOps Professionals utilize MLflow to:  Manage the lifecycles of trained models, both pre and post deployment. Deploy models securely to production environments. Audit and review candidate models prior to deployment. Manage deployment dependencies.  Data Science Managers interact with MLflow by:  Reviewing the outcomes of experimentation and modeling activities. Collaborating with teams to ensure that modeling objectives align with business goals.  Prompt Engineering Users use MLflow for:  Evaluating and experimenting with large language models. Crafting custom prompts and persisting their candidate creations. Deciding on the best base model suitable for their specific project requirements.     Use Cases of MLflow  MLflow is versatile, catering to diverse machine learning scenarios. Here are some typical use cases:  Experiment Tracking: A data science team leverages MLflow Tracking to log parameters and metrics for experiments within a particular domain. Using the MLflow UI, they can compare results and fine-tune their solution approach. The outcomes of these experiments are preserved as MLflow models. Model Selection and Deployment: MLOps engineers employ the MLflow UI to assess and pick the top-performing models. The chosen model is registered in the MLflow Registry, allowing for monitoring its real-world performance. Model Performance Monitoring: Post deployment, MLOps engineers utilize the MLflow Registry to gauge the model's efficacy, juxtaposing it against other models in a live environment. Collaborative Projects: Data scientists embarking on new ventures organize their work as an MLflow Project. This structure facilitates easy sharing and parameter modifications, promoting collaboration.    Scalability in MLflow  MLflow is architected to seamlessly integrate with diverse data environments, from small datasets to Big Data applications. It's built with the understanding that quality machine learning outcomes often hinge on robust data sources, and as such, scales adeptly to accommodate varying data needs. Here's how MLflow addresses scalability across different dimensions:  Distributed Execution: MLflow runs can operate on distributed clusters. For instance, integration with Apache Spark allows for distributed processing. Furthermore, runs can be initiated on the distributed infrastructure of your preference, with results relayed to a centralized Tracking Server for analysis. Notably, MLflow offers an integrated API to initiate runs on Databricks. Parallel Runs: For use cases like hyperparameter tuning, MLflow can orchestrate multiple runs simultaneously, each with distinct parameters. Interoperability with Distributed Storage: MLflow Projects can interface with distributed storage solutions, including Azure ADLS, Azure Blob Storage, AWS S3, Cloudflare R2 and DBFS. Whether it's automatically fetching files to a local environment or interfacing with a distributed storage URI directly, MLflow ensures that projects can handle extensive datasets – even scenarios like processing a 100 TB file. Centralized Model Management with Model Registry: Large-scale organizations can benefit from the MLflow Model Registry, a unified platform tailored for collaborative model lifecycle management. In environments where multiple data science teams might be concurrently developing numerous models, the Model Registry proves invaluable. It streamlines model discovery, tracks experiments, manages versions, and facilitates understanding a model's intent across different teams.  By addressing these scalability dimensions, MLflow ensures that users can capitalize on its capabilities regardless of their data environment's size or complexity.        Previous Next                © MLflow Project, a Series of LF Projects, LLC. All rights reserved.          "