Business Software for NVIDIA Triton Inference Server

Build, deploy, and scale machine learning (ML) models faster, with fully managed ML tools for any use case. Through Vertex AI Workbench, Vertex AI is natively integrated with BigQuery, Dataproc, and Spark. You can use BigQuery ML to create and execute machine learning models in BigQuery using standard SQL queries on existing business intelligence tools and spreadsheets, or you can export datasets from BigQuery directly into Vertex AI Workbench and run your models from there. Use Vertex Data Labeling to generate highly accurate labels for your data collection. Vertex AI Agent Builder enables developers to create and deploy enterprise-grade generative AI applications. It offers both no-code and code-first approaches, allowing users to build AI agents using natural language instructions or by leveraging frameworks like LangChain and LlamaIndex.

An end-to-end open source machine learning platform. TensorFlow is an end-to-end open source platform for machine learning. It has a comprehensive, flexible ecosystem of tools, libraries and community resources that lets researchers push the state-of-the-art in ML and developers easily build and deploy ML powered applications. Build and train ML models easily using intuitive high-level APIs like Keras with eager execution, which makes for immediate model iteration and easy debugging. Easily train and deploy models in the cloud, on-prem, in the browser, or on-device no matter what language you use. A simple and flexible architecture to take new ideas from concept to code, to state-of-the-art models, and to publication faster. Build, deploy, and experiment easily with TensorFlow.

Amazon Elastic Container Service (Amazon ECS) is a fully managed container orchestration service. Customers such as Duolingo, Samsung, GE, and Cook Pad use ECS to run their most sensitive and mission-critical applications because of its security, reliability, and scalability. ECS is a great choice to run containers for several reasons. First, you can choose to run your ECS clusters using AWS Fargate, which is serverless compute for containers. Fargate removes the need to provision and manage servers, lets you specify and pay for resources per application, and improves security through application isolation by design. Second, ECS is used extensively within Amazon to power services such as Amazon SageMaker, AWS Batch, Amazon Lex, and Amazon.com’s recommendation engine, ensuring ECS is tested extensively for security, reliability, and availability.

Kubernetes (K8s) is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery. Kubernetes builds upon 15 years of experience of running production workloads at Google, combined with best-of-breed ideas and practices from the community. Designed on the same principles that allows Google to run billions of containers a week, Kubernetes can scale without increasing your ops team. Whether testing locally or running a global enterprise, Kubernetes flexibility grows with you to deliver your applications consistently and easily no matter how complex your need is. Kubernetes is open source giving you the freedom to take advantage of on-premises, hybrid, or public cloud infrastructure, letting you effortlessly move workloads to where it matters to you.

Run advanced apps on a secured and managed Kubernetes service. GKE is an enterprise-grade platform for containerized applications, including stateful and stateless, AI and ML, Linux and Windows, complex and simple web apps, API, and backend services. Leverage industry-first features like four-way auto-scaling and no-stress management. Optimize GPU and TPU provisioning, use integrated developer tools, and get multi-cluster support from SREs. Start quickly with single-click clusters. Leverage a high-availability control plane including multi-zonal and regional clusters. Eliminate operational overhead with auto-repair, auto-upgrade, and release channels. Secure by default, including vulnerability scanning of container images and data encryption. Integrated Cloud Monitoring with infrastructure, application, and Kubernetes-specific views. Speed up app development without sacrificing security.

Transition seamlessly between eager and graph modes with TorchScript, and accelerate the path to production with TorchServe. Scalable distributed training and performance optimization in research and production is enabled by the torch-distributed backend. A rich ecosystem of tools and libraries extends PyTorch and supports development in computer vision, NLP and more. PyTorch is well supported on major cloud platforms, providing frictionless development and easy scaling. Select your preferences and run the install command. Stable represents the most currently tested and supported version of PyTorch. This should be suitable for many users. Preview is available if you want the latest, not fully tested and supported, 1.10 builds that are generated nightly. Please ensure that you have met the prerequisites (e.g., numpy), depending on your package manager. Anaconda is our recommended package manager since it installs all dependencies.

Amazon SageMaker is an advanced machine learning service that provides an integrated environment for building, training, and deploying machine learning (ML) models. It combines tools for model development, data processing, and AI capabilities in a unified studio, enabling users to collaborate and work faster. SageMaker supports various data sources, such as Amazon S3 data lakes and Amazon Redshift data warehouses, while ensuring enterprise security and governance through its built-in features. The service also offers tools for generative AI applications, making it easier for users to customize and scale AI use cases. SageMaker’s architecture simplifies the AI lifecycle, from data discovery to model deployment, providing a seamless experience for developers.

Power your metrics and alerting with a leading open-source monitoring solution. Prometheus fundamentally stores all data as time series: streams of timestamped values belonging to the same metric and the same set of labeled dimensions. Besides stored time series, Prometheus may generate temporary derived time series as the result of queries. Prometheus provides a functional query language called PromQL (Prometheus Query Language) that lets the user select and aggregate time series data in real time. The result of an expression can either be shown as a graph, viewed as tabular data in Prometheus's expression browser, or consumed by external systems via the HTTP API. Prometheus is configured via command-line flags and a configuration file. While the command-line flags configure immutable system parameters (such as storage locations, amount of data to keep on disk and in memory, etc.). Download: https://sourceforge.net/projects/prometheus.mirror/

FauxPilot is an open source, self-hosted alternative to GitHub Copilot. It utilizes the SalesForce CodeGen models on NVIDIA's Triton Inference Server with the FasterTransformer backend for local code generation. It requires Docker, an NVIDIA GPU with sufficient VRAM, and the ability to split the model across multiple GPUs if needed. The setup involves downloading models from Hugging Face and converting them for FasterTransformer compatibility.

​LiteLLM is a versatile platform designed to streamline interactions with over 100 Large Language Models (LLMs) through a unified interface. It offers both a Proxy Server (LLM Gateway) and a Python SDK, enabling developers to integrate various LLMs seamlessly into their applications. The Proxy Server facilitates centralized management, allowing for load balancing, cost tracking across projects, and consistent input/output formatting compatible with OpenAI standards. This setup supports multiple providers. It ensures robust observability by generating unique call IDs for each request, aiding in precise tracking and logging across systems. Developers can leverage pre-defined callbacks to log data using various tools. For enterprise users, LiteLLM offers advanced features like Single Sign-On (SSO), user management, and professional support through dedicated channels like Discord and Slack.

Accelerate the end-to-end machine learning lifecycle. Empower developers and data scientists with a wide range of productive experiences for building, training, and deploying machine learning models faster. Accelerate time to market and foster team collaboration with industry-leading MLOps—DevOps for machine learning. Innovate on a secure, trusted platform, designed for responsible ML. Productivity for all skill levels, with code-first and drag-and-drop designer, and automated machine learning. Robust MLOps capabilities that integrate with existing DevOps processes and help manage the complete ML lifecycle. Responsible ML capabilities – understand models with interpretability and fairness, protect data with differential privacy and confidential computing, and control the ML lifecycle with audit trials and datasheets. Best-in-class support for open-source frameworks and languages including MLflow, Kubeflow, ONNX, PyTorch, TensorFlow, Python, and R.

The fully managed Azure Kubernetes Service (AKS) makes deploying and managing containerized applications easy. It offers serverless Kubernetes, an integrated continuous integration and continuous delivery (CI/CD) experience, and enterprise-grade security and governance. Unite your development and operations teams on a single platform to rapidly build, deliver, and scale applications with confidence. Elastic provisioning of additional capacity without the need to manage the infrastructure. Add event-driven autoscaling and triggers through KEDA. Faster end-to-end development experience with Azure Dev Spaces including integration with Visual Studio Code Kubernetes tools, Azure DevOps, and Azure Monitor. Advanced identity and access management using Azure Active Directory, and dynamic rules enforcement across multiple clusters with Azure Policy. Available in more regions than any other cloud providers.

Amazon Elastic Kubernetes Service (Amazon EKS) is a fully managed Kubernetes service. Customers such as Intel, Snap, Intuit, GoDaddy, and Autodesk trust EKS to run their most sensitive and mission-critical applications because of its security, reliability, and scalability. EKS is the best place to run Kubernetes for several reasons. First, you can choose to run your EKS clusters using AWS Fargate, which is serverless compute for containers. Fargate removes the need to provision and manage servers, lets you specify and pay for resources per application, and improves security through application isolation by design. Second, EKS is deeply integrated with services such as Amazon CloudWatch, Auto Scaling Groups, AWS Identity and Access Management (IAM), and Amazon Virtual Private Cloud (VPC), providing you a seamless experience to monitor, scale, and load-balance your applications.

Run, manage, control and secure the apps, data and IT that run your business, from edge to cloud. HPE Ezmeral advances digital transformation initiatives by shifting time and resources from IT operations to innovations. Modernize your apps. Simplify your Ops. And harness data to go from insights to impact. Accelerate time-to-value by deploying Kubernetes at scale with integrated persistent data storage for app modernization on bare metal or VMs, in your data center, on any cloud or at the edge. Harness data and get insights faster by operationalizing the end-to-end process to build data pipelines. Bring DevOps agility to the machine learning lifecycle, and deliver a unified data fabric. Boost efficiency and agility in IT Ops with automation and advanced artificial intelligence. And provide security and control to eliminate risk and reduce costs. HPE Ezmeral Container Platform provides an enterprise-grade platform to deploy Kubernetes at scale for a wide range of use cases.

Tencent Cloud is a secure, reliable and high-performance cloud compute service provided by Tencent. Tencent is now the largest Internet company in China, and even Asia. It's providing services for hundreds of millions of people via its flagship products like QQ and WeChat. Cloud Virtual Machine (CVM) provides safe and reliable elastic computing services. You can expand or reduce computing resources in real time, adapt to changing business needs, and only need to charge based on the resources actually used. Using CVM can greatly reduce your software and hardware procurement costs and simplify IT operation and maintenance. Cloud databases provide enterprises with complete relational databases, non-relational databases, analytical databases and database ecological tools.

A hybrid front-end seamlessly transitions between Gluon eager imperative mode and symbolic mode to provide both flexibility and speed. Scalable distributed training and performance optimization in research and production is enabled by the dual parameter server and Horovod support. Deep integration into Python and support for Scala, Julia, Clojure, Java, C++, R and Perl. A thriving ecosystem of tools and libraries extends MXNet and enables use-cases in computer vision, NLP, time series and more. Apache MXNet is an effort undergoing incubation at The Apache Software Foundation (ASF), sponsored by the Apache Incubator. Incubation is required of all newly accepted projects until a further review indicates that the infrastructure, communications, and decision-making process have stabilized in a manner consistent with other successful ASF projects. Join the MXNet scientific community to contribute, learn, and get answers to your questions.

Alibaba CloudAP offers enterprise-level Wi-Fi management capability and provides Wi-Fi and BLE network coverage for places such as campuses, schools, hospitals, shopping malls, or supermarkets. CloudAP can be remotely managed and controlled by using CloudAC, and supports fast deployment of the Wi-Fi network and BLE network. You do not need to deploy an AC for CloudAP or deploy an authentication system for network access, which are necessary for traditional Wi-Fi products. This greatly reduces costs. CloudAP can be wirelessly powered through Power Over Ethernet (PoE) ports, which makes onsite device installation easier.

NVIDIA Morpheus is a GPU-accelerated, end-to-end AI framework that enables developers to create optimized applications for filtering, processing, and classifying large volumes of streaming cybersecurity data. Morpheus incorporates AI to reduce the time and cost associated with identifying, capturing, and acting on threats, bringing a new level of security to the data center, cloud, and edge. Morpheus also extends human analysts’ capabilities with generative AI by automating real-time analysis and responses, producing synthetic data to train AI models that identify risks accurately and run what-if scenarios. Morpheus is available as open-source software on GitHub for developers interested in using the latest pre-release features and who want to build from source. Get unlimited usage on all clouds, access to NVIDIA AI experts, and long-term support for production deployments with a purchase of NVIDIA AI Enterprise.

NVIDIA's DeepStream SDK is a comprehensive streaming analytics toolkit based on GStreamer, designed for AI-based multi-sensor processing, including video, audio, and image understanding. It enables developers to create stream-processing pipelines that incorporate neural networks and complex tasks like tracking, video encoding/decoding, and rendering, facilitating real-time analytics on various data types. DeepStream is integral to NVIDIA Metropolis, a platform for building end-to-end services that transform pixel and sensor data into actionable insights. The SDK offers a powerful and flexible environment suitable for a wide range of industries, supporting multiple programming options such as C/C++, Python, and Graph Composer's intuitive UI. It allows for real-time insights by understanding rich, multi-modal sensor data at the edge and supports managed AI services through deployment in cloud-native containers orchestrated with Kubernetes.