PerfKit Benchmarker Files

Breaking changes:

• Remove ubuntu1404 and debian OS types.
  • Ubuntu 14.04 is now in LTS and has not been fully maintained since April 2019.
  • debian os_type alias referred to Ubuntu 14.04 on all clouds.
  • Ubuntu 16.04 is now the default.
• Remove the Runtime metric for Horovod.
• Change windows201X os_types to windows201X_core and windows201X_base to windows201X_desktop.
  • Windows Server Core is now referred to as windows201X_core.
    • This is recommended for most benchmarks, because PKB does not benefit from GUIs except for debugging.
  • Windows Server with Desktop Experience is now referred to as windows201X_desktop.
  • See the microsoft documentation for more information on the differences.
• Remove deprecated versionless windows os_type.
  • You will have to specify windows2012_core.
• Removed deprecated versionless rhel os_type.
  • You will have to specify rhel7 on most providers or amazonlinux1 on AWS.
• Deprecate Amazon Linux 1 amazonlinux1 os_type.
  • AL1 is EOL on 2021-01-01: https://aws.amazon.com/blogs/aws/update-on-amazon-linux-ami-end-of-life/ at which point it will be removed from PKB.
  • You can use the recommended amazonlinux2 instead.
• Remove ping_also_run_using_external_ip from ping benchmark
  • You should now use the ip_addresses flag to specify whether to test with internal IPs, external IPs or both. This brings ping into alignment with how other network benchmarks function.
• The flag --emr-release-label is now deprecated in favor of setting dpb_service.version.
• In EMR DPB benchmarks setting: dpb_service.version is now Required.

New features:

• Add infiniband support in nccl benchmark.
• Added AwsVpcS3Endpoint for VPC connectivity to S3.
• Add --tcp_max_receive_buffer flag to set net.ipv4.tcp_rmem sysctl value
• Add --tcp_max_send_buffer flag to set net.ipv4.tcp_wmem sysctl value
• Add --rmem_max flag to set net.core.rmem_max sysctl value
• Add --wmem_max flag to set net.core.wmem_max sysctl value
• Add --os_type=debian9 support for AWS and Azure Providers.
• Add --os_type=debian10 support for GCP, AWS and Azure Providers.
• Add function to tune NCCL parameters.
• Add Coremark demo scripts at PerfKitBenchmarker/tools/demos/coremark.
• Add RHEL 8 to AWS, Azure and GCP providers
• Add CentOS 8 to Azure and GCP providers
• Add BERT models in Horovod benchmark
• Add timeline support in Horovod benchmark
• Add GceNfsService to support Filestore on GCP.
• Add maven package and use it in ycsb.
• Add replication cluster support to bigtable.
• Add AWS and Azure VPC peering support
• Add Snowflake Warehouse support
• Add large_scale_boot benchmark for linux and windows.
• Add Python client library for GCS alongside Boto connection library.
• Enabled OpenMPI version selection
• Adds support for running network benchmarks across a VPN.
  • Includes GCP implementation with static routes.
• Add a directory for PKB tutorials.
• Add two new tutorials: one for beginners and one for network dashboard.
• Update Boto client library for S3 to boto3.
• Add MultiStreamDelete to Object Storage API Tests.
• Add GCE Placement Group support. By default, placement groups are not created.
• Add ICMP ping support for Azure using EXTERNAL ip address.

Enhancements:

• Update Specsfs2014 to use the SP2 update rather than SP1.
• Update Multichase version to enable AARCH64 build support.
• Added flag --cassandra_maven_repo_url to use a maven repo when building cassandra, for example https://maven-central.storage-download.googleapis.com/maven2
• Optimize Azure Files settings.
• Migrate EKS cluster creation to use eksctl (requires a local eksctl).
  • Deprecate --eks_zones in favor of setting container_cluster.vm_spec.AWS.zone.
  • Make container_cluster.vm_spec.AWS.zone accept a region or comma separated list of zones.
  • Delete the obsolete --eks-verify-ssl
• Separate NVIDIA driver functionality from CUDA toolkit
• Added --runspec_tar option to SpecCPU (specifying installation tarball).
• Added generic support for using PreprovisionedData
  • Added the preprovision_ignore_checksum flag that can be used when downloading content that is not registered in PREPROVISIONED_DATA
  • Allow _InstallData in virtual_machine.py to catch the NotImplemented error, in order to proceed to downloading resources via their URL
• Implemented PreprovisionedData in the Maven package and make Maven more robust
  • Added maven_mirror_url to specify a custom Maven mirror
  • Added sha256 checksums for a few Maven versions
  • Made retrieving JAVA_HOME more robust
  • Fixed a java lib issue on CentOS
• Add OpenFOAM support for timing individual commands.
• Added --gce_subnet_name to specify existing GCE subnet to use.
• Run a Full Sweep (Create/Stat/Delete) on Mdtest when not dropping caches.
• Added version field to dpb_service configs. This corresponds to image version in Google Cloud Dataproc and release label in AWS EMR.
• Support delete timing on provider implementations of object_storage_api script
• Enable support for AWS's m6g family.
• Add SPEC CPU 2017 flag allowing to build but not run a suite.
• Enable support for specifying newer versions of GCC than what the OS supports by default.
• Sort AWS AMIs by CreationDate instead of name to get latest image.
• Add Spec17 configs compatible with v1.1.
• Better support for pytyping.
  • Make BaseVirtualMachine inherit from BaseOsMixin.
• Add connect_via_internal_ip flag for testing windows tests over the internal IP, similar to the ssh_via_internal_ip flag for linux.
  • Deprecate --ssh_via_internal_ip in favor of this new flag.

Bug fixes and maintenance updates:

• AWSBaseVirtualMachine subclasses require IMAGE_OWNER and IMAGE_NAME_FILTER. Fixes issue where some windows 2012 AMIs were selected from the wrong project.
• Retry yum install commands to bypass transient issues.
• Azure defaults to no placement group created.
• Add Sql Server support to managed relational DB
• Remove py27 tox tests.
• Update AKS cluster creation service principle handling, which was broken
• Update glibc version to 2.31.
• Propagate Azure credentials to AKS when running as a service principal. This is consistent with propagating VM credentials in azure_credentials.py.
  • This can be disabled by setting --bootstrap_azure_service_principal=False.
• Registered AWS VPC quota failures as quota failures.
• Do not try to use the GCP service account that is running PKB inside a GKE cluster if it obviously belongs to the wrong project.
• Upgrade psutil version to 5.6.6 from 3.0.0.
• Change default ior script to use 1GB sizes rather than 4GB.
• Remove nfs_gce_ip_range from filestore options.
• Upgrade YCSB to version 0.17.0.
• Do not pass a default --boot-disk-size or --boot-disk-type to GCE.
  • This allows GCE to set the preferred options for both.
  • Fixes an issue where CentOS and RHEL images grew beyond the default 10GB (to 20GB).
  • The current defaults leave other Linux OSes at 10GB and Windows OSes at 50GB.
• Explicitly setting Netperf to Python 3.
• Update PKB cloud datastore ycsb benchmark:
  • change private_keyfile_dir from a constant value to a FLAG
  • delete all db entries as part of Cleanup
• Remove 'default' keyword from AWS and Azure boot_disk_size
• Check ycsb proportion explicitly against none.
• Fix issue in calculating Geometric means.
• Add support to PKB cloud datastore ycsb benchmark to use GCS files for datastore keyfiles.
• Add SSH keys to dpb EMR clusters for debuggability.
  • Security groups of the clusters will have to be manually edited to allow SSH.
• Cast sample values to floats, ensuring they conform to the specification
• Improve database deletion performance of Cloud datastore YCSB benchmark.
• After installing the python package tries to set default version of python if not set.
• Add check in Prepare stage of PKB cloud datastore ycsb benchmark to ensure db is empty before running.
• Categorize k8s cluster creation errors better.
• Improve database deletion performance of Cloud datastore YCSB benchmark.
• Change deletion batch size of Cloud datastore YCSB benchmark.