This is the release notes of v0.9.0. See here for the complete list of solved issues and merged PRs.

This release note only covers the difference from v0.9.0rc3; for all highlights and changes, please refer to the release notes of the pre-releases:

alpha1 alpha2 beta1 beta2 rc1 rc2 rc3

Changes that break compatibility

From v0.9 on, Python 3.6 is dropped support.

Highlights

• Performance is fully optimized in this version, welcome to give your feedback.

New Features

• Oscar
• Stop importing main module when starting Mars local cluster (#3113)
• Tensor
• Integrate special error functions (#3062)
• Integrate part of scipy elliptic functions and integrals (#3112)
• DataFrame
• Support sort=True for Groupby (#3063, thanks @sak2002!)

Enhancements

• Dump remote tracebacks to make local ones more friendly (#3030)
• Optimize import speed for Mars package (#3035)
• [Ray] Implement ray task executor progress (#3065)
• Shuffle both sides at the same time for md.merge (#3066)
• Refine ThreadedServiceContext.get_chunks_meta usage (#3067)
• Do not aggressively choose tree method in tile of groupby for distributed setting (#3070)
• Disable bloom filter in merge for now (#3071)
• [Ray] Implements get_chunks_result for Ray execution context (#3072)
• Use tell when remove mapper data after execution (#3073)
• Assign reducer ops in task assigner to make them more balanced across cluster (#3075)
• [Ray] Destroy Ray executor when the task finish (#3074)
• Combine tree and shuffle methods in DataFrameGroupBy.agg tile (#3077)
• [Ray] Implements get_chunks_meta for Ray execution context (#3076)
• Use OS-designated ports instead of random ports to create sub pools (#3087)
• Call immutable web API only once when previous call blocks (#3088)
• Unify DataFrameGroupByAgg's tile logic for auto method (#3094)
• [Ray] Support basic subtask retry and lineage reconstruction (#3097)
• Simplify argument passing in actor batch calls (#3100)
• [Ray] Implements get_total_n_cpu for Ray execution context (#3104)
• Optimize performance of transfer (#3105)
• Add n_reducers and reducer_ordinal to shuffle operands (#3107)
• [Ray] Implement cancel method on Ray task executor (#3093)
• [Ray] Create RayTaskState actor as needed by default (#3114)
• [Ray] Implement gc for ray task executor context (#3116)
• Optimize serializable memory (#3126)

Bug fixes

• Patch pandas to make pickle compatible between 1.2 and 1.3 (#3050)
• Fix errors when deleting mapper data (#3064)
• Fix chunk index error in auto_merge_chunks (#3068)
• Fix recursive_tile that it may cause duplicated tile for one tileable (#3069)
• [Ray] Fix ray worker failover (#3115)
• [Ray] Fix pandas schema parsing when reading Ray dataset (#3117)
• [Ray] fix auto scale-in hang (#3125)
• [Metric] Fix prometheus metric backend (#3127)
• Fix mt.{cumsum, cumprod} when the first chunk is empty (#3136)

Tests

• Check initialization of serializables on CI (#3013)
• [Ray] Optimize Ray CI execution time and stability (#3121)
• Update pytest imports for test_special.py (#3131)
• [Ray] Fix flaky test test_optional_supervisor_node (#3135)

Others

• Build web code before CIBW when deploying to PyPI (#3016)