metpx-devel — Discussions about development

[Metpx-devel] NEW: Improve your efficiency with team scheduling

[Nik F. <ni...@ca...>]

Be more productive

( https://calendar.com )

Hey this is Nik Fowler, hope all is well! I am reaching out to efficiency-minded people to introduce the next evolution in efficient scheduling and calendar management.

****************
Unified Calendar
****************

No matter where your calendars may have originated from, Calendar brings them all into one place making it easy to see what is coming up in your busy life and allowing you to be as productive and efficient as possible.

******************
Calendar Analytics
******************

The analytic features will allow you to see where you are spending your time so you can adjust to become more efficient and time-aware than ever before.

*******************
Find a Time to Meet
*******************

Calendar's find a time feature allows you to compare your team's schedules to find a time that works for everyone and eliminates the time-wasting back and forth emails trying to find a time that works for everyone.

*********************
Scheduling Time Slots
*********************

With Calendar's customizable scheduling links, anyone can sign up on your or your team’s calendar and choose a time that works for everyone, making scheduling with external people a breeze.

Calendar is the answer to all your time-wasting scheduling headaches and is available at the best price because a world where everyone is efficient and organized is a better world for everyone!

*Visit Calendar.com ( https://calendar.com ) to start your 30-day free trial*

Be More Productive. Save Time. Focus.
Join Calendar for free today. ( https://calendar.com )

© 2021 Calendar.­com
su...@ca...

Unsubscribe ( http://url9539.calendar.com/wf/unsubscribe?upn=FuzxX0hNbr14ArM4UKZhmaPzla-2BDtouNroil513QmXNMWoEJ5EfijxovsQ2bAQJTZ3xFLgEGKgZ6P2hq5dPYwbyqSypDJkIQO1vPfCfVE48VlRit2ZNHAUoV8CJ4kSOIn9Rn64TqaPsZQXfk2WDC-2FjuuE-2Fchs02H0a-2B2gnnyoEPALnZvCIO78YFILGNUIIFFVzRlX5qLu3wiVmAPXDpvZBue6bcD8MRBbzi4PELj3Uuhc5GOY0Wt5G0-2F7y3VOyviGUML7RFKkXMEHgmHI-2FQmUjvEHXuLU-2FFj2E49T-2FKW5bO9xffEpn-2FvuHAn8uzF2Em6xlQsNAqTgJjx8iVSy41fF1ASdbSe7RtT7-2Flq3mpmQf-2BMAvG-2FHeQcw7l0y-2FeX-2FS-2BlLX-2FnVRDhDYIRafHPd9d-2B-2BHV2zCSIwoc9BGj7B9MOvw8eDmASfh5wRMKIwVNBO0XGS3yVt7wo0mh2i20LEluaJVezuidmnZQoFTSbp3KSHb1zGzEGo93KQXlOGMYtd92za61yqOYUmk0Vuc6sBZncqvmrhu8SqpBf3imwRI0RrTnGrOCueXGWnvc63TKAPnHIr9yMIGkqn5A1E04zz0vxWPqLg9rrsRNg57AaCnkmH4n6nTPcae3SdhrE5D3FfRMt0qG9I1PxYjVBCl7zlp6MRUjHZFbWy3pU4ANsUrvs-2BD-2F-2FXAI2MyL43vcRkexF3ZORZKlE942rRNdJtp-2BtSLDt3oDFt5cYMNgxC9VJPA7LWDqQZeIlGRoFT8GsLP7Ra04Qa64XY0p-2BI6PdCm0tKi9Py8o9pScnKBKtUlvRSIQoDRWOnbIp-2BMvkY-2F6yNdGsyBqI1kP4quNjw2oLQoD1rvD6PYvowMouwNWerHyZsNxiHgk3AZav2I1ACHdmWGbBht1ZPVt8iLNciJc6-2BZ9AAKtW0XOH9nMikSHxc3N2LNIgpf4-3D )

[Metpx-devel] sarrac - v2.19.06b3 vip support

[Peter S. <pe...@bs...>]

New release of the C package with a single bugfix, adding support
for the vip option in config files, for greater compatibility with python
(Issue #61 )

packages in the usual repo:

https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

or build from source:

  git clone https://github.com/MetPX/Sarrac/

[Metpx-devel] metpx-sarrac v2.1906b2 released

[Peter S. <pe...@bs...>]

major bugfixes:

  * bugfix: issue #59 - cpost doesn't notice mkdirs in watched trees.
  * bugfix: issue #58 - log file names made consistent with python
  * bugfix: issue #57 - cpost tries to post files with invalid names.
  * bugfix: issue #35 - stops publishing.. the #57 and #69 fixes fix this.
  * cosmetic: issue #47 reformatted source to respect kernel C style guide.

[Metpx-devel] metpx-sarrac v2.19.06b1 released.

[Peter S. <pe...@bs...>]

New version of the C package with major fixes in it.  Issue #35, was
causing sr_cpost to stop noticing files, which is a big problem when the
purpose of a program is to notice files.  It seems it would stop looking at
files when a directory tree was deleted, and then a new directory was
created, and it had the same inode.  This tended to happen when maintenance
operations happenned around 0Z.

We also noticed that sr_cpump wasn't processing half the messages it
consumed.  Fixed.

details:

  * bugfix:  issue #35 avoid crashes/hangs when directory trees are
removed. (M. saraga!)
  * bugfix:  issue #38 consumer was skipping half of the messages queued.
  * bugfix:  buffer overrun fix from Gregory Rappa.
  * bugfix:  issue #41, core dump
  * new:  issue #44 added checksum caching support. (from M. Saraga)
  * new:  issue #52 added log_reject to print log messages when rejecting
messages.

packages in the usual repo:

https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

or build from source:

  git clone https://github.com/MetPX/Sarrac/

[Metpx-devel] sarrac 2.19.05b1 released

[Peter S. <pe...@bs...>]

A new version of linux only C implementation has been released:

  * bugfix: issue#29 adding utf8 check to avoid posting corrupted
messages.  (msaraga!)
  * new:    issue#39 logging refactored to match python implementation
(msaraga)

The utf8 checks are needed to address problems encountered with HPC
mirroring use case.

installation guide:
https://github.com/MetPX/sarracenia/blob/master/doc/Install.rst

packages available for ubuntu from here:

    https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

source code:
    git clone https://github.com/MetPX/sarrac

[Metpx-devel] release 2.19.04b2

[Peter S. <pe...@bs...>]

A bug was discovered in checksum storage in extended attributes.
It seemed important enough to justify a quick release.
The rest is just work in progress.

details:
* bugfix: #158 there was a bug with extended attributes in checksums (yes,
another one.)
* new:    moved flow_test away from xpublic to allow continuous use
*         on an active pump without interfering with normal flows.
* info:   mostly refactoring for future work by Benoit.
*         Noureddine wants to run flow_test continuously somewhere
*         in addition to the travis-ci.com stuff.
*         there are some issues that crept in over the summer:
*         #182, #179, are opened to track progress on that.

Installation Guide:

https://github.com/MetPX/sarracenia/blob/master/doc/Install.rst

packages available for ubuntu from here:

    https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

python packages:

   https://pypi.org/project/metpx-sarracenia/

windows installer:

   http://hpfx.collab.science.gc.ca/~pas037/Sarracenia_Releases/

source code:
    git clone https://github.com/MetPX/sarracenia

[Metpx-devel] Sarracenia release v2.19.04b1

[Peter S. <pe...@bs...>]

There was a regression that prevented posting v03 messages with v2.19.03b6.
This is an emergency fix for that version.  It also adds a WMO_mesh_post
example.

details:

* new:    adding WMO_mesh_post example.
* bugfix: regression all posts in b6 were v02 (v03 inadvertantly disabled.)
* bugfix: print improvements.
* bugfix: there was an issue with extended attribute in checksums.


Installation Guide:

https://github.com/MetPX/sarracenia/blob/master/doc/Install.rst

packages available for ubuntu from here:

    https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

python packages:

   https://pypi.org/project/metpx-sarracenia/

windows installer:

   http://hpfx.collab.science.gc.ca/~pas037/Sarracenia_Releases/

source code:
    git clone https://github.com/MetPX/sarracenia

[Metpx-devel] sarracenia release v2.19.03b6

[Peter S. <pe...@bs...>]

incremental release, optional upgrade, more resistant to missing fields
from other implementations, and first release that uses alternative data
streams on windows to store file checksums.

detailed changes:

* new:    made code use instance variables instead of repeatedly parsing
*         elements of the message: topic_prefix->version, sum-> event.
*         more robust/easier to maintain.
* bugfix: updated WMO_mesh examples to use v03 and /var/www/html (as per
feedback.)
* new:    add post_override_del option to post_override plugin.
*         allows deletion of headers on post.
* bugfix: Issue #175 documentation needs to use amqps for dd.weather, not
amqp anymore.
* new:    wmo_mesh example now deletes a bunch of headers to shorten &
simplify messages.
* new:    switching http port in flow test to 8001.  8000 is too popular.
* new:    switching flow_test to use hpfx.collab.science.gc.ca instead of
dd.weather.gc.ca
* bugfix: Issue #168 have httpd unit test retrieve pick a file <= 2kb.  Big
files
*         were causing hangs in the flow test... this is one fix, but not
enough to close
*         the bug.
* new:    Issue #54 is finally closed.
* new:    Issue #54 added xattr_disable to turn off extended attributes
feature.
* new:    Issue #54 added version of extended attributes for NTFS.



Installation Guide:

https://github.com/MetPX/sarracenia/blob/master/doc/Install.rst

packages available for ubuntu from here:

    https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

python packages:

   https://pypi.org/project/metpx-sarracenia/

windows installer:

   http://hpfx.collab.science.gc.ca/~pas037/Sarracenia_Releases/

source code:
    git clone https://github.com/MetPX/sarracenia

[Metpx-devel] sarrac release 2.19.03b5

[Peter S. <pe...@bs...>]

An incremental new release of the linux-only C implementation of a subset
of Sarracenia functionality is now available.

Detailed Changes in this release:

  * bugfix: issue #14 strip works for cpost, as per python. (from Michael!)
  * bugfix: noticed expiry setting interpreted as 1000 times longer than it
should be.
  * new:    additional documentation, support for doxygen.
  * new:    added support for suppress_duplicates_basis option, as per
python. (from Michael!)

binary packages available from launchpad.net:

sudo add-apt-repository ppa:ssc-hpc-chp-spc/metpx
sudo apt-get update
sudo apt-get install sarrac

There are no pre-built binaries for other linux distributions.

(install librabbitmq > 0.8 )
git clone https://github.com/MetPX/Sarrac
cd Sarrac
make

[Metpx-devel] Sarracenia v2.19.03b5 release

[Peter S. <pe...@bs...>]

incremental bugfixes.
updated WMO ETCTS related configurations to use v03 message format.

details:

* bugfix: transparently accept setxattr failure for readonly files.
* bugfix: several bugs with roundtripping v03->v02 checksums.
* bugfix: changed documentation for rename (used what is changed by strip
option)
* bugfix: found issues with arbitrary-application checksum
* bugfix: mode/atime/mtim restoration was missing for inlined data.
* new:    upgrade wmo_mesh mqtt publishing plugin to work with v03

Installation Guide:

https://github.com/MetPX/sarracenia/blob/master/doc/Install.rst

packages available for ubuntu from here:

    https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

python packages:

   https://pypi.org/project/metpx-sarracenia/

windows installer:

   http://hpfx.collab.science.gc.ca/~pas037/Sarracenia_Releases/

source code:
    git clone https://github.com/MetPX/sarracenia

[Metpx-devel] metpx-sarracenia v2.19.03b3 and b4

[Peter S. <pe...@bs...>]

new versions of sarracenia, each one containing only a single correction
for v02 to v03 conversion.  For some reason, not testing properly yet in
dev, so only finding issue in after
deployment. all changes only affect v03 usage.

[Metpx-devel] Sarracenia 2.19.03b2

[Peter S. <pe...@bs...>]

What is new in this release:

* new:    issue #134 added on_data plugin entry point, to allow data
tranformation.
* new:    removed recompute_chksum setting. application now decides on its
own.
* new:    added post_on_start option so control whether sr_watch posts all
files on startup.

* bugfix: issue #172 add inlining of data when the header is missing in a
message being forwarded.
* dev:    issue #159 Contiguous Integration with travis-ci added, improved
testing.
*            testing of four different python versions now automated.
* new:    added msg_rawlog to allow clearer viewing of v02 and v03 messages.

Installation Guide:

https://github.com/MetPX/sarracenia/blob/master/doc/Install.rst

packages available for ubuntu from here:

    https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

python packages:

   https://pypi.org/project/metpx-sarracenia/

windows installer:

   http://hpfx.collab.science.gc.ca/~pas037/Sarracenia_Releases/

source code:
    git clone https://github.com/MetPX/sarracenia

for those upgrading:

https://github.com/MetPX/sarracenia/blob/master/doc/UPGRADING.rst

Those working with Ubuntu packages:

The package on Launchpad.net is now (since the last release) called
metpx-sarracenia (formerly called python3-metpx-sarracenia) so to upgrade
from previous versions, the easiest way is to remove the old package, then
install the new one. For those using other methods, the package name was
always metpx-sarracenia.

[Metpx-devel] new release v2.19.03b1

[Peter S. <pe...@bs...>]

What is new in this release:

incremental release with improvements in support for the experimental v03
format
to support work with the WMO.  Fixes for older versions of python3-amqp
library (present on older operating systems, such as ubuntu 16.04 and
14.04.) removed some error messages that were just leftovers from debugging
inadvertently added in the previous release.


Installation Guide:

https://github.com/MetPX/sarracenia/blob/master/doc/Install.rst

packages available for ubuntu from here:

    https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

python packages:

   https://pypi.org/project/metpx-sarracenia/

windows installer:

   http://hpfx.collab.science.gc.ca/~pas037/Sarracenia_Releases/

source code:
    git clone https://github.com/MetPX/sarracenia

for those upgrading:

https://github.com/MetPX/sarracenia/blob/master/doc/UPGRADING.rst

Those working with Ubuntu packages:

The package on Launchpad.net is now (since the last release) called
metpx-sarracenia (formerly called python3-metpx-sarracenia) so to upgrade
from previous versions, the easiest way is to remove the old package, then
install the new one. For those using other methods, the package name was
always metpx-sarracenia.

[Metpx-devel] new release v2.19.02b2

[Peter S. <pe...@bs...>]

what's new in this release:

Inremental release, nothing major. There was an issue for the last few
months where some messages were appearing on the screen instead of being
re-directed. That has been fixed. As the developer was in that code, log
rotation functionality was improved at the same time. Example:

logrotate_interval 1h
logrotate 5

Allows one to have logs rotated every hour, and keep five log files.
There is some work on a new prototype format (v03) that doesn't affect
operational use.

https://github.com/MetPX/sarracenia/blob/master/CHANGES.rst

**2.19.02b2**

*
* new:    implemented WMO expert team on computing and telecommunication
systems
*         (ETCTS, latest meeting: 2019/02) recommendations for changes to
v03 format.
* new:    ETCTS201902 v03 whole messages is a single JSON *object*
              (like a python *dictionary*)
* new:    issue #146 ETCTS201902 v03 timestamps now have a "T" in them.
* new:    issue #148 ETCTS201902 v03 *sum* header changed to *integrity*,
encoding changed from hex
*         to base64.
* new:    issue #147 ETCTS201902 v03 added **inline** support to include
file data in the announcements.
*         data is encoded in either utf-8 or base64.
* new:    issue #153 log rotation interval can now be set. Minutes if you
like.
* bugfix: issue #140 messages on console instead of log.
*


Installation Guide:

https://github.com/MetPX/sarracenia/blob/master/doc/Install.rst

packages available for ubuntu from here:

    https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

python packages:

   https://pypi.org/project/metpx-sarracenia/

NEW! windows installer:

   http://hpfx.collab.science.gc.ca/~pas037/Sarracenia_Releases/

source code:
    git clone https://github.com/MetPX/sarracenia

for those upgrading:

https://github.com/MetPX/sarracenia/blob/master/doc/UPGRADING.rst

Those working with Ubuntu packages:

The package on Launchpad.net is now (since the last release) called
metpx-sarracenia (formerly called python3-metpx-sarracenia) so to upgrade
from previous versions, the easiest way is to remove the old package, then
install the new one.
For those using other methods, the package name was always metpx-sarracenia.

[Metpx-devel] Sarracenia news, and releases to be announced here again.

[Peter S. <pe...@bs...>]

It's been virtually silent here for a few years.  We had a few specialized
clients, and were announcing in more particular ways, but now we will start
announcing releases of the new stack to this mailing list, as there may be
additional interest at this time.  As a general introduction:

There a major refactoring done during the last half of 2017, basically
completed by Jan. 2018, and releases in the past year or so after that
point have essentially been bug-fix releases.  This denoted by the change
in designation of releases from a (alpha), to b (beta) and we are now
approaching long term stable release, likely to happen at some point in
2019.

Configurations for versions prior to that point have incompatibilities with
>=2.18.01 (releases on or after January 2018.) so that the move to a
current version needs care. Once at a recent version, all upgrades should
be seamless. The only impact of upgrades should be getting bugfixes.


Things that have happenned in 2018:

  - the datamart http://dd.weather.gc.ca has moved from a pre-standard
prototype configuration, to a standard Sarracenia one with no particular
customizations.  It is now using only TLS secured connections for AMQP
announcements.

 - http://hpfx.collab.science.gc.ca is a second public facing data pump
available for customized or other selected data feeds. The installation
isn't complete yet, but certain specialized feeds are in use.  It's usage
should expand during 2019.

- A windows installer to substantially simplify installation is now
available (in 2019.)
There was many issues with supporting windows as part of the rollout of the
upgraded datamart. Hopefully, this is addressed by the new installer.

- We have begun work with the World Meteorological Organization on renewal
of the Global Telecommunications System (GTS) or WMO Information Service
(WIS) using pub/sub and AMQP technology.  This work is with prototype
feeds, not the same as what is in production use today.  We expect to
follow this work, and have a compliant configuration once the specification
stabilizes.  References to v03 or ETCTS are on this topic.


What's new in each version is always available here:

https://github.com/MetPX/sarracenia/blob/master/CHANGES.rst



For users on versions prior to v2.18:

While new installations can use the existing configuration, when upgrading
from an existing configuration, it may need adjustment.  This is documented
in the UPGRADING file, here:

https://github.com/MetPX/sarracenia/blob/master/doc/UPGRADING.rst

In general when moving between any two versions, one needs to study the
entries in that file between the old and new versions to determine the
changes to an existing configuration needed.  in 2018, there are very few
such entries, because the application has stabilised considerably.

[Metpx-devel] Pardon the dust. Move from sf.net to github.com in progress.

[Peter S. <pe...@bs...>]

Hello,

http://metpx.sf.net (aka. http://metpx.sourceforge.net)  has been the
website for the Meteorological Product Exchanger project since 2006.
For some time, sf.net has been getting progressively slower, and has
been declining in popularity.
Around February 13th, 2018, sourceforge.net became unusable.  It has
sporadically been that way since that time. In the first week of March
2018, the master git repositories for all MetPX projects were moved
github.com/MetPX.

Packages for currently supported ubuntu releases are built by automated recipes
on launchpad.net. These recipes have now been updated to pull from github
instead of souceforge, but there were some errors during that process.
The daily snapshot recipe wrote (in error) to the (supposedly stable)
MetPX PPA.
and the MetPX Dailly PPA has not yet been updated.

The MetPX Daily PPA should sort itself out tonight, and the stable PPA should
get sorted at the next stable release... likely next week.

We currently don't have an alternate location for downloadable packages, so
we will continue to upload for sourceforge for now.

The metpx.sf.net website continues to be updated (and it now points to
github for a variety of uses.) while we select an approach for replacing it.

[Metpx-devel] Sarracenia 2.18.01a5 now available

[Peter S. <pe...@bs...>]

Hello All,

Metpx-sarracenia release 2.18.01a5 is now available. The package is
available from all our usual distribution locations:


UBUNTU PPA:   https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

via pip:   https://pypi.python.org/pypi/MetPX-sarracenia

package files: https://sourceforge.net/projects/metpx/files/sarracenia

Changelog:    http://metpx.sourceforge.net/CHANGES.html
Upgrade Notes: https://sourceforge.net/p/metpx/sarracenia/ci/master/tree/doc/UPGRADING.rst

What's new:

retry logic was not working very well and has been substantially
improved.  When problems in connectivity happen, components will save
queues to local disk.  When conectivity is re-established, real-time
feeds will have priority, but whenever there is a gap in new products,
the code will look in the transfers that failed and try again to
transfer it.

exchange_suffix option is added to make most use cases easier to write.

[Metpx-devel] Sarracenia 2.17.12a1 now available.

[Peter S. <pe...@bs...>]

Hello All,


Metpx-sarracenia release 2.17.12a1 is now available. The package is
available from all our usual distribution locations:

https://sourceforge.net/projects/metpx/files/sarracenia
https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
https://pypi.python.org/pypi/MetPX-sarracenia

Changelog: http://metpx.sourceforge.net/CHANGES.html
Upgrade Notes: https://sourceforge.net/p/metpx/sarracenia/ci/master/tree/doc/UPGRADING.rst

The major focus of the last few weeks has been improving reliability.
Enhanced reliability should come from three main features:

  -- timeouts: all remote operations now have associated timeouts, and
then attempt to re-connect.
  -- pulse: brokers now have a separate queue to keep idle connections
alive transparently using a pulse message sent periodically.
  -- retry_queue: instead of attempting to send X times (where X is
set by "attempts"), the transfer is set aside to a .retry file.
     This automates use of the save/restore logic, so no analyst
intervention is needed. When there are no messages to send, the
subscriber or sender will pick a message from the retry queue and try
to send it. So the senders should be able to transparently recover
from even prolonged outages.  The old behaviour can be restored via
*retry off* setting in the configuration file.


There is one change in interpretation of configuration files compared
to previous versions:

When an sr_subscribe or an sr_sender has no accept/reject clauses,
then previous versions would send everything. The new version sends
nothing. To have it send everything, as before, append to the config
file:

accept_unmatch

Aside from the above, there are also several bugfixes made since 11a3
(SENDER=, and "Sends:" in particular.)

[Metpx-devel] sarracenia release 2.17.10a3 now available

[Peter S. <pe...@bs...>]

Hi all,

Metpx-sarracenia release 2.17.10a3 is now available. The package is
available from all our usual distribution locations:

https://sourceforge.net/projects/metpx/files/sarracenia
https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
https://pypi.python.org/pypi/MetPX-sarracenia

Changelog: http://metpx.sourceforge.net/CHANGES.html
Upgrade Notes: https://sourceforge.net/p/metpx/sarracenia/ci/master/tree/doc/UPGRADING.rst

Changelog: http://metpx.sourceforge.net/CHANGES.html
Upgrade Notes: https://sourceforge.net/p/metpx/sarracenia/ci/master/tree/doc/UPGRADING.rst

Release Notes:

This is the first major code drop including a re-factoring by Michel
to eliminate a lot of
code duplication and make all components behave more consistently. As
it was expected that
significant issues may arise from the re-factoring the earlier
releases ( 10a1 and 10a2 ) were
not announced on this list, so this release note covers all three
versions in October:


* C: directories posted during rename. not sure what effect is.
* documentation consolidated to sr_subscribe, much duplication gone.
* C: sr_cpost force_polling works properly now (using cache.)
* C: bugfix double free segfault on exit.
* C: added directory support to sr_post_rename
* C: libsrshim: added support for the truncate(2) system call.
* support file names with spaces in them.
* call on_file plugins when symbolic link created.
* sr_config    : environment variable substituted for option value
* sr_sender    : on_msg msg_2localfile now by default (so no longer
need to specify for every sender.)
* sr_subscribe : changing determination of source
(source_from_exchange or missing)
* rmdir support (python only.)
* add regexp option to strip.
* now support environment variables in config files with ${var}
* bugfix: misbehaved when file names have blanks in them.
* added -header option to sr_post.
* fix for bug #74 - error messages on shutdown of amqps connection.
* C: cpost setup/cleanup/declare/restart etc... some were broken, fixed.
* C: added sighandler to avoid cache corruption when terminating.
* C: add rename support to cpost (was only in libcshim and python before.)
* C: bugfix: C was inventing fields if not provided (mode=0, mtime="").
* C: added tx.select & tx.confirm (publish acknowledgements)
* C: FIXME: not yet: basic_ack (consumer acknowledges only after
successful processing, rather than on receipt.)
* C: integrated into flow_tests.
* cleanup/declare/setup actions (all programs): no exit, log with configname
* sr_subscribe/sr_sarra/sr_sender : do_task plugin (initialised to
proper module for now)
* sr_subscribe: headers' source and from_cluster forced when
source_from_exchange
* sr_subscribe: add substitution for ${DR} ${PDR} ${YYYYMMDD} ${SOURCE} ${HH}
* sr_subscribe  log ignore message when already in cache
* sr_subscribe: events option is consider to perform link and delete messages
* sr_subscribe: modified to be a base class instantiated from most programs
* sr_subscribe: integration of restore_queue, process report_daemons,
save/restore
* sr_subscribe: help module : treats sr_shovel,sr_winnow,sr_sarra cases
* sr_sender: for R and L messages skip offset/length setting in module
set_local()
* sr_shovel: caching optional default to False
* sr_config: some save,restore and cache defaults
* sr_config: inflight supports duration_from_str (for sr_watch/post)
* sr_config: duration_from_str  time suffix [sS] [mM] [hH] [dD] [wW]
where applicable
* sr_config: module configure cleans up extended options (proper reload)
* sr_config: option -headers to add,delete or reset user's  key,value
pair in message headers
* sr_ftp,sr_sftp: connect/reconnect resets cdir (current dir)
* sr_ftp,sr_sftp,sr_http: standardisation, http exception (no hang)
* sr_ftp,sr_sftp,sr_http: fix Eric's os.getcwd bug, add preventive
fp.flush and os.fsync
* msg_total.py: plugin skip total byte increment when no partstr in message
* sr_message: move support with oldname/newname (impact
watch,post,subscribe,sarra,sender to come)
* sr_message: srcpath turned to baseurl, set_notice(baseurl,relpath)
***impact all programs
* sr_message: trim_headers for user added headers key,value pair
***impact all programs
* sr_cache: module cache.check_msg ... process correctly message
without parts (sum L and R)
* sr_audit,sr speed up through class instantiation and direct broker connection
* sr_audit fix permissions for source and subscribe users
* sr_amqp,sr_pika: cleanup skip removal of exchanges xpublic,xreport,xwinnow*
* sr_util:  startup_args catches -help when only args given
* flow_test: several changes to make it more reliable.

[Metpx-devel] release 2.17.09a1 now available.

[Peter S. <pe...@bs...>]

Hi all,

Metpx-sarracenia release 2.17.09a1 is now available. The package is
available from all our usual distribution locations:

https://sourceforge.net/projects/metpx/files/sarracenia
https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
https://pypi.python.org/pypi/MetPX-sarracenia

Changelog: http://metpx.sourceforge.net/CHANGES.html
Upgrade Notes: https://sourceforge.net/p/metpx/sarracenia/ci/master/tree/doc/UPGRADING.rst

Release Notes:

* sr_subscribe now can post files for further processing or transfers
(like sr_sarra)
* all components now use a new suppress_duplicates (cache)
implementation, so they can all
  recognize and ignore duplicates.  cache is cleaned every heartbeat interval.
* sr_watch memory use substantially improved.
* fix (bug# 88) for sr_audit creating report queues with no consumers.
*   expire DEFAULT CHANGED:  7 days -> 5 minutes.  Avoiding pump
overloading turns out to be critical.
* new plugin msg_to_clusters, simplified replacement of inter-cluster
routing logic (which was removed in 08a1.)

[Metpx-devel] release of sarracenia v2.17.07a1

[Peter S. <pe...@bs...>]

Hi all,

Metpx-sarracenia release 2.17.07a1 is now available. The package is
available from all our usual distribution locations:

https://sourceforge.net/projects/metpx/files/sarracenia
https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
https://pypi.python.org/pypi/MetPX-sarracenia

Changelog: http://metpx.sourceforge.net/CHANGES.html

Release Notes:

 -- permissions have changed significantly.  Formerly umask was forced
    to 0, now that does not happen, so need to set up umask externally.

-- command line 'action' is now at the beginning of the list of arguments.
   like systemctl,   rather than at the end, as was standard with older
init  file standard.

-- users are now expected to be able to declare their own exchanges.
   expect warning messages about failures to declare exchanges when
   this version connects to pumps running older versions of the software.
   those messages are harmless. and should go away once the pumps
   are upgraded.

features:

--  hearbeat 600
    execute on_heartbeat routine every 600 seconds.
    to schedule periodic execution of arbitrary code.
    default is to write a message to the log.

-- cat filenames | sr_post -pipe
   post can read the list of files to post from standard input when
   given the pipe argument.

-- can now sr_poll http sites ( in addition to ftp, ftps, sftp)

[Metpx-devel] metpx-sarracenia 2.16.11a4 released

[Ebrahimpour, K. (SSC/SPC) <kho...@ca...>]

Hi all,

Metpx-sarracenia release 2.16.11a4 is now available. The package is available from all our usual distribution locations:

https://sourceforge.net/projects/metpx/files/sarracenia
https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
https://pypi.python.org/pypi/MetPX-sarracenia

--
Khosrow

[Metpx-devel] metpx-sarracenia 2.16.11a3 released

[Ebrahimpour, K. (SSC/SPC) <kho...@ca...>]

Hi all,

Metpx-sarracenia release 2.16.11a3 is now available. The package is available from all our usual distribution locations:

https://sourceforge.net/projects/metpx/files/sarracenia
https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
https://pypi.python.org/pypi/MetPX-sarracenia

--
Khosrow

[Metpx-devel] metpx-sarracenia 2.16.11a2 and metpx-sundew 0.16.11b1 released

[Ebrahimpour, K. (SSC/SPC) <kho...@ca...>]

Hi all,

Metpx-sarracenia release 2.16.11a2 is now available. The package is available from all our usual distribution locations:

https://sourceforge.net/projects/metpx/files/sarracenia
https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
https://pypi.python.org/pypi/MetPX-sarracenia

In addition, metpx-sundew version 0.16.11b1 has been released. It is also available at the following locations:

https://sourceforge.net/projects/metpx/files/sundew
https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

--
Khosrow

[Metpx-devel] metpx-sarracenia 2.16.11a1 and metpx-sundew 0.16.11 released

[Ebrahimpour, K. (SSC/SPC) <kho...@ca...>]

Hi all,

Metpx-sarracenia release 2.16.11a1 is now available. The package is available from all our usual distribution locations:

https://sourceforge.net/projects/metpx/files/sarracenia
https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx
https://pypi.python.org/pypi/MetPX-sarracenia

In addition, metpx-sundew version 0.16.11 has been released. It is also available at the following locations:

https://sourceforge.net/projects/metpx/files/sundew
https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

--
Khosrow