pmacct Files

[NOTE WELL: Please download the pmacct-1.7.7.tar.gz asset; DO NOT download any of the assets called "Source code" in either tar.gz or zip format]

• BGP, BMP, Streaming Telemetry daemons: introduced parallelization of dump events via a configurable amount of workers where the unit of parallelization is the exporter (BGP, BMP, telemetry exporter), ie. in a scenario where there are 4 workers and 4 exporters each worker is assigned one exporter data to dump.
• pmtelemetryd: added support for draft-ietf-netconf-udp-notif: a UDP-based notification mechanism to collect data from networking devices. A shim header is proposed to facilitate the data streaming directly from the publishing process on network processor of line cards to receivers. The objective is a lightweight approach to enable higher frequency and less performance impact on publisher and receiver process compared to already established notification mechanisms. Many thanks to Alex Huang Feng ( @ahuangfeng ) and the whole Unyte team.
• BGP, BMP, Streaming Telemetry daemons: now correctly honouring the supplied Kafka partition key for BGP, BMP and Telemetry msg logs and dump events.
• BGP, BMP daemons: a new "rd_origin" field is added to output log/ dump to specify the source of Route Distinguisher information (ie. flow vs BGP vs BMP).
• pre_tag_map: added ability to tag new NetFlow/IPFIX and sFlow sample_type types: "flow-ipv4", "flow-ipv6", "flow-mpls-ipv4" and "flow-mpls-ipv6". Also added a new "is_bi_flow" true/false key to tag (or exclude) NSEL bidirectional flows. Added as well a new "is_multicast" true/false config key to tag (or exclude) IPv4/IPv6 multicast destinations.
• maps_index: enables indexing of maps to increase lookup speeds on large maps and/or sustained lookup rates. The feature has been remplemented using stream-lined structures from libcdada. This is a major work that helps preventing the unpredictable behaviours caused by the homegrown map indexing mechanism. Many thanks to Marc Sune ( @msune ).
• maps_index: support for indexing src_net and dst_net keywords has been added.
• Added <daemon_name>_ipv6_only config directives to optionally enable the IPV6_V6ONLY socket option. Also changed the wrong setsockopt() IPV6_BINDV6ONLY id to IPV6_V6ONLY.
• Added log function to libserdes to debug transactions with the Schema Registry when kafka_avro_schema_registry is set.
• nDPI: newer versions of the library (ie. >= 3.5) bring changes to the API. pmacct is now aligned to compile against these.
• pmacctd: added pcap_arista_trailer_offset config directive since Arista has changed the structure of the trailer format in recent releases of EOS. Thanks to Jeremiah Millay ( @floatingstatic ) for his patch.
• More improvements carried out on the Continuous Integration (CI) side by migrating from Travis CI to GitHub Actions. Huge thanks to Marc Sune ( @msune ) to make all of this possible.
• More improvements also carried out in the space of the Docker images being created: optimized image size and a better layered pipeline. Thanks to Marc Sune ( @msune ) and Daniel Caballero ( @dcaba ) to make all of this possible.
• libcdada shipped with pmacct was upgraded to version 0.3.5. Many thanks Marc Sune ( @msune ) for his work with libcdada. ! build system: several improvements carried out in this area, ie. improved MySQL checks, introduced pcap-config tool for libpcap, compiling on BSD/old compilers, etc. Monumental thanks to Marc Sune ( @msune ) for his continued help. ! fix, nfacctd: improved euristics to support the case of flows with both IPv4 and IPv6 source / destination addresses (either or populated). Also improved euristics to distinguish event data vs traffic data in NetFlow v9/IPFIX from Cisco 9300/9500, ASA firewalls and Cisco 4500X. ! fix, nfacctd: improved support for initiatorOctets (IE [#231]) and responderOctets (IE [#232]). Thanks to Esben Laursen ( @hyberdk ) for reporting the issue. ! fix, nfacctd: in NF_mpls_vpn_id_handler() double ntohl() calls were applied for the case of 'vrfid'-encoded mpls_vpn_rd field. ! fix, sfacctd: wrong ethertype set for VLAN-tagged, MPLS-labelled IPv6 traffic. Impacting BGP resolution among others. Thanks to Jeremiah Millay ( @floatingstatic ) for his help resolving the problem. ! fix, BGP, BMP daemons: parsing improvements: added a check for BGP Open message and BGP Open Options lengths. Strengthened parsing of Peer Up, Route Monitoring and Peer Down v4 messages. ! fix, BGP, BMP daemon: when using Avro encoding and Avro Schema Registry, attempt to reconnect if serdes schemas are voided. Also now checking for serdes schema definitions before doing a serdes_schema_serialize_avro() to avoid triggering a SEGV. Finally improved serdes logging. ! fix, BGP, Streaming Telemetry daemons: in daemon logs, summary counters for amount of tables / entries dumped were wrong. ! fix, BGP daemon: distinguish among null and zero value AIGP and Prefix SID attributes. Same applies for Local Preference and MED attributes. ! fix, BMP daemon: resolved a memory leak in bgp_peers_free(). Thanks to Pether Pothier ( @pothier-peter ) for his patch. Also resolved a leak caused by an invalid BGP message contained in a BMP Route Message v4. ! fix, BMP daemon: correctly setting peer_ip and peer_tcp_port JSON fields for Term messages. Also the correct bmp_router value when bmp_daemon_parse_proxy_header feature is enabled. ! fix, BMP daemon: several encoding issues when using Apache Avro ie. u_int64_t now correctly encoded with avro_value_set_long(), certain u_int32_t fields switched to avro_value_set_long() due to lack of unsignedness in Avro encoding, improved various aspectes of Avro-JSON format output, etc. ! fix, pmtelemetryd: wrong parsing of pm_tfind() output was leading to mistaken data attribution of UDP-based peers (always first peer to connect was being picked). ! fix, pmtelemetryd: when set, the pidfile config directive was not being correctly honoured. ! fix, RPKI: the RTR PDU element for maxLength is uint8, therefore it might have been possible to transmit incorrect RTR data. Thanks to Job Snijders ( @job ) for his patch. ! fix, SQL plugins: amended the text composition of SQL queries that are involving latitude and longitude keys. ! fix, MySQL plugin: check for 'unix:' prefix string only when a sql_host configuration directive is specified. ! fix, nfprobe: modernized Application Information export. Until the previous release pmacct was adhering to aging NBAR model whereas now NBAR2 has been implemented. Thanks to Rob Cowart ( @robcowart ) for helping out resolving this issue. ! fix, tee plugin: restored usefulness of tee_source_ip which was broken in 1.7.6. Thanks to Jeremiah Millay ( @floatingstatic ) for reporting the issue. ! fix, maps_index: indexing of mpls_pw_id was broken. Also now, when the feature is enabled, actual data is being referenced in the index structure instead of creating a copy of it; thanks to Sander van Delden ( @SanderDelden ) for reporting the memory leak that was resulting from the copy. ! fix, kafka_common.c: solved memory leak in p_kafka_set_topic() when Kafka session was getting in down state. Many thanks to Peter Pothier ( @pothier-peter ) for nailing the issue. ! fix, net_aggr.[ch]: when a networks_file is specified in the config, gracefully handle max memory structure depth; added also de-duplication of entries. ! fix, pmacct-defines.h: if PCAP_NETMASK_UNKNOWN is not defined, ie. in libpcap < 1.1.0, let's define it. ! fix, SO_REUSEPORT feature was being restricted to Linux only in previous releases: now it has been unlocked to all other OS that do support the feature. ! fix, split SO_REUSEPORT and SO_REUSEADDR setsockopt() calls. Thanks to @eduarrrd for reporting and resolving the issue. ! fix, several code warnings catched gcc9 and clang.
• Obsoleted sql_history_since_epoch, pre_tag_map_entries and refresh_maps configuration directives.