We'd like to thank Aaron J. Scantlin (@scantlina), Aashish Sharma (@initconf), AbdolRashid Forghani (@soorooghadim), Anthony Alayo (@anthonyalayo), Bauti Peirone (@bautipeirone), @Breppe, @cccs-graeme, @cccs-will, Connor (@agent-connor), Craig Leres (@leres), @dagecko, Doğukan Çağatay (@dogukancagatay), Fatema Bannat Wala (@fatemabw), François De Keersmaeker (@fdekeers), Jan Grashöfer (@J-Gras), Klemens Nanni (@klemensn), Maor Hamami (@mamaorha), Martin Camara (@martincmr), Michael Peters (@MP-Corelight), Mohan Dhawan (@Mohan-Dhawan), Octave Charrin (@OctaveCharrin), Peter Cullen (@pbcullen), @RageAgainstTheOrganic, @siavashta, Steve Smoot (@stevesmoot), Stefan Götz (@stefangotz), @timo-mue and Zach Robinette (@zrobinette12) for their contributions to this release.

Breaking Changes

• The minimum version requirement for ZeroMQ is now 4.3.0. This version has been released at the end of 2018 - a good 7 years ago.

• Enums are now nominally typed. Zeek will produce warnings of the following form if you have scripts where enum types are mixed.

  Remove in v9.1. Mixing incompatible enum types color and city will become an error.

With version 9.1, these warnings will become errors. Previously, all enum types were compatible with each other, resulting in quirks and making certain optimizations more difficult than they should be.

• The BifEnum namespace was moved to be part of the zeek namespace for consistency with the other namespaces generated by bifcl (BifType, BifConst, etc).

• The MetricsAddress option in zeekctl.cfg default was changed from 0.0.0.0 to 127.0.0.1. If you've been using a remote Prometheus server for metrics scraping or run Zeek in a container environment, you'll now need to explicitly set MetricsAddress to 0.0.0.0 (or the appropriate listen address) in zeekctl.cfg.

• The packet captures in the btest Traces library have standardized on .pcap and .pcapng file suffixes. This may cause breakage in plugins that were using the old .trace suffix.

• The public vlan and inner_vlan members within C++ Packet has changed from uint32_t to an optional VlanTag struct. The corresponding dei and pcp members have been removed and merged into the VlanTag struct. This does not change scripts. In order to access the ID as before, check the optional first:

  if ( packet.vlan ) my_vlan = packet.vlan->id;

This allows Zeek to properly handle VLAN IDs of 0, which before were handled as if there was no VLAN.

• Reverted the change that went into Zeek 8.1 where ASCII control characters < 32 were encoded as \u00XX in JSON strings to the prior \xXX style. The escaping style will become configurable to allow for reliably representing the contained bytes in strings with Zeek 9.0.

See the discussion at https://github.com/zeek/zeek/discussions/5240 for more background and feel free to chime in with feedback.

• FreeBSD is no longer an officially-supported platform. We will make our best efforts to provide fixes to reported issues, like any other platform, but will not longer do regular builds on our CI provider.

New Functionality

• Added a new attribute &publish_on_change for global tables and sets as an alternative to the Broker-specific and deprecated &backend and &broker_store synchronization attributes. The &publish_on_change attribute allows efficient publishing of selected table changes as remote events to a configurable cluster topic. Other cluster nodes by default apply received changes to their local table instances.

  Publish any new entry as Cluster::table_change_infos() event to /zeek/table/endpoints

  global endpoints: set[addr, addr] &write_expire=5min &publish_on_change=[ changes=set(TABLE_ELEMENT_NEW), ];

See the online documentation for a detailed description and further examples:

https://docs.zeek.org/en/master/reference/zeekscript/attributes.html#attr-&publish_on_change

• Well-known ports of built-in analyzers can now be configured via redef. For example:

  redef HTTP::ports += { 8181/tcp };

For protocol analyzers, the variable name is generally Analyzer_Module::ports, but in certain cases (SSL, QUIC, RDP, ...) the variable name includes the protocol name to distinguish different internal analyzers. SSL::ssl_ports and SSL::dtls_ports as a concrete example. Check the analyzer module's export section or online documentation. For packet analyzers, the analyzer name is included in the variable name for historical reasons. Example:

redef PacketAnalyzer::VXLAN::vxlan_ports += { 4711/udp };

• The dns.log now also contains information about DNS notifications (RFC 1996). The opcode is set to 4 and opcode_code name will be "notify".

• ZeroMQ cluster communication can now be encrypted using the CURVE mechanism built into ZeroMQ. This requires a pair of keys, one for the server role and one for the client role. The central XPUB/XSUB sockets and LOG pull sockets are configured as CURVE servers, all other sockets (the connecting Zeek nodes) as CURVE clients.

You can generate Z85-encoded keypair as follows with Zeek:

$ zeek -e 'print to_json(Cluster::Backend::ZeroMQ::generate_keypair())' | jq
{
  "public": "}}6zuTySXq(C(6dF#D3Kvt@A5D}LcFlOgVav5{%)",
  "secret": "h.<[iyv]O#cWdO%2yuLiEJQ}6tC)@HxN6Hp?![td"
}

When using ZeekControl, keys are automatically generated and configured when a multi-node cluster based on node IP addresses is detected. To force encryption to be always on, set the following ZeroMQ specific setting in zeekctl.cfg (default is "auto"):

cluster_backend_zeromq.use_curve_encryption = 1

More details and background is available in the policy/frameworks/cluster/backend/zeromq/main.zeek script documentation.

• The number of messages and bytes processed by the central ZeroMQ XPUB/XSUB cluster component is now exposed as metrics of the form:

  zeek_cluster_zeromq_proxy_{frontend,backend}{bytes,messages}{sent,received}

by the process running this component in a cluster. Note that Zeek uses multipart messages for events: Publishing a single event currently counts as four messages as we're simply exposing ZeroMQ's low-level zmq::proxy_steerable() statistics.

• The @load directive now supports loading shared object files. This can be leveraged to experiment with alternative package and plugin directory layouts where a __load__.zeek file in a directory loads the required shared object explicitly rather than the current "magic discovery" via ZEEK_PLUGIN_PATH and __zeek_plugin__ files.

Files are expected to end with .so regardless of the platform and need to be explicitly loaded with the suffix provided:

@load ./plugin.so

Such shared objects should instantiate a single zeek::plugin::Plugin instance at load time. The zeek::plugin::Plugin() will register it accordingly.

• bifcl now accepts missing return types in functions signatures, creating void functions rather than complaining. The C++ signature will continue to have a ValPtr return type and the implementation should return a Val::nil, but the Zeek script function signature will have a void return type.

• The &on_change attribute for tables or sets now accepts variable argument functions.

• Added the modulo BiF, which behaves differently from Zeek's % operator for negative numbers.

• Added the connection_timing_out hook which, if break is used within, will prevent Zeek from timing out a particular connection. The motivation is to align a shunted connection's timeout with Zeek. Since Zeek does not see packets from a shunted connection, it needs special handling to keep the connection alive while shunted.

• The Zeek documentation has received a major refresh. A new tutorial walks newcomers through setup, basic Zeek invocations, the use of Zeek packages, ZeekControl for cluster maintenance, the basics of Zeek's logs, as well as scripting in Zeek's own language as well as JavaScript. A new reference section houses the introduction to common logs, the Zeek scripting language, Zeek's frameworks, and the scripting index.

• A new analyzer for IGMP was imported from an external project, originally written by François De Keersmaeker (@fdekeers). It adds new events for each of the types of IGMP messages, but does not add a new log.

• Windows support has been greatly improved, thanks to a raft of contributions from an engineering team at Microsoft. The btest suite now runs on CI builds. The great majority of btests pass, with a small number skipped for various documented reasons.

Changed Functionality

• Protocol parsers implemented in Spicy now suppress analyzer violations when a gap occurs and the parser doesn't implement error recovery. The most user visible aspect is that the analyzer's identifier is not removed from the service column in the conn.log.

• The to_subnet() built-in function's parsing of IPv6 subnets was changed to align with how IPv6 subnet literals are parsed in Zeek scripts. Previously, any subnet containing the ::ffff:0:0/96 IPv4-mapped prefix would produce an IPv4 subnet, even if provided prefix was shorter than the required 96 bits.

Internally, the behavior of the IPPrefix::ConvertString() API was changed to accommodate for this.

• The policy/frameworks/analyzer/detect-protocols script now respects analyzer violations to suppress Server_Found or Protocol_Found for connections which first had an analyzer confirmation, but later a analyzer violation happened. Previously, Zeek would report a HTTP server when a client issued a HTTP request to an SSH servers, or SSL when a client sent a TLS Client Hello to a HTTP server.

• The events ssl_extension_pre_shared_key_server_hello and ssl_extension_pre_shared_key_client_hello can now be used independently. Previously, both had to have a handler implemented for either of them to actually be invoked due to some logic confusion in the analyzer.

• Zeek new properly handles VLAN IDs of 0. Before, these were simply discarded.

• X.509 parsing now supports additional key types that were not handled previously. The certificate key length field will now be filled for additional ciphers, if supported by your OpenSSL version (like ML-DSA). Similarly, the key_type field will be filled out for additional ciphers, if supported by OpenSSL.

• The SMTP analyzer now forwards the first gap encountered during RFC 822 message analysis to the file object. This allows to determine if message analysis stopped due to a gap by inspecting the missing_bytes field during file_state_remove(), or in files.log.

The SMTP analyzer tears down the active file upon encountering a gap, so the reported missing bytes will the gap length only.

Deprecated Functionality

• The Broker::create_master(), Broker::create_clone() and Cluster::create_store() builtin functions have been deprecated. Consider using the new &publish_on_change attribute on tables for distributing table entries, explicit Cluster::publish() calls with remote events, or using the storage framework for persistence.

When ZeroMQ is selected as cluster backend, calling any of the above functions results in a fatal error as these are all Broker specific.