[Archive-crawler-cvs] [internetarchive/heritrix3] f16db3: Support * and $ wildcards in robots.txt

[Alex O. <no...@gi...>]

  Branch: refs/heads/robotstxt-wildcard
  Home:   https://github.com/internetarchive/heritrix3
  Commit: f16db3997b9af6b6ff5292aa2763ee1280ea7b49
      https://github.com/internetarchive/heritrix3/commit/f16db3997b9af6b6ff5292aa2763ee1280ea7b49
  Author: Alex Osborne <aos...@nl...>
  Date:   2025-06-06 (Fri, 06 Jun 2025)

  Changed paths:
    M README.md
    M docs/configuring-jobs.rst
    M modules/src/main/java/org/archive/modules/net/RobotsDirectives.java
    M modules/src/test/java/org/archive/modules/net/RobotstxtTest.java

  Log Message:
  -----------
  Support * and $ wildcards in robots.txt



To unsubscribe from these emails, change your notification settings at https://github.com/internetarchive/heritrix3/settings/notifications