[Archive-crawler-cvs] [internetarchive/heritrix3] c7b7ee: Support * and $ wildcards in robots.txt

[Alex O. <no...@gi...>]

  Branch: refs/heads/robotstxt-wildcard
  Home:   https://github.com/internetarchive/heritrix3
  Commit: c7b7ee18420de21c5deee15a7216e035b0d65e59
      https://github.com/internetarchive/heritrix3/commit/c7b7ee18420de21c5deee15a7216e035b0d65e59
  Author: Alex Osborne <aos...@nl...>
  Date:   2025-06-07 (Sat, 07 Jun 2025)

  Changed paths:
    M CHANGELOG.md
    M README.md
    M docs/configuring-jobs.rst
    M modules/src/main/java/org/archive/modules/net/RobotsDirectives.java
    M modules/src/test/java/org/archive/modules/net/RobotstxtTest.java

  Log Message:
  -----------
  Support * and $ wildcards in robots.txt



To unsubscribe from these emails, change your notification settings at https://github.com/internetarchive/heritrix3/settings/notifications