[Archive-crawler-cvs] [internetarchive/heritrix3] 1644d0: Support * and $ wildcards in robots.txt

[Alex O. <no...@gi...>]

  Branch: refs/heads/robotstxt-wildcard
  Home:   https://github.com/internetarchive/heritrix3
  Commit: 1644d054a1cba08d6e0f5e5b7e04fd1db53d6f8f
      https://github.com/internetarchive/heritrix3/commit/1644d054a1cba08d6e0f5e5b7e04fd1db53d6f8f
  Author: Alex Osborne <aos...@nl...>
  Date:   2025-06-07 (Sat, 07 Jun 2025)

  Changed paths:
    M README.md
    M docs/configuring-jobs.rst
    M modules/src/main/java/org/archive/modules/net/RobotsDirectives.java
    M modules/src/test/java/org/archive/modules/net/RobotstxtTest.java

  Log Message:
  -----------
  Support * and $ wildcards in robots.txt



To unsubscribe from these emails, change your notification settings at https://github.com/internetarchive/heritrix3/settings/notifications