[Archive-crawler-cvs] [internetarchive/heritrix3] c7b7ee: Support * and $ wildcards in robots.txt

[Alex O. <no...@gi...>]

  Branch: refs/heads/master
  Home:   https://github.com/internetarchive/heritrix3
  Commit: c7b7ee18420de21c5deee15a7216e035b0d65e59
      https://github.com/internetarchive/heritrix3/commit/c7b7ee18420de21c5deee15a7216e035b0d65e59
  Author: Alex Osborne <aos...@nl...>
  Date:   2025-06-07 (Sat, 07 Jun 2025)

  Changed paths:
    M CHANGELOG.md
    M README.md
    M docs/configuring-jobs.rst
    M modules/src/main/java/org/archive/modules/net/RobotsDirectives.java
    M modules/src/test/java/org/archive/modules/net/RobotstxtTest.java

  Log Message:
  -----------
  Support * and $ wildcards in robots.txt


  Commit: 7e15d2fda1bcac53e6b0be125910c02dd8ecffba
      https://github.com/internetarchive/heritrix3/commit/7e15d2fda1bcac53e6b0be125910c02dd8ecffba
  Author: Alex Osborne <aos...@nl...>
  Date:   2025-06-09 (Mon, 09 Jun 2025)

  Changed paths:
    M CHANGELOG.md
    M README.md
    M docs/configuring-jobs.rst
    M modules/src/main/java/org/archive/modules/net/RobotsDirectives.java
    M modules/src/test/java/org/archive/modules/net/RobotstxtTest.java

  Log Message:
  -----------
  Merge pull request #656 from internetarchive/robotstxt-wildcard

Support * and $ wildcards in robots.txt


Compare: https://github.com/internetarchive/heritrix3/compare/497d79500575...7e15d2fda1bc

To unsubscribe from these emails, change your notification settings at https://github.com/internetarchive/heritrix3/settings/notifications