v3.3.0

This release introduces major enhancements to deployment infrastructure, data leak monitoring capabilities, and SSL certificate tracking. The update focuses on improving operational flexibility with a modern modular deployment system, adding regex pattern matching for keywords, migrating to the actively maintained SearxNG project, implementing comprehensive SSL certificate expiration monitoring, and introducing a local CertStream server for improved certificate transparency monitoring.

⚠️ Important - Breaking Changes

This is a major infrastructure update that requires manual configuration changes:

• ✅ New CertStream service must be added to docker-compose.yml
• ✅ SearxNG migration: Service renamed from searx to searxng
• ✅ Environment variables: Three variables must be updated in .env
• ✅ Configuration file: New certstream-config.yaml required
• ✅ Database migrations: New fields for regex keywords and SSL tracking

Update Procedure

For existing deployments:

Important: This release includes significant infrastructure changes. Please follow these steps carefully.

Step 1: Backup your data

Before proceeding, ensure you have a complete backup of your database and configuration files.

Step 2: Pull the latest version

:::bash
docker compose pull

Step 3: Stop running containers

:::bash
docker compose down

Step 4: Update docker-compose.yml

This release introduces a new CertStream service for local certificate transparency monitoring. Update your docker-compose.yml file with the following changes:

1. Add the CertStream service (add this section before the searxng service): ```yaml certstream: container_name: certstream image: 0rickyy0/certstream-server-go:latest restart: always networks: default: ipv4_address: 10.10.10.7 volumes:

   • ./certstream-config.yaml:/app/config.yaml:ro ports:
   • "8080:8080" healthcheck: test: ["CMD", "curl", "-f", "http://localhost:8080/"] interval: 10s timeout: 5s retries: 5 start_period: 10s environment:
   • TZ=${TZ} ```

2. Rename the Searx service from searx to searxng:

3. Change container_name: searx to container_name: searxng
4. Change hostname: searx to hostname: searxng
5. Update image from searx/searx:1.1.0-69-75b859d2 to searxng/searxng:latest
6. Update volume mount from ./Searx/searx:/etc/searx:rw to ./Searx/searx:/etc/searxng:rw

7. Remove the command: ${SEARX_COMMAND:-} line (no longer needed)

8. Update the Watcher service dependencies: yaml depends_on: db_watcher: condition: service_healthy searxng: condition: service_started certstream: condition: service_healthy

9. Add health check to db_watcher (if not already present): yaml healthcheck: test: mysqladmin --user=${DB_USER} --password=${DB_PASSWORD} status interval: 2s timeout: 1s retries: 10 start_period: 30s

10. Update the Watcher service environment to bypass proxy for internal services: yaml environment: no_proxy: "10.10.10.3,10.10.10.5,10.10.10.6,10.10.10.7,certstream"

Step 5: Update your .env file

The following environment variables must be updated:

1. SearxNG URL (Required): env DATA_LEAK_SEARX_URL=http://searxng:8080/

2. CertStream URL (Required - New variable): env CERT_STREAM_URL=ws://certstream:8080

3. NO_PROXY configuration (Required for proper internal routing): env NO_PROXY=certstream,10.10.10.7,localhost,127.0.0.1,10.10.10.3,10.10.10.5,10.10.10.6

4. Optional updates:

5. Review and update SEARX_HOSTNAME if needed (default: searxng:8080)
6. You can remove SEARX_COMMAND as it's no longer used

Step 6: Add CertStream configuration file

Create or update the certstream-config.yaml file in your project root directory:

:::yaml
# Configuration for certstream-server-go
# This file is mounted read-only in the Docker container

webserver:
  # Listen on all interfaces for Docker networking
  listen_addr: "0.0.0.0"
  listen_port: 8080

  # WebSocket endpoints
  lite_url: "/"                    # Recommended endpoint for Watcher
  full_url: "/full-stream"         # Full stream with all certificate data
  domains_only_url: "/domains-only" # Only domain names

  # Enable real IP detection for reverse proxy scenarios
  real_ip: true

# Prometheus metrics for monitoring
prometheus:
  enabled: true
  listen_addr: "0.0.0.0"
  listen_port: 9090
  metrics_url: "/metrics"

This configuration file is already included in the repository. If you've customized it, ensure your changes are preserved.

Step 7: Apply database migrations

:::bash
docker compose run watcher bash
python manage.py migrate

Step 8: Rebuild and restart containers

:::bash
docker compose up -d

Step 9: Verify the deployment

Check that all services are running correctly:

:::bash
docker compose ps

For new modular deployment (recommended for production):

The new deployment system provides enhanced reliability and maintainability. To migrate:

1. Navigate to the new deployment directory: bash cd deployment

2. Initialize the environment: bash make init

3. Configure your environment:

4. Copy .env.example to .env and customize
5. Update domain and network settings

6. Configure database credentials

7. Deploy the stack: bash make deploy

For detailed migration instructions, see the deployment README.

What's Changed

Deployment Infrastructure Overhaul

Modern Modular Docker Deployment System

A complete redesign of the deployment architecture provides improved maintainability, reliability, and operational flexibility:

• Modular Compose Architecture
• compose_apps.yaml: Application services (Watcher, SearxNG, CertStream)
• compose_databases.yaml: Database services with optimized configurations
• compose_reverse_proxy.yaml: Traefik reverse proxy with TLS termination

• Main docker-compose.yml orchestrates all services with health checks

• Automated Deployment Scripts (deployment/scripts/)

• init.sh: Complete environment initialization and validation
• deploy.sh: One-command full stack deployment
• migrate.sh: Database migration automation
• backup-db.sh: Automated database backup utility
• populate-db.sh: Sample data population
• create-superuser.sh: Interactive superuser creation
• check-network.sh: Docker network validation and creation
• replace-tls.sh: Automatic TLS configuration updates

• openssl-certificates-generator.sh: TLS certificate generation

• Makefile Integration

• Simple commands: make up, make down, make deploy, make migrate
• Automatic environment validation before operations

• Integrated backup and initialization workflows

• Enhanced Configuration Management

• Centralized .env file with comprehensive examples
• Template-based configuration with validation
• Automatic TLS certificate generation

• Network isolation with custom subnet management

• Traefik Reverse Proxy

• Automatic TLS termination with custom certificates
• Dynamic routing configuration
• Access logging and request tracing
• Dashboard for monitoring (optional)

---

Data Leak Module Enhancements

Regex Pattern Matching for Keywords

Advanced keyword matching capabilities significantly improve data leak detection:

• Regex Support
• New is_regex boolean field in Keyword model
• Backend validation of regex patterns before saving
• Compilation error handling with user-friendly messages

• Performance optimization with compiled pattern caching

• Frontend UI Improvements

• Checkbox to enable/disable regex mode per keyword

• Visual badge indicators:

  • "RegEx" badge for regex patterns (blue)
  • "Exact" badge for literal matches (green)

• Use Cases

• Email pattern matching: [a-zA-Z0-9._%+-]+@company\.com
• Domain variations: (company|brand|product)\.(com|net|org)
• Complex string patterns: API[_-]?KEY[:\s]+[A-Za-z0-9]+

---

Search Engine Infrastructure

Migration to SearxNG

Transition from deprecated Searx to actively maintained SearxNG ensures long-term reliability:

• Service Update
• Container name: searx to searxng
• Official SearxNG image with latest security patches

• Optimized configuration for data leak detection

• Configuration Improvements

• Simplified environment variables
• JSON format enabled for API compatibility
• Optimized for containerized environments
• Enhanced engine configuration (GitHub, GitLab, Bitbucket, NPM, StackOverflow)

• Disabled rate limiting for internal use

• Settings Synchronization

• Unified configuration between settings.py and .env
• Consistent URL handling across all modules

---

SSL Certificate Monitoring

Comprehensive SSL Expiration Tracking

New infrastructure for monitoring SSL certificate expiration across all domains:

• New SSL Fields
• LegitimateDomain.ssl_expiry: Track certificate expiration dates
• Site.ssl_expiry: Monitor site certificate validity
• Site.domain_created_at: Track domain registration dates

• Alert.new_ssl_expiry / Alert.old_ssl_expiry: Track certificate changes

• SSLCertificateChecker Utility

• Automatic certificate fetching and parsing
• Expiration date extraction and validation
• Support for various certificate formats
• Domain name cleaning and normalization
• Issuer information extraction

• Subject Alternative Names (SAN) parsing

• Automated Monitoring

• Periodic certificate checks
• Expiration alerts and notifications
• Integration with existing alert system
• Historical tracking of certificate changes

---

CertStream Infrastructure

Enhanced Certificate Transparency Monitoring

This release introduces a local CertStream server deployment, replacing the reliance on external CertStream services. This architectural change provides better reliability, control, and performance for certificate transparency monitoring.

• Local CertStream Server (New)
• Self-hosted CertStream server using 0rickyy0/certstream-server-go:latest
• Local WebSocket endpoint: ws://certstream:8080
• Eliminates dependency on external services (e.g., wss://certstream.calidog.io)
• Improved reliability and reduced latency
• Configurable via certstream-config.yaml
• Health checks for service reliability

• Runs on dedicated network IP: 10.10.10.7

• New CertStreamClient Class (dns_finder/certstream_client.py)

• WebSocket client with automatic reconnection
• Enterprise proxy support with authentication
• Automatic proxy detection and bypass for internal URLs
• Periodic ping mechanism to maintain connections

• Smart handling of connection failures with exponential backoff

• Proxy Intelligence

• Automatic detection of internal networks
• Smart proxy bypass for Docker networking (NO_PROXY configuration)
• Support for HTTP_PROXY, HTTPS_PROXY environment variables

• Seamless integration with corporate proxy environments

• Configuration File

• New certstream-config.yaml for service configuration
• WebSocket endpoint customization
• Prometheus metrics integration

• Flexible endpoint configuration (lite, full, domains-only)

• Docker Integration

• Updated to use 0rickyy0/certstream-server-go:latest
• Health checks for service reliability
• Dedicated network configuration with static IP
• Volume mounting for configuration management
• Environment variable support for timezone and proxy settings

New Contributors

• @TheoBhang made their first contribution in https://github.com/thalesgroup-cert/Watcher/pull/266)

Full Changelog: https://github.com/thalesgroup-cert/Watcher/compare/v3.2.2...v3.3.0

What's Changed

• Release v3.3.0 – Modular deployment overhaul, local CertStream server, SSL monitoring, and SearxNG migration by @ygalnezri in https://github.com/thalesgroup-cert/Watcher/pull/267

Full Changelog: https://github.com/thalesgroup-cert/Watcher/compare/v3.2.2...v3.3.0