Audit and improve testproxy.py indicators with comprehensive enhancements

- Fixed critical bug in WAF indicator loading - added proper parsing function for Header:Value:WAF_Name format
- Organized WAF indicators file with comments, removed duplicates, and added modern providers (Google Cloud Armor, Fastly, Azure Front Door, AWS Shield)
- Expanded proxy indicators with extensive cloud service support (Cloudflare, AWS CloudFront, Azure, Fastly, Akamai, Google Cloud)
- Added comprehensive file format validation with error handling and UTF-8 support
- Improved error handling throughout the indicator loading process with detailed logging
- Standardized header formats using lowercase for consistent matching
- Enhanced detection of modern CDN and load balancer infrastructure
- Maintained backward compatibility with default indicator lists

Author: geeknik

proxy_indicators.txt

@@ -1,197 +1,140 @@
+# Common Proxy/Load Balancer Headers (Standard)
 X-Forwarded-For
 X-Real-IP
 Via
 X-Forwarded-Host
 X-Forwarded-Proto
-X-Load-Balancer
-Proxy-Connection
-X-Proxy-ID
-Forwarded
-X-Forwarded-Server
 X-Forwarded-Port
-X-Original-URL
-X-Rewrite-URL
-X-Proxy-Cache
-X-Cache
-X-Cache-Lookup
-X-Varnish
-X-Azure-Ref
-CF-RAY
-X-Amzn-Trace-Id
+Forwarded
 X-Client-IP
-X-Host
-X-Forwarded-By
 X-Originating-IP
-X-Backend-Server
-X-Served-By
-X-Timer
-Fastly-Debug-Digest
-X-CDN
-X-CDN-Provider
-X-Edge-IP
-X-Backend-Host
-X-Proxy-Host
-X-Akamai-Transformed
-X-True-Client-IP
-Fly-Request-ID
-Server-Timing
-X-Cache-Hit
-X-Cache-Status
-X-Middleton-Response
-X-Origin-Server
-X-Cloudflare-Visitor
-X-Cloudtrace-Context
+Client-IP
+X-Remote-IP
+X-Remote-Addr
+
+# Cloudflare Proxy Indicators
+CF-RAY
 CF-Connecting-IP
-X-Cloud-ID
-X-Google-Forwarding
-X-Forwarded-Scheme
-X-Original-Host
-X-Accel-Buffering
-X-Fastly-Request-ID
-X-Envoy-Internal
-X-Edge-Request-ID
-X-Edge-Cache
-X-Proxy-Backend
-True-Client-IP
-X-Azure-FDID
-X-Correlation-ID
-X-VPN-IP
-X-Anon-Client-IP
-X-Anonymous-Request-ID
-X-Tor-Exit-Node
-X-Proxy-User
-X-Onion-Request
-X-Tunnel-ID
-X-VPN-Forwarded-For
-X-Security-Proxy
-X-Identity-Forwarded
-X-WAF-Proxy
-X-Security-Appliance
-X-Security-Service-ID
-X-WAF-Trace
-X-DDoS-Request
-X-Malware-Scan
-X-Firewall-ID
-X-Mobile-Proxy
-X-IoT-Forwarding
-X-Mobile-Forwarded-For
-X-MIOT-Device-ID
-X-Device-Proxy-ID
-X-SIM-Proxy
-X-Cellular-Forwarding
-X-MMS-Forwarding
+X-Cloudflare-Visitor
+X-Cloudflare-Country
+
+# Amazon AWS Proxy Indicators
+X-Amzn-Trace-Id
 X-AWS-Edge-Trace
 X-Edge-Lambda-Invoke
-X-Compute-Region
-X-Edge-Trace
 X-CloudFront-Viewer-Country
 X-CloudFront-Forwarded-Proto
 X-CloudFront-Is-Desktop-Viewer
 X-CloudFront-Is-Mobile-Viewer
 X-CDN-Edge-ID
-X-Cloudflare-Country
-X-Balancer-Server
+X-Forwarded-ELB-IP
+X-Backend-Server
+X-Forwarded-Server-IP
+
+# Microsoft Azure/IIS ARR Proxy Indicators
+X-Azure-Ref
+X-Azure-FDID
+X-Azure-CDN
+X-Azure-Edge-Forwarded
+X-ARR-SSL
+X-Forwarded-Ssl
+X-Forwarded-For-Original
+
+# Fastly Proxy Indicators
+Fastly-Debug-Digest
+X-Fastly-Request-ID
+X-Served-By
+X-Cache-Hit
+X-Cache-Status
+X-Cache-Miss
+X-Cache
+X-Timer
+
+# Akamai Proxy Indicators
+X-Akamai-Transformed
+X-True-Client-IP
+X-Akamai-Origin-Hop
+X-EdgeConnect-MidMile-RTT
+X-EdgeConnect-Origin-MEX-Latency
+
+# Google Cloud Proxy Indicators
+X-Cloudtrace-Context
+X-Google-Forwarding
+X-Cloudtrace-ID
+
+# Load Balancer Specific Headers
+X-Load-Balancer
 X-Load-Balancer-ID
-X-CDN-Node-ID
+X-Balancer-Server
+X-LoadBalancer-Server
+X-Backend-Host
+X-Proxy-Host
+X-Proxy-Backend
+X-Upstream-Status
+X-Backend-Status
+
+# Reverse Proxy/General Proxy Headers
 X-Reverse-Proxy-ID
 X-Nginx-Proxy
-X-Heroku-Request-ID
-X-CF-Connecting-IP
+Proxy-Connection
+X-Proxy-ID
+X-Proxy-Via
+X-Proxy-Authenticate
+X-Proxy-Authtype
+X-Proxy-Connection
+X-Proxy-Client-IP
 X-Proxy-IP-Chain
-X-Cache-Request
+X-ProxyUser-IP
+
+# CDN Specific Headers
+X-CDN
+X-CDN-Provider
+X-Edge-IP
+X-CDN-Node-ID
+X-Edge-Cache
+X-Edge-Request-ID
+X-Edge-Trace
+
+# Heroku/Cloud Platform Proxy Indicators
+X-Heroku-Request-ID
+X-Forwarded-Https
+Front-End-Https
+X-Forwarded-Scheme
+X-Forwarded-Protocol
+X-Origin-Protocol
+X-Origin-Scheme
+
+# Application Gateway Proxy Indicators
 X-Request-Cluster-ID
 X-Gateway-Request-ID
 X-Global-Trace-ID
-X-Azure-CDN
-X-Azure-Edge-Forwarded
 X-Kubernetes-Service-Proxy
-X-Cloudtrace-ID
-X-Forwarded-Protocol
-X-Pound-Forwarded-For
-X-Original-Forwarded-For
-Front-End-Https
-X-ARR-SSL
-X-Forwarded-Ssl
-X-Forwarded-For-Original
-X-Rewrite-Url
-X-Original-Remote-Addr
-X-Forwarded-Client-IP
-X-Originating-Client-IP
-X-ProxyUser-IP
-X-Real-Client-IP
-X-Forwarded-For-IP
-X-Remote-IP
-X-Remote-Addr
-Client-IP
-X-Forwarded-For-Cluster
-Cluster-Client-IP
-X-Forwarded-For-Client
-X-True-IP
-X-Origin-IP
-X-Remote-User
-X-Forwarded-Host-Original
-X-Forwarded-Server-IP
-X-Forwarded-Remote-User
-X-Forwarded-From
-X-Forwarded-For-Original
-X-Forwarded-Access-Token
+
+# SSL/TLS Proxy Indicators
 X-Client-SSL-Cipher
 X-Client-SSL-Cert
 X-Client-SSL-Protocol
 X-Server-SSL-Cipher
 X-Server-SSL-Cert
 X-Server-SSL-Protocol
-X-Proxy-Via
-X-Proxy-Authenticate
-X-Proxy-Authtype
-X-Proxy-Connection
-X-Cache-Control
-X-Forwarded-Authorization
-X-Forwarded-Ssl
-X-Forwarded-For-Original
-X-Forwarded-For-IP
-X-Origin-IP
-X-Real-Host
-X-Requested-With
-X-Forwarded-Proto-Version
-X-Forwarded-Proto-Original
-X-Forwarded-Protocol-Version
-X-HTTP-Host-Override
-X-Original-Url
-X-Forwarded-Protocol
-X-Original-Method
-X-Original-Path
-X-Proxy-URL
-X-Rewriting-Url
-X-Forwarded-Response-Status
-X-Upstream-Status
-X-Upstream-Cache-Status
-X-Backend-Status
-X-Akamai-Origin-Hop
-X-EdgeConnect-MidMile-RTT
-X-EdgeConnect-Origin-MEX-Latency
+
+# Geographic/Location Proxy Indicators
 X-Country-Code
-X-Device
 X-GeoIP-Country
 X-GeoIP-Region
 X-GeoIP-City
 X-GeoIP-Latitude
 X-GeoIP-Longitude
-X-Forwarded-ELB-IP
-X-LoadBalancer
-X-LoadBalancer-Server
-X-Forwarded-Port
-X-Forwarded-Protocol
-X-True-Host
-X-True-Method
-X-True-Path
-X-Real-Method
-X-Real-Path
-X-Rewrite-Method
-X-Rewrite-Path
-X-Origin-Protocol
-X-Origin-Scheme
-X-Proxy-Client-IP
+
+# Mobile/WAP Proxy Indicators
+X-Mobile-Proxy
+X-IoT-Forwarding
+X-Mobile-Forwarded-For
+X-MIOT-Device-ID
+X-Device-Proxy-ID
+X-SIM-Proxy
+X-Cellular-Forwarding
+X-MMS-Forwarding
 X-WAP-Profile
 X-ATT-DeviceId
 X-WAP-Profile-Diff
@@ -200,18 +143,85 @@ X-Network-Info
 X-Network-Access-Node
 X-Network-Operator
 X-Network-Type
+
+# Security/VPN/Media Proxy Indicators
+X-VPN-IP
+X-Anon-Client-IP
+X-Anonymous-Request-ID
+X-Tor-Exit-Node
+X-Proxy-User
+X-Onion-Request
+X-Tunnel-ID
+X-VPN-Forwarded-For
+X-Security-Proxy
+X-Identity-Forwarded
+X-WAF-Proxy
+X-Security-Appliance
+X-Security-Service-ID
+X-WAF-Trace
+X-DDoS-Request
+X-Malware-Scan
+
+# Cache/Acceleration Proxy Indicators
+X-Proxy-Cache
+X-Cache
+X-Cache-Lookup
+X-Cache-Request
+X-Upstream-Cache-Status
+X-Varnish
+X-Accel-Buffering
+
+# Request Rewriting Proxy Indicators
+X-Original-URL
+X-Rewrite-URL
+X-Original-Remote-Addr
+X-Forwarded-Host-Original
+X-Forwarded-Access-Token
+X-Forwarded-Response-Status
+X-Forwarded-Authorization
+X-HTTP-Host-Override
+X-Requested-With
+X-Forwarded-Proto-Version
+X-Forwarded-Protocol-Version
+X-Proxy-URL
+X-Rewriting-Url
+X-Rewrite-Url
+
+# User Agent Forwarding Proxy Indicators
 X-Forwarded-User-Agent
 X-Forwarded-Referer
 X-Original-User-Agent
 X-Original-Referer
-X-Forwarded-Host
+
+# Additional Service-Specific Proxy Indicators
+X-Origin-Host
+X-Real-Host
+X-True-Host
+X-Host
+X-Forwarded-By
+X-Forwarded-From
+X-Origin-IP
+X-True-IP
+X-Real-Client-IP
+X-Forwarded-For-IP
+X-Forwarded-Client-IP
+X-Originating-Client-IP
+X-Forwarded-For-Cluster
+Cluster-Client-IP
+X-Forwarded-For-Client
+X-Remote-User
+X-Forwarded-Remote-User
 X-Forwarded-Server
 X-Forwarded-User
 X-Forwarded-Group
 X-Forwarded-Context
 X-Forwarded-Role
-X-Proxy-Authorization
 X-Custom-IP-Authorization
-Forwarded-For
-Forwarded-Host
-Forwarded-Server
+X-Compute-Region
+X-Middleton-Response
+X-Origin-Server
+Fly-Request-ID
+Server-Timing
+X-Corpo-ID
+Copy-ID
+X-Forwarded-Scheme-Protocol