Firewall & WAF Integration

Configure your WAF or CDN to allow betterfound's signed fetch requests through bot protection without exposing your site to unauthorised traffic. Choose the integration tier that matches your security posture.

Two integration tiers

Tier A — Public-site allow-list (lightweight)

Allow betterfound's outbound IP range past your bot challenges, optionally augmented by signature validation.

You do:

• Allow-list betterfound's egress CIDRs (published at https://betterfound.ai/ips.json) on your WAF / Bot Manager
• Optionally add a rule requiring a valid X-Shopmind-Signature matching your tenant secret
• Add to robots.txt:

  User-agent: shopmind-fetch
  Allow: /
  Crawl-delay: 1

Trade-offs: Simplest to enable; works against your live site (cache hits identical to user traffic); you accept that betterfound can read any public URL on your brand host.

Tier B — Dedicated content endpoint (recommended for enterprise)

You expose a single private endpoint that returns rendered text for any of your URLs. betterfound routes every fetch_url through it instead of hitting your public site.

You do:

• Stand up an endpoint (ideally cached at your edge) at e.g. https://api.your-brand.com/_betterfound/content
• Receive ?path=<https://www.your-brand.com/cameras/eos-r5> (always absolute, always within your brand host)
• Validate the four betterfound headers (see Authentication & Signing)
• Return text/html or text/plain of the requested page

Trade-offs: More setup but strongest security posture — no public-site traffic, fully observable in your logs, rate-limited via your own quota system. Typically ~10× faster than public-site fetches because you can serve from your CMS cache.

Reference endpoint (Express / Node.js)

import express from 'express'
import fetch from 'node-fetch'
import { verifyBetterfoundRequest } from './verify.js'

const app          = express()
const ALLOWED_HOST = 'www.your-brand.com'

app.get('/_betterfound/content', async (req, res) => {
  const path = String(req.query.path ?? '')
  const url  = (() => { try { return new URL(path) } catch { return null } })()

  if (!url || url.hostname !== ALLOWED_HOST)
    return res.status(400).json({ error: 'invalid path' })

  if (!verifyBetterfoundRequest(req, path))
    return res.status(401).json({ error: 'signature invalid' })

  const upstream = await fetch(url.toString())
  res.set('Content-Type', 'text/html')
     .status(upstream.status)
     .send(await upstream.text())
})

app.listen(8080)

WAF configuration snippets

Cloudflare (WAF + Bot Fight Mode)

Create a custom rule under Security → WAF → Custom rules:

(http.user_agent contains "shopmind-fetch/1.0"
 and http.request.headers["x-shopmind-signature"][0] matches "^[a-f0-9]{64}$"
 and ip.src in {52.x.x.0/24 18.x.x.0/24})

Action: Skip → "All Bot Fight Mode rules" + "All managed rules"

For Tier B, create a Page Rule on api.your-brand.com/_betterfound/content* that allows betterfound's CIDRs and rate-limits to 4 rps per X-Shopmind-Tenant header.

AWS WAF (with CloudFront / ALB)

Web ACL rule — priority 50, before any rate-limit rules:

{
  "Name": "AllowBetterfoundFetchAgent",
  "Action": { "Allow": {} },
  "Statement": {
    "AndStatement": {
      "Statements": [
        { "ByteMatchStatement": {
            "FieldToMatch": { "SingleHeader": { "Name": "user-agent" } },
            "PositionalConstraint": "STARTS_WITH",
            "SearchString": "shopmind-fetch/",
            "TextTransformations": [{ "Priority": 0, "Type": "NONE" }]
        }},
        { "RegexMatchStatement": {
            "FieldToMatch": { "SingleHeader": { "Name": "x-shopmind-signature" } },
            "RegexString": "^[a-f0-9]{64}$",
            "TextTransformations": [{ "Priority": 0, "Type": "NONE" }]
        }},
        { "IPSetReferenceStatement": { "ARN": "<arn-of-betterfound-cidrs-set>" } }
      ]
    }
  }
}

Pair with a RateBasedStatement keyed on x-shopmind-tenant — 4 rps soft cap, 240 over a 5-minute window.

Akamai (Bot Manager Premier)

Bot Manager → Custom Bot Categories → New Category "betterfound agent":

• Match condition 1: Header User-Agent starts with shopmind-fetch/
• Match condition 2: Header X-Shopmind-Signature matches regex ^[a-f0-9]{64}$
• Action: Allow request, override Bot Score to 0

For Tier A, also add the published CIDRs as a Match Target allow-list.

Imperva (Cloud WAF)

Site → Security → Custom Rules → New rule:

• Match: User-Agent matches shopmind-fetch/* AND Client IP in betterfound CIDR list
• Action: Whitelist (skip all subsequent security checks)

Onboarding checklist

• ☐ Download betterfound egress CIDRs from https://betterfound.ai/ips.json
• ☐ Choose Tier A or Tier B
• ☐ Install the WAF rule for your platform (snippets above)
• ☐ Test in staging with the smoke test curl command
• ☐ If Tier B — build and deploy your content endpoint, return the URL to betterfound dashboard → Settings → Fetch Agent
• ☐ Smoke test in production with a single test prompt; confirm betterfound logs show 200 from your origin