Core Architecture

ProfitWheel is structured across six layers that together move data from ad platforms through intelligence systems to actionable insights.

📎 Evidence

Six-layer intro sentence — pw-docs/src/content/docs/core-architecture.mdx:8 — exact intro text; six ## Layer N headings (lines 28-50) confirm the model. ✅

Six-layer model — pw-enterprise/CLAUDE.md enumerates Flask API, PostgreSQL, Redis+RQ, LangChain/LangGraph/CrewAI AI, Elasticsearch, and GCP/Azure cloud, matching the doc’s layer stack. ✅

flowchart TD
FE["Client: React 19 SPA<br/>Cloudflare Pages"] --> BFF["Cloudflare Worker BFF<br/>CSP / headers / edge bindings"]
BFF --> API["Flask API monolith<br/>~52 blueprints"]
API --> PG[("PostgreSQL")]
API --> RD[("Redis")]
API --> ES[("Elasticsearch")]
API -- enqueue --> RQ["RQ workers<br/>fast / default / slow"]
CM["cron_master.py<br/>scheduled dispatcher"] --> RQ
CM -- scheduled --> PG
BQ[("BigQuery")]
CM -- scheduled --> BQ
RQ --> AI["AI layer<br/>LangChain / LangGraph / CrewAI · 12+ agents"]
AI --> LLM["LLM vendors<br/>OpenAI · Anthropic · Groq · Gemini · Cohere"]
RQ --> EXT["External integrations<br/>Google · Meta · TikTok · Pinterest · LinkedIn · Snapchat · GA4"]
AI --> PG

📎 Evidence

Client: React 19 SPA on Cloudflare Pages — pw-enterprise-frontend/package.json:108,114, pw-enterprise-frontend/wrangler.toml:3 — react/react-dom ^19.1.0; pages_build_output_dir confirms Pages. ✅

BFF: CSP, headers, edge bindings — pw-enterprise-frontend/functions/_middleware.ts:9-16,42-50 — Pages middleware sets CSP/security headers and Env declares AI, 3x Vectorize, Hyperdrive bindings. ✅

Flask API: ~52 blueprints — pw-enterprise/app.py:234-284,445 — grep -c app.register_blueprint = 52 (51 in 234-284 + the SSE blueprint at :445). ✅

PostgreSQL store — pw-enterprise/db_connection.py:9-45 — psycopg2 ThreadedConnectionPool (dev+prod, 1-100 conns); env keys POSTGRES_HOST/POSTGRES_PORT in .env:140-141. ✅

Redis store + RQ queues — pw-enterprise/.env:22-25,41-42, pw-enterprise/api/intent_insights.py:141-143 — redis_host/redis_port and rq_env/rq_env_fast/rq_env_slow keys read at runtime. ✅

Elasticsearch store — pw-enterprise/library/utils.py:17,83-90, pw-enterprise/pyproject.toml:37 — Elasticsearch client via search_engine_ip; elasticsearch>=9.2.0 dependency. ✅

RQ workers: fast/default/slow — pw-enterprise/api/intent_insights.py:141-143,1452,3037, pw-enterprise/pyproject.toml:82 — Queue(rq_env) and Queue(rq_env_slow) instantiated; rq>=2.6.1 dependency. ✅

API enqueues to RQ — pw-enterprise/api/intent_insights.py:1452 — qc = Queue(rq_env, connection=redis_conn, timeout=600) inside API module. ✅

cron_master.py dispatcher — pw-enterprise/cron_master.py:23 — run_type = sys.argv[1] then dispatches; 58 run_type == branches counted. ✅

cron_master dispatches to cron jobs (which enqueue RQ) — pw-enterprise/cron_master.py:8-11,23-28 — sets up tracing then dispatches run_type to cron/ functions; cron jobs use Queue(...). ✅

cron writes to Postgres — pw-enterprise/cron/audience_size_cron.py:603 — comment Write to Postgres first (before BigQuery which can hang) in a cron_master-dispatched job. ✅

BigQuery store — pw-enterprise/api/common.py:4,11, pw-enterprise/pyproject.toml:100 — from google.cloud import bigquery / client = bigquery.Client(); google-cloud-bigquery>=3.39.0 dependency. ✅

cron writes to BigQuery — pw-enterprise/cron/audience_size_cron.py:646,667 — BigQuery dedup+insert into xenon-antonym-363411.profitboost_dataset.trend_interest_country. ✅

AI layer LangChain/LangGraph/CrewAI; 12+ agents — pw-enterprise/pyproject.toml:12-16,33, pw-enterprise/v3/langraph_main.py:91-105 — all three frameworks declared; agent_description_mapping() has 12 named agents (library/ai_services.py:417-450 lists 13–16 more). ✅

LLM vendors OpenAI/Anthropic/Groq/Gemini/Cohere — pw-enterprise/.env:3,4,88,96, pw-enterprise/v3/langraph_main.py:119,124, pw-enterprise/api/reports.py:2620-2622 — keys + ChatAnthropic/ChatGoogleGenerativeAI imports + cohere.Client. ✅

External integrations (7+) — pw-enterprise/library/ — tiktok_package.py:20-21 plus pinterest_package.py, linkedin_package.py, snapchat_package.py, google_ads_package.py, facebook_package.py, google_analytics_4_package.py all present. ✅

RQ job kicks off AI crew — pw-enterprise/v3/main.py:27-46 — py_crew_run builds PythonCrew(...).crew().kickoff(inputs=inputs), the RQ→AI path. ✅

Cron/RQ jobs hit external platforms — pw-enterprise/cron/tiktok.py, pw-enterprise/cron/facebook.py — platform cron jobs exist and are dispatched by cron_master.py onto RQ. ✅

AI results written to Postgres — pw-enterprise/v3/main.py:53-56 — after crew completes, execute_query("UPDATE custom_report SET knowledge_bank=... WHERE id=...", commit=True). ✅

Layer 1: Client (Frontend)

A React 19 single-page application delivered via Cloudflare Pages. A thin Cloudflare Worker BFF sits in front of the SPA to enforce security headers and Content Security Policy, and to expose edge bindings (AI, Vectorize, Hyperdrive) without shipping credentials to the browser. (Cloudflare R2 object storage is used server-side via boto3 — it is not a Worker edge binding.)

📎 Evidence

React 19 SPA — pw-enterprise-frontend/package.json:108,114 — react/react-dom ^19.1.0. ✅

Cloudflare Pages delivery — pw-enterprise-frontend/wrangler.toml:3, pw-enterprise-frontend/package.json:15 — pages_build_output_dir and deploy script wrangler pages deploy dist. ✅

BFF enforces CSP + security headers — pw-enterprise-frontend/functions/_middleware.ts:24-50 — buildCSP + getSecurityHeaders (X-Frame-Options, X-Content-Type-Options, etc.) on HTML responses. ✅

BFF edge bindings: AI, Vectorize ×3, Hyperdrive — pw-enterprise-frontend/functions/_middleware.ts:9-16 and worker-configuration.d.ts:7-13 declare AI, three VECTORIZE_INDEX_*, and a Hyperdrive binding in the Worker Env (configured via the Pages dashboard; only AI is in wrangler.toml). R2 is NOT a Worker binding — it is used server-side via boto3 (pw-enterprise/library/storage.py). ✅

Layer 2: API (Backend)

A Flask monolith (app.py) with approximately 52 blueprint modules handling the full surface area of the product — campaigns, recommendations, reporting, user management, and more. This layer services synchronous request/response interactions for all user-facing calls.

📎 Evidence

Flask monolith app.py — pw-enterprise/app.py:9-10, pw-enterprise/pyproject.toml:44 — from flask import Flask (line 9; cited 7-10 includes dotenv); flask>=3.1.2 dependency. ✅

~52 blueprints — pw-enterprise/app.py:234-284,445 — grep -c app.register_blueprint = 52 (51 in 234-284 + the SSE blueprint at :445). ✅

MethodView pattern — pw-enterprise/api/amazon.py:22, pw-enterprise/api/authenticate.py:21 — from flask.views import MethodView across API modules. ✅

Layer 3: Async / Background

The operational backbone: cron_master.py acts as a scheduled dispatcher, feeding work into RQ workers across three priority queues (fast, default, slow) backed by Redis. Heavy operations — twin creation, large reports, embedding generation, and ad-platform syncs — are enqueued here, never run inline in the API request path.

📎 Evidence

cron_master.py dispatcher — pw-enterprise/cron_master.py:23-28 — run_type = sys.argv[1] branches to matching cron/ functions; 58 branches. ✅

Three RQ queues backed by Redis — pw-enterprise/.env:22-25, pw-enterprise/api/content_insights.py:552 — rq_env/rq_env_fast/rq_env_slow (plus unused rq_env_medium); Queue(rq_env_slow, connection=redis_conn). ✅

Heavy ops run as cron/RQ jobs — pw-enterprise/cron_master.py:369-386,497, pw-enterprise/cron/run_reports.py — report_embeddings and ai_twin_update_cron branches dispatch embedding/twin jobs. ✅

Three RQ tiers backed by Redis — pw-enterprise/config/constants.py:33-34 loads rq_env + rq_env_slow; .env:22-25 also defines rq_env_fast (and rq_env_medium), confirming the queue tiers. ✅

Layer 4: AI / Intelligence

LangChain, LangGraph, and CrewAI orchestrate 12+ specialist agents (the LangGraph mapping defines 12; the copilot agent list returns 13–16) that handle tasks ranging from budget recommendations to ad creative analysis. The agents run across multiple LLM vendors (OpenAI, Anthropic, Groq, Gemini, Cohere) and are instrumented in Langfuse (with a default 50% trace sample rate).

📎 Evidence

LangChain/LangGraph/CrewAI orchestration — pw-enterprise/pyproject.toml:12-16,33, pw-enterprise/v3/crew_f.py:10-14, pw-enterprise/v3/langraph_main.py:126-128 — deps + crewai imports + langgraph.graph StateGraph. ✅

12+ agents — pw-enterprise/v3/langraph_main.py:91-105 — agent_description_mapping() has 12 named agents; library/ai_services.py:417-450 lists 13–16 more. ✅

OpenAI vendor — pw-enterprise/.env:3, pw-enterprise/library/llm_completions.py:48-63 — openai_api_key key + "openai": ProviderConfig(api_key_env="openai_api_key"...). ✅

Anthropic vendor — pw-enterprise/.env:88, pw-enterprise/v3/langraph_main.py:2537-2540 — claude_key + ChatAnthropic(model_name="claude-sonnet-4-5", api_key=claude_key). ✅

Groq vendor — pw-enterprise/.env:96, pw-enterprise/library/llm_completions.py:33-47, pw-enterprise/pyproject.toml:15 — groq_api_key + "groq": ProviderConfig(...) + langchain-groq>=1.1.1. ✅

Gemini vendor — pw-enterprise/pyproject.toml:20, pw-enterprise/v3/langraph_main.py:329-330 — langchain-google-genai>=4.2.0 + ChatGoogleGenerativeAI(model="gemini-3-flash-preview"). ✅

Cohere vendor — pw-enterprise/.env:4, pw-enterprise/api/reports.py:2620-2622, pw-enterprise/pyproject.toml:32 — cohere_api_key + co = cohere.Client(cohere_api_key) + cohere>=5.20.0. ✅

Langfuse tracing — pw-enterprise/library/langsmith_metadata.py:16-34, pw-enterprise/cron_master.py:8-11, pw-enterprise/pyproject.toml:105-106 — setup_langfuse_tracing() auto-instruments LangChain; called at cron startup. ✅

Langfuse tracing (50% default sample) — pw-enterprise/library/langsmith_metadata.py:25-27 defaults LANGFUSE_SAMPLE_RATE to 0.5; instrumentation hooks all LangChain calls but ~50% are sampled by default. ✅

Layer 5: Data Stores

Four purpose-fit stores: PostgreSQL is the system of record for all structured business data; Redis serves as both the in-memory cache and the RQ job queue; Elasticsearch powers full-text search and time-series metric queries; BigQuery is the analytics warehouse for large-scale reporting and ML feature pipelines.

📎 Evidence

PostgreSQL system of record — pw-enterprise/db_connection.py:26-45 — dev + prod ThreadedConnectionPool (1-100 conns each). ✅

Redis backs RQ queues (cache role under-evidenced) — pw-enterprise/.env:41-42, pw-enterprise/api/content_insights.py:552 — redis_host/redis_port + Queue(..., connection=redis_conn); cited lines prove queue but not cache. ⚠️ Needs a Redis cache GET/SET citation.

Elasticsearch search + time-series — pw-enterprise/library/utils.py:83-90, pw-enterprise/cron/es_refresh_cron.py, pw-enterprise/cron/behaviour_timeseries_cron.py — ES client + dedicated refresh and time-series crons exist. ✅

BigQuery analytics warehouse — pw-enterprise/api/common.py:4,11, pw-enterprise/cron/audience_size_cron.py:667 — bigquery.Client() + cron inserts trend/audience data; ‘ML feature pipelines’ phrase not directly evidenced. ⚠️

Cloudflare R2 object storage — pw-enterprise/library/storage.py:1-7, pw-enterprise/.env:144-148 — R2 upload helpers + R2_ACCOUNT_ID/R2_ACCESS_KEY_ID/etc. env keys. ✅

Azure Blob (legacy) — pw-enterprise/library/storage.py:6,46 — docstring notes migration from Azure Blob to R2; Convert old Azure Blob Storage URL to R2 URL helper. ✅

GCP Secret Manager (primary) — pw-enterprise/library/gcp_secrets.py:1-13, pw-enterprise/app.py:16,30,51-52 — secretmanager module; app loads 18 secrets in parallel via ThreadPoolExecutor at startup. ✅

Azure Key Vault (legacy) — pw-enterprise/pyproject.toml:27-28, pw-enterprise/library/gcp_secrets.py:4-5 — azure-identity/azure-keyvault-secrets deps still present; docstring says GCP replaced Azure Key Vault. ✅

Layer 6: External Integrations

Seven-plus ad platforms — Google, Meta, TikTok, Pinterest, LinkedIn, Snapchat, and GA4 — feed raw performance data in and receive optimized signals out. Additional document and media AI services support creative intelligence use cases.

📎 Evidence

7+ ad platforms — pw-enterprise/library/ — dv360_package.py:32-33, tiktok_package.py:20-21, youtube_package.py:14-16 plus google_ads/facebook/pinterest/linkedin/snapchat/google_analytics_4 packages — 8+ integrations, ‘7+’ is a safe minimum. ✅

---

Key flows

The layer model shows what the system is made of. These diagrams trace how work actually moves through it — the paths a senior engineer needs to reason about the platform.

1. Research & twin lifecycle

The core product loop. Twins are not generic chatbots: they are grounded in real audience signals pulled from ad platforms, distilled into behavior / intent / mentions reports plus persistent memory, and then used to run qualitative and quantitative research that informs marketing decisions.

flowchart LR
META["Ad-platform audiences<br/>(Meta & others)"] --> SIG["Audience signals<br/>behavior · intent · mentions"]
SIG --> RPT["Grounding reports<br/>+ twin memory"]
RPT --> TWIN["AI Twin persona<br/>ai_twin/ module"]
TWIN --> QUAL["Qualitative<br/>focus groups · interviews"]
TWIN --> QUANT["Quantitative<br/>surveys · respondents"]
QUAL --> INS["Insights"]
QUANT --> INS
INS --> MKT["Informs marketing<br/>& campaign decisions"]

📎 Evidence

Ad-platform audiences are the source node — pw-enterprise/cron_master.py:199-236 audience_run imports/calls facebook_run, snapchat_run, google_analytics_run, tiktok_run, pinterest_run, linkedin_run, and library/facebook_package.py wraps Meta. ✅

Signals = behavior / intent / mentions — pw-enterprise/ai_twin/respondents/respondent_prompt_composer.py:8-10 docstring lists --behavior, --intent, --mentions report inputs. ✅

Signals distilled into reports + twin memory — pw-enterprise/ai_twin/respondents/respondent_prompt_composer.py:57-118 prompt mandates 2-3 memory fragments per respondent built from behavior/intent/mentions summaries. ✅

Twin persona lives in ai_twin/ — pw-enterprise/ai_twin/group/main.py:1-20 (focus-group entry) and ai_twin/respondents/agents.py:49 (SurveyAnalysisAgent) confirm both submodules. ✅

Qualitative focus groups + interviews — pw-enterprise/ai_twin/group/main.py:85 builds focus-group prompts and :487-499 builds interview-mode (Says/Thinks) prompts. ✅

Quantitative surveys with respondents — pw-enterprise/ai_twin/respondents/agents.py:49,62 (SurveyAnalysisAgent.fetch_survey_data) and response_aggregator.py:2580 (_get_real_respondent_count). ✅

Insights inform marketing decisions — pw-enterprise/ai_twin/respondents/agents.py:899-935 system prompt: ‘strategic survey insights advisor … actionable business insights … better business decisions’. ✅

2. Async job pipeline

Heavy work never runs inline in the request path. cron_master.py dispatches scheduled cron/ jobs, and both cron and the API enqueue onto three Redis-backed RQ priority queues drained by workers. Apache Airflow provides higher-level orchestration on top.

flowchart TD
API["Flask API"] -- enqueue --> ENQ{"RQ enqueue"}
CM["cron_master.py<br/>dispatcher"] --> CRON["cron/ jobs (~50)"]
CRON --> ENQ
AF["Apache Airflow<br/>library/airflow_client.py"] -. orchestrates .-> CM
ENQ --> FAST["queue: fast"]
ENQ --> DEF["queue: default"]
ENQ --> SLOW["queue: slow"]
FAST --> W["RQ workers"]
DEF --> W
SLOW --> W
RD[("Redis<br/>queue backend")] -. backs .-> ENQ
W --> PG[("PostgreSQL")]
W --> ES[("Elasticsearch")]
W --> AIW["AI agents"]

📎 Evidence

Flask API enqueues to RQ — pw-enterprise/api/content_insights.py:547-556 creates Queue(rq_env_slow,...) and qc.enqueue(...); api/survey.py:2064-2067 uses Queue(rq_env,...). ✅

cron_master.py is the CLI dispatcher — pw-enterprise/cron_master.py:2 reads sys.argv and branches on run_type (e.g. audience_run at :199) to import/call cron jobs. ✅

cron/ holds ~50 job files — pw-enterprise/cron/ has 55 entries, 54 excluding __init__.py; doc’s ‘~50’ is within rounding. ✅

Airflow orchestration via library/airflow_client.py — file docstring says ‘HTTP wrapper for Airflow REST API’ and imports AIRFLOW_API_BASE_URL etc. from config/airflow_config.py. ✅

Three RQ tiers (fast/default/slow) — pw-enterprise/.env:22-25 defines rq_env, rq_env_fast, rq_env_medium, rq_env_slow; api/content_insights.py:31 consumes rq_env_slow. ✅ (note: an rq_env_medium also exists)

Redis backs RQ queues — pw-enterprise/api/planner.py:4558 Redis(host=redis_host,...) connects every Queue(); same pattern in api/content_insights.py:551. ✅

Workers write PG/ES and invoke AI agents — pw-enterprise/cron_master.py:253-257 syncs PostgreSQL↔Elasticsearch; library/ai_services.py:414-450 lists dispatched agents. ✅

3. Request lifecycle & auth

Every user-facing call passes through the Cloudflare Worker BFF (which adds CSP, security headers, and edge bindings), then hits a Flask MethodView endpoint. Each endpoint authenticates the Authorization header before running business logic.

flowchart LR
U["Browser<br/>React 19 SPA"] --> CF["Cloudflare Worker BFF<br/>CSP · headers · edge bindings"]
CF --> EP["Flask MethodView<br/>endpoint"]
EP --> AUTH["authentication(auth_header)<br/>JWT / session"]
AUTH -- valid --> BL["Business logic"]
AUTH -- invalid --> R401["401 Unauthorized"]
BL --> PG[("PostgreSQL")]
BL --> RD[("Redis cache")]
BL --> RESP["JSON response"]

📎 Evidence

Client is a React 19 SPA — pw-enterprise-frontend/package.json:108 pins "react": "^19.1.0". ✅

Cloudflare Worker BFF adds CSP/headers/bindings — pw-enterprise-frontend/functions/_middleware.ts:9-16,44-45 declares Env bindings (AI, Vectorize x3, Hyperdrive) and sets Content-Security-Policy + X-Frame-Options: DENY. ✅

Flask MethodView endpoints — pw-enterprise/api/misc.py:30 imports from flask.views import MethodView; :160 class fbAdaccountApi(MethodView). ✅

authentication(auth_header) validates JWT — pw-enterprise/api/jwt.py:111-121 defines it and delegates to decode_jwt_token; imported in app.py:133. ✅

JWT HS256 with configurable expiry — pw-enterprise/api/jwt.py:10-14 sets JWT_ALGORITHM = "HS256"; encode_jwt_token (:65-79) takes custom_days/is_trusted for expiry (default 168h). Note: module-level JWT_EXP_DELTA_HOURS = 1 is defined but unused. ✅

Invalid JWT path — pw-enterprise/api/jwt.py:103-108 returns {"status":"error"} for expired/invalid tokens, but decode_jwt_token/authentication return an error dict, not an HTTP 401; the 401 mapping happens in individual endpoints and was not verified here. ⚠️

Business logic uses PostgreSQL + Redis — pw-enterprise/db_connection.py:26-46 builds PG ThreadedConnectionPool(1,100); cache.py:10 import redis. ✅

4. AI orchestration

Agents are composed with LangChain / LangGraph / CrewAI and dispatched to whichever LLM vendor fits the task — some via the shared llm_completions.py layer (OpenAI/Groq), others via direct vendor clients. Calls are instrumented in Langfuse (50% default sample rate); retrieval is backed by FAISS vectors and Cloudflare Vectorize.

flowchart TD
REQ["Agent task<br/>(from RQ / API)"] --> ORCH["Orchestration<br/>LangChain · LangGraph · CrewAI"]
ORCH --> ROUTE{"LLM dispatch<br/>llm_completions.py + direct clients"}
ROUTE --> OAI["OpenAI"]
ROUTE --> ANT["Anthropic"]
ROUTE --> GRQ["Groq"]
ROUTE --> GEM["Gemini"]
ROUTE --> COH["Cohere"]
ROUTE --> CER["Cerebras · DeepSeek · Perplexity · OpenRouter"]
ORCH --> VEC["FAISS · Vectorize<br/>retrieval"]
ORCH -. traced .-> LF["Langfuse"]
OAI --> OUT["Result"]
ANT --> OUT
GRQ --> OUT
GEM --> OUT
COH --> OUT
CER --> OUT

📎 Evidence

LangChain / LangGraph / CrewAI orchestration — pw-enterprise/requirements.txt:518 langchain==1.2.7, :563 langgraph==1.0.7, :154 crewai==1.6.1. ✅

LLM dispatch: shared layer for OpenAI/Groq, direct clients for the rest — pw-enterprise/library/llm_completions.py:32-64 PROVIDER_CONFIGS covers openai and groq; Anthropic/Gemini/Cohere/Cerebras/etc. are instantiated directly at their call sites. Diagram node relabeled accordingly. ✅

OpenAI vendor — pw-enterprise/.env:3 openai_api_key; loaded in app.py:304. ✅

Anthropic vendor — pw-enterprise/.env:136 ANTHROPIC_API_KEY; ai_twin/respondents/agents.py:337-338 instantiates ChatAnthropic(model_name="claude-sonnet-4-5") (using env claude_key). ✅

Groq vendor — pw-enterprise/.env:96 groq_api_key; library/ai_services.py:345-347 imports ChatGroq and reads groq_api_key. ✅

Gemini vendor — pw-enterprise/.env:131 GOOGLE_API_KEY; ai_twin/group/main.py:8 imports ChatGoogleGenerativeAI. ✅

Cohere vendor — pw-enterprise/.env:4 cohere_api_key; api/planner.py:12 import cohere. ✅

Cerebras vendor — pw-enterprise/.env:107 CEREBRAS-API-KEY; v3/langraph_main.py:396-398 imports Cerebras and calls cerebras_client.chat.completions.create. ✅

DeepSeek vendor — pw-enterprise/.env:110 deepseek_api_key; consumed in v3/langraph_main.py:32. ✅

Perplexity vendor — pw-enterprise/.env:111 perplexity_api_key; library/ai_services.py:310-316 calls https://api.perplexity.ai/chat/completions. ✅

OpenRouter vendor — pw-enterprise/.env:168 openrouter; api/assets/routers/conversations.py:4126-4131 sets openai_api_base="https://openrouter.ai/api/v1". ✅

FAISS + Cloudflare Vectorize retrieval — pw-enterprise/app.py:1219-1227 builds faiss.IndexFlatL2 FAISS store; add_vectors.py:17-23 VectorizeBase calls the Cloudflare AI embeddings API. ✅

Langfuse tracing (50% default sample) — pw-enterprise/library/langsmith_metadata.py:16-33 calls LangChainInstrumentor().instrument() and :25-27 sets a default LANGFUSE_SAMPLE_RATE of 0.5; intro now states the sampling. ✅

12+ agents (13–16 in the copilot list) — pw-enterprise/library/ai_services.py:417-431 returns 13 agents in app mode and :433-450 returns 16 otherwise (plus a separate SurveyAnalysisAgent); v3/langraph_main.py:91-105 defines 12. ✅

5. Data & storage topology

Four query stores plus two object stores, fronted by secret managers. Each store is purpose-fit rather than overlapping.

flowchart TD
API["Flask API + RQ workers"]
SEC["Secrets<br/>GCP Secret Manager · Azure Key Vault"] -. injected .-> API
API --> PG[("PostgreSQL<br/>system of record")]
API --> RD[("Redis<br/>cache + RQ queues")]
API --> ES[("Elasticsearch<br/>search · time-series")]
API --> BQ[("BigQuery<br/>analytics warehouse")]
API --> R2["Cloudflare R2<br/>object storage"]
API --> AZ["Azure Blob<br/>storage (legacy)"]

📎 Evidence

Flask API is the primary write source — pw-enterprise/app.py:234-445 registers 52 blueprints (e.g. register_blueprint(research_bp) at :234, sse at :445). ✅

GCP Secret Manager injects secrets — pw-enterprise/library/gcp_secrets.py:1-18 wraps google.cloud.secretmanager; app.py:30-52 loads _SECRETS in parallel via load_secret_to_env. ✅

Azure Key Vault is legacy — pw-enterprise/requirements.txt:62 still pins azure-keyvault-secrets==4.10.0, but library/secrets.py:109-115 set_secret_to_key_vault/get_secret_from_key_vault now delegate to GCP Secret Manager. ✅

PostgreSQL is the system of record — pw-enterprise/db_connection.py:9-46 uses psycopg2 ThreadedConnectionPool(1,100) for dev/prod databases. ✅

Redis = cache + RQ queue — pw-enterprise/.env:41-43 redis_host/port/password; cache.py:10 import redis (cache) while RQ queues use the same Redis. ✅

Elasticsearch active — pw-enterprise/library/utils.py:17 from elasticsearch import Elasticsearch; .env:26 elastic_key. ✅

BigQuery warehouse — pw-enterprise/api/common.py:4-11 imports from google.cloud import bigquery and instantiates bigquery.Client(). ✅

Cloudflare R2 object storage — pw-enterprise/library/storage.py:1-7,32-37 documents migration to R2 and get_r2_client() builds a boto3 S3 client at *.r2.cloudflarestorage.com. ✅

Azure Blob is legacy — pw-enterprise/library/storage.py:29 keeps AZURE_STORAGE_URL_PATTERN = "https://pwdevstorage.blob.core.windows.net/" for migration; .env:47-48 still has Azure storage connections. ✅

6. External integration map

Dozens of outbound integrations feed the platform, grouped into ad/social platforms, retail-commerce data, and search/web/enrichment providers. In-house wrappers in library/ and scheduled cron jobs normalize them into Elasticsearch and PostgreSQL.

flowchart LR
subgraph ADS["Ad / social platforms"]
  G["Google Ads · DV360"]
  M["Meta"]
  T["TikTok · Pinterest"]
  L["LinkedIn · Snapchat"]
  Y["YouTube · GA4"]
end
subgraph RETAIL["Retail / commerce data"]
  RF["Rainforest (Amazon)"]
  BB["BigBox · BlueCart · RedCircle"]
end
subgraph WEB["Search / web / enrichment"]
  TV["Tavily · Serper · ValueSERP"]
  SC["ZenRows · scraping APIs"]
  DM["Census · Precisely"]
end
ADS --> ING["Ingestion<br/>library/*_package.py · cron jobs"]
RETAIL --> ING
WEB --> ING
ING --> ES[("Elasticsearch")]
ING --> PG[("PostgreSQL")]

📎 Evidence

Google Ads + DV360 wrappers — pw-enterprise/library/google_ads_package.py:42-46 and library/dv360_package.py:32-36 (dv360_client_id/secret, def dv360_auth). ✅

Meta/Facebook wrapper — pw-enterprise/library/facebook_package.py exists (e.g. :53 cohere_api_key); finder’s comment-header snippet was fabricated but the file is the Meta wrapper. ✅

TikTok + Pinterest wrappers — pw-enterprise/library/tiktok_package.py and library/pinterest_package.py both exist. ✅

LinkedIn + Snapchat wrappers — pw-enterprise/library/linkedin_package.py and library/snapchat_package.py both exist. ✅

YouTube + GA4 wrappers — pw-enterprise/library/youtube_package.py:14-16 (youtube_client_secret, youtube.force-ssl scope) and library/google_analytics_4_package.py exists. ✅

Rainforest (Amazon) integration — pw-enterprise/api/product.py:89 rainforest_key, :311 calls https://api.rainforestapi.com/request, caches into rainforest_request table. ✅

BigBox / BlueCart / RedCircle — pw-enterprise/api/social_insights.py:79-81 loads all three keys; :536 calls bluecartapi.com, :603 calls redcircleapi.com; secrets loaded in app.py:37-39. ✅

Tavily configured but usage unverified — pw-enterprise/.env:137 has TAVILY_API_KEY, but no active SDK import/call was found in the codebase; evidence is env-key only. ⚠️

Serper integration — pw-enterprise/custom_answer_engine.py:12,17,29 imports GoogleSerperAPIWrapper and instantiates it with SERPER_API_KEY. ✅

ValueSERP integration — pw-enterprise/library/valueserp_helper.py:12,165 reads valueserp_api_key and calls https://api.valueserp.com/search. ✅

ZenRows scraping integration — pw-enterprise/config/constants.py:40 and v3/tools.py:56 both load zenrow_token = os.environ["zenrow_token"]. ✅

US Census enrichment — pw-enterprise/ai_twin/respondents/universe_stage_fetcher.py:246,310 defines _query_census hitting https://api.census.gov/data/...; .env:138 CENSUS_API_KEY. ✅

Precisely enrichment — pw-enterprise/api/household.py:200 def precisely_household(...); app.py:33-34 loads PRECISELY-SECRET/PRECISELY-KEY from GCP. ✅

Ingestion via library/*_package.py — pw-enterprise/library/ has 11 *_package.py files (dv360, facebook, google_ads, google_analytics, google_analytics_4, linkedin, pinterest, snapchat, spotify, tiktok, youtube). ✅

Cron jobs normalize into ES + PG — pw-enterprise/cron/sync_audience_between_pg_and_es.py exists and cron_master.py:253-257 runs sync_audience_between_pg_and_es_cron(). ✅

---

Client → BFF → Flask API → (Postgres / Redis) → RQ workers → AI layer → external platforms, with cron jobs feeding the data stores on a schedule.