Skip to content

Base Architecture & System Design

Base is the core customer intelligence engine of the Social Signal platform. It converts raw, un-enriched customer data (e.g. email hashes, zip codes, LTV, revenue metrics) into deep, multi-dimensional customer insights for human marketers, visual dashboards, and autonomous AI agents.

1. System Vision & Core Concept

Base operates as the architectural inverse of Beacon:

 ┌────────────────────────────────────────────────────────────────────────┐
 │                              BEACON                                    │
 │  Web Visitors  ──>  Capture Engine  ──>  Lead Capture  ──>  BigQuery   │
 └────────────────────────────────────────────────────────────────────────┘
                                     ▲
                                     │ Spatial Persona Intelligence
                                     ▼
 ┌────────────────────────────────────────────────────────────────────────┐
 │                               BASE                                     │
 │  Customer Data  ──>  Snapshot Ingest  ──>  Resolution  ──> Insights    │
 └────────────────────────────────────────────────────────────────────────┘
  • Beacon (Traffic $\rightarrow$ Leads): Captures anonymous real-time web visits, identifies leads, and streams event telemetry into BigQuery (visits_raw, leads_raw).
  • Base (Customer Lists $\rightarrow$ Insights): Ingests point-in-time customer lists, resolves 72 Spatial Mosaic persona segments, and generates audience briefs, demographic indexing, and AI agent context files.

2. Core Domain Model & Entity Architecture

Base decouples batch submission headers (Snapshots) from item-level customer data (Customers) to achieve clean multitenancy, high query performance, and zero data duplication.

                  ┌──────────────────────────────┐
                  │        BaseSnapshot          │
                  │   (base_snapshots Table)     │
                  │   header metadata & status   │
                  └──────────────┬───────────────┘
                                 │
                                 │ 1 : N (Compound Tenant Key)
                                 │ (snapshot_id, _id, domain)
                                 ▼
                  ┌──────────────────────────────┐
                  │       BaseCustomerRaw        │
                  │ (base_customers_raw Table)   │
                  │   staged customer records    │
                  └──────────────┬───────────────┘
                                 │
                                 │  + lk_email_segments
                                 │  + zip_persona_arrays
                                 ▼
                  ┌──────────────────────────────┐
                  │         BaseCustomer         │
                  │    (base_customers View)     │
                  │ enriched single source-of-truth│
                  └──────────────────────────────┘

Entity Schemas

A. BaseSnapshot (socialsignal.beaconv2.base_snapshots)

Stores upload metadata header records. * snapshot_id: 1–32 character lower-case slug identifier (/^[a-z0-9_-]+$/, e.g. q3-customers, black-friday). * pid: Partner ID (channel host / white-label account). * _id: User / Account ID (tenant key). * domain: Target domain associated with the snapshot. * batch_type: Ingestion mode (CSV_UPLOAD, API_STREAM, INTEGRATION, MANUAL). * value_type: Value metric interpretation (LTV, AOV, Total_Revenue, ARPU). * status: Lifecycle state (ACTIVE, ARCHIVED, DELETED). * uploaded_at: Ingestion timestamp (CURRENT_TIMESTAMP()).

B. BaseCustomerRaw (socialsignal.beaconv2.base_customers_raw)

Stores individual staged customer rows linked to parent snapshot and tenant keys. * uuid: Unique row UUID (GENERATE_UUID()). * snapshot_id: Foreign key linking to BaseSnapshot. * _id: User / Account ID (tenant key). * domain: Target domain (tenant key). * fkID: External CRM / Customer ID. * email_hash: SHA-256 hash of customer email address. * zip: 5-digit US ZCTA postal code. * value: Monetary value (defined by snapshot's value_type). * segment: Assigned Spatial persona segment (e.g. A01, B04, or NULL if unassigned). * persona_version: Persona taxonomy version (e.g. 26.01). * status: Row-level status (ACTIVE, ARCHIVED, DELETED). * snapshot_at: Temporal cutoff timestamp of customer record.

C. BaseCustomer (socialsignal.beaconv2.base_customers View)

Single source-of-truth View that dynamically joins raw customer rows with snapshot headers and email resolution lookups. * Dynamic Persona Resolution: Automatically falls back to historical email matches (lk_email_segments) when c.segment is NULL. * Soft-Delete Enforcement: Automatically excludes rows where s.status = 'DELETED' or c.status = 'DELETED'. * Compound Multitenant JOIN: Joins ON c.snapshot_id = s.snapshot_id AND c._id = s._id AND c.domain = s.domain to guarantee zero cross-tenant data pollution.

3. Data Ingestion & Flow Execution Pipeline

 ┌────────────────┐      ┌────────────────┐      ┌─────────────────────────┐      ┌──────────────┐
 │  Dashboard UI  │      │ Flow Runner    │      │ batch-publish-to-bq     │      │ GCP BigQuery │
 │  Partner UI    ├─────>│ Edge Gateway   ├─────>│ GCP Cloud Function      ├─────>│ beaconv2     │
 │  AI Agent/API  │      │ (BRouter)      │      │ (insert_rows_json)      │      │ Dataset      │
 └────────────────┘      └────────────────┘      └─────────────────────────┘      └──────────────┘
  1. Edge Gateway Validation: Incoming JSON requests pass through BRouter using Zod schemas (BaseSnapshotSchema, BaseCustomerRawSchema), validating snapshot_id slug rules (1-32 chars, lowercase) and tenant context.
  2. Parallel Streaming Ingestion: The edge flow runner dispatches batch payloads to the batch-publish-to-bq Cloud Function, streaming header and customer rows into BigQuery in parallel with zero lock contention.
  3. Declarative Flow Templates:
  4. base.snapshot.create: POST endpoint to register snapshot header & stream customer rows.
  5. base.snapshot.list: GET endpoint to query active snapshots by user/domain.
  6. base.snapshot.get: GET endpoint to retrieve snapshot metadata, total counts, email match rate %, and value statistics.
  7. base.snapshot.delete: DELETE endpoint to soft-delete a snapshot (status = 'DELETED').

4. Persona Resolution & Demographic Indexing

When customer lists are uploaded without pre-assigned segments, Base resolves persona identities using a multi-tiered resolution cascade:

  1. Direct Segment (DIRECT): Segment provided directly in raw record payload.
  2. Email Hash Lookup (EMAIL): Matched against historical email hashes in lk_email_segments.
  3. Zip Persona Array (ZIP / ZIP-BQ): Sampled from US Census ZCTA persona probability arrays (zip_persona_arrays).
  4. Unmatched (UNMATCHED): Fallback for records with missing or invalid location/email data.

5. Multi-Surface Output Architecture

Base is engineered to serve three distinct consumption surfaces:

                               ┌────────────────────────────────────────┐
                               │             BASE OUTPUTS               │
                               └──────────────────┬─────────────────────┘
                                                  │
         ┌────────────────────────────────────────┼────────────────────────────────────────┐
         │                                        │                                        │
         ▼                                        ▼                                        ▼
┌──────────────────┐                    ┌──────────────────┐                    ┌──────────────────┐
│ Human Marketers  │                    │ Visual Dashboards│                    │ AI Agents & MCP  │
│ - PDF Briefs     │                    │ - Force Graphs   │                    │ - Markdown Docs  │
│ - Executive Summaries                 │ - D3 Centering   │                    │ - JSON Payload   │
│ - Audience Index │                    │ - Segment Charts │                    │ - MCP Tools      │
└──────────────────┘                    └──────────────────┘                    └──────────────────┘
  1. Human Marketers: Executive audience briefs (base.generate.brief), brand alignment guides, and demographic index scores ($+20\%$ over-indexing / $-20\%$ under-indexing).
  2. Visual Dashboards & Force Graphs: Short 1-32 char slug identifiers (q3-customers), centered demographic handles, and segment affinity visualizations.
  3. AI Agents & MCP Integration: Structural .md and .json context outputs consumed by autonomous LLM agents and Anthropic/Google MCP servers.