Backend Data Unification Layer Plan - v4

name: Unified Metrics Service Layer (v4) overview: Restructure MongoDB, build a repository layer, and expose a flexible REST API to create a single, accurate metrics service that serves the AI assistant, frontend charts, Lambdas, and future features like per-metric drilldowns and AI-constructed dashboards. todos:

id: phase-0-verify-mongo content: Verify Atlas MongoDB version supports Time Series collections (5.0+) status: pending
id: phase-0-schema-cleanup content: “Pre-work: remove dead fields from platform metrics schemas (SpotifyMetrics.tracks/releases, migrate smartRollupService.posts read to ScheduledPost)” status: pending
id: phase-1-constants content: Create canonical metric constants (packages/server/constants/metrics.js) status: pending
id: phase-1-normalizer content: Build MetricsNormalizer with per-platform title maps (covers EVO-3322 + EVO-3323) status: pending
id: phase-1-models content: Create MetricPoint time series model (demographics stored as namespaced MetricPoints) status: pending
id: phase-1-repository content: Build SocialMetricsRepository with read + write methods against new models status: pending
id: phase-2-write-migration content: Refactor all 16 genesis-web write sites to use repository (dual-write to old + new), retire tiktokCronService.js status: pending
id: phase-2-read-migration content: Refactor 3 chart helpers, 3 overview helpers, and smart rollup to use repository reads status: pending
id: phase-2-data-backfill content: Migration script to backfill MetricPoint from existing collections status: pending
id: phase-3-rest-api content: Build REST API routes, controllers, validation, auth middleware, mount on Express status: pending
id: phase-4-frontend content: Migrate frontend hooks to call new REST API instead of existing endpoints status: pending
id: phase-5-lambdas content: Add service auth, migrate Lambda read/write operations to new REST API status: pending
id: phase-6-ai-agent content: Give AI agent direct REST API access for ad-hoc metrics queries and custom dashboards status: pending
id: future-post-metrics content: “Future iteration: unify ScheduledPost per-post metrics with the analytics layer” status: pending isProject: false

Unified Metrics Service Layer (v4)

v4 changes from v3: Removed ContentSnapshot collection after audit revealed it was designed around dead/deprecated fields. Demographics stored as namespaced MetricPoint documents instead. Added tiktokCronService.js as missing write site (16 total, not 15) with explicit retirement plan. Added schema cleanup pre-work (dead fields in SpotifyMetrics, InstagramMetrics). Corrected || 0 scope estimate from 30+ to ~80. Documented audit corrections.

v3 changes from v2: REST API instead of GraphQL (team consensus); clarified ContentSnapshot vs ScheduledPost relationship; added ScheduledPost / Content Studio / Top Performing Posts impact analysis; added AI service impact analysis; added incremental migration strategy with documented callsites; added consumer callsite inventory for divided work.

Design Priorities

AI accuracy — the AI assistant must be able to query any metric with precision; wrong data = wrong decisions
Visualization quality — charts should be fed by simple, indexed queries, not fragile in-app array scanning
Future extensibility — per-metric detail charts, custom AI dashboards, new platforms with zero code changes at the query layer

Every design choice below is grounded in these three priorities.

Why This Change Is Necessary

The current architecture allows consumers of the same data to silently diverge. A chart that needs the last week of Instagram data for display and the AI service needing the same week of Instagram data to give a thoughtful response should make the same query and get the same result. Today, they don’t — and the bugs that result are not mistakes any individual developer made, but structural problems that the codebase makes inevitable.

The Core Problem: No Shared Query Path

There are 20+ callsites across genesis-web and genesis-aws-services that each independently query the same metrics collections, each writing their own MongoDB queries, title string lookups, and date handling. When two callsites need the same data but construct their queries differently, they get different numbers — and nobody knows until a user notices.

Concrete example — date boundary divergence:

The Instagram chart helper and smartRollupService both query InstagramMetricsModel for a date range. But only the chart helper normalizes the end date:

javascript

// instagramChartsHelper.js — includes the full end day
endDateObj.setHours(23, 59, 59, 999);
const instagramData = await InstagramMetricsModel.find({ user: userId, date: { $gte: startDateObj, $lte: endDateObj } });

// smartRollupService.js — does NOT normalize, potentially misses the last day
const instagramData = await InstagramMetricsModel.find({ user: userId, date: { $gte: startDate, $lte: endDate } });

The fix here is trivial — add .setHours() to the rollup query. But the real problem is that it’s even possible for this divergence to exist. The developer writing smartRollupService shouldn’t have needed to know about .setHours(). And going further — even if both callers used a shared repository method like repository.getTimeSeries(userId, 'instagram', 'Total Followers', startDate, endDate), they could still diverge if they compute startDate and endDate differently (timezone handling, inclusive vs exclusive end dates, what “last week” means).

The right abstraction is to not make the caller deal with dates at all for common cases. A call like repository.getTimeSeries(userId, 'instagram', 'Total Followers', { period: 'week' }) delegates all date math to the server — which knows the user’s timezone from their profile and handles UTC conversion, end-of-day normalization, and boundary logic in one place. Every consumer that asks for “last week” gets exactly the same date range.

Concrete example — title string casing:

javascript

// smartRollupService.js — lowercase 'f', silently returns undefined → falls back to 0
const currentFollowers = mostRecent.metrics?.find(m => m.title === 'Total followers')?.value || 0;

// instagramChartsHelper.js — capital 'F', matches what's stored in the DB
const followerMetric = metrics.find(metric => metric?.title === 'Total Followers');

// overviewHelper.js — uses constants, always correct
const followerMetric = metrics.find(m => m.title === INSTAGRAM_METRIC_TITLES.TOTAL_FOLLOWERS);

Three callsites, three approaches to finding the same metric, two different results. The constants exist (INSTAGRAM_METRIC_TITLES.TOTAL_FOLLOWERS) but nothing forces their use. A repository method resolves the canonical title internally — callers can’t get it wrong.

Concrete example — TikTok data model divergence:

TikTok metrics exist in two formats depending on how they were written:

javascript

// smartRollupService.js — expects per-day documents
const currentMetrics = await TikTokMetricsModel.find({ user: userId, date: { $gte, $lte } });

// tiktokChartHelper.js — expects a single document with a history array
const tiktokData = await TikTokMetricsModel.findOne({ user: userId }).lean();
// then reads from tiktokData.history

If a user’s data is in the history format, the rollup query returns nothing. The chart works fine. Same collection, same user, different results.

What We Lose by Not Changing

Without a shared query layer, every new feature that reads metrics will:

Need to know which of 3 collections to query (or that TikTok has two data formats)
Need to hardcode the correct title strings (and hope they match what’s stored)
Need to remember date normalization patterns
Need to be manually checked against every other consumer to ensure consistency

The Lambdas in genesis-aws-services have the same problem — they query the same collections with their own Mongoose calls, introducing another axis of potential divergence.

Current Schema Structure (for reference)

javascript

// One document per user per day (Instagram and Spotify)
{
  user: ObjectId,
  date: Date,
  metrics: [{ title: String, value: Number }],  // 5-10 items per day
  demographics: { gender: [...], country: [...], ... },
  posts: [...]  // or tracks/videos
}

// TikTok — mixed: some users have per-day docs, some have a single doc with history array

Additional structural problems:

No compound index on { user, date } on any metrics model. Chart queries do full collection scans.
To get one metric’s history, the DB returns ALL metrics + demographics + posts for every day, then app code scans the metrics array by title.
Three separate collections with no shared interface means every cross-platform comparison requires three queries and manual merging.
Per-metric detail charts require the same “fetch everything, scan array” pattern — no way to ask the DB for just “Profile Views over time.”
AI-constructed dashboards need to issue ad-hoc queries. Today that means knowing which of 3 collections to query, which title string variant to use, and how to post-process the results.

Scope Boundary: Account-Level Metrics vs Post-Level Metrics

This plan covers account-level metrics — daily platform-wide numbers like Total Followers, Profile Views, Impressions, etc. that are currently stored across InstagramMetricsModel, SpotifyMetricsModel, and TikTokMetricsModel.

Post-level metrics (likes, comments, shares, engagement_rate per individual post) live on the ScheduledPost model in platformStatus.<platform>.metrics and are a separate concern that will be addressed in a future iteration.

Why This Distinction Matters

The ScheduledPost collection serves a dual purpose:

Content management — media assets, captions, scheduling, platform status, publish state
Post-level performance — per-post engagement metrics (likes, comments, shares, views, engagement_rate) written by daily cron jobs and used by Content Studio, Top Performing Posts, and Campaign analytics

ScheduledPost retains ownership of individual post metrics because:

Post metrics are tightly coupled to content management (media, captions, scheduling)
They’re updated on different cadences (per-post insight fetching with exponential backoff vs daily account snapshots)
They serve different consumers (Content Studio filtering/sorting vs account-level dashboard charts)
The ScheduledPost model has complex middleware (campaign metrics sync, soft delete, TikTok validation) that shouldn’t be disrupted

ScheduledPost Consumers (Current State)

These services read or write ScheduledPost.platformStatus.<platform>.metrics and are out of scope for this plan but documented here for the future iteration:

Service	File	What It Does
Content Studio API	`schedulePostService.js` → `getPostsService`	Reads `engagement_rate` for `isTopPerformingPost` flag and performance sorting
Single Post API	`schedulePostService.js` → `getScheduledPostService`	Same `isTopPerformingPost` logic for individual post view
Top Performing Detection	`postPerformanceService.js` → `checkPostPerformance`	Reads `engagement_rate` to compare against baseline thresholds
Campaign Analytics	`postTrackingService.js`	Aggregation pipeline sums `impressions`, `views`, `likes`, `comments`, `shares`, `saves`, `reach` across posts
Instagram Overview	`overviewHelper.js` → `computeEngagementRateForPeriod`	Reads `engagement_rate` from ScheduledPost for period-level engagement
Smart Rollup (AI)	`smartRollupContentService.js` → `getContentStudioData`	Fetches top 5 posts sorted by engagement for AI context
IG Daily Cron	`InstagramDailyScheduler.js`	Writes `platformStatus.instagram.metrics` from Instagram Graph API insights
TikTok Daily Cron	`TikTokDailyScheduler.js`	Writes `platformStatus.tiktok.metrics` from TikTok API
External Post Sync	`externalPostsService.js`	Writes `platformStatus.instagram.metrics` during external post ingestion
Post Metrics Update	`postTrackingService.js` → `updatePostMetrics`	Writes metrics from campaign flows

Known bug: smartRollupContentService.js sorts by platformStatus.instagram.metrics.engagement but the model defines engagement_rate. This causes incorrect sorting/values for AI context.

Future Iteration: Unified Post Metrics

In a future iteration, post-level metrics could be extracted from ScheduledPost into a dedicated collection (e.g., PostMetricPoint) that follows the same patterns as MetricPoint. This would:

Allow Content Studio and Top Performing Posts to query post metrics independently of content management
Enable cross-platform post performance comparisons through a consistent interface
Reduce the size and complexity of the ScheduledPost document

This is explicitly deferred — the account-level metrics unification is the priority. Once that’s stable and proven, the same patterns can be applied to post-level metrics incrementally without disrupting existing functionality.

Target Schema: MetricPoint (Single Unified Collection)

MetricPoint (MongoDB Time Series Collection)

One document per user/platform/metric/date. The DB does the work.

javascript

const MetricPointSchema = new Schema({
  date: { type: Date, required: true },
  meta: {
    user: { type: Schema.Types.ObjectId, ref: 'User', required: true },
    platform: { type: String, enum: ['instagram', 'spotify', 'tiktok'], required: true },
    metric: { type: String, required: true }  // canonical title from constants
  },
  value: { type: Number, required: true }
}, {
  timeseries: {
    timeField: 'date',
    metaField: 'meta',
    granularity: 'hours'
  }
});

What this enables:

javascript

// "What were my Instagram followers each day this month?"
MetricPoint.find({
  'meta.user': userId,
  'meta.platform': 'instagram',
  'meta.metric': 'Total Followers',
  date: { $gte: startDate, $lte: endDate }
}).sort({ date: 1 });
// One indexed query. No in-app scanning. No title string guessing.

// "Compare followers across all platforms"
MetricPoint.find({
  'meta.user': userId,
  'meta.metric': 'Total Followers',
  date: { $gte: startDate, $lte: endDate }
}).sort({ date: 1 });
// One query, group by meta.platform in results.

// "What metrics are available for TikTok?"
MetricPoint.distinct('meta.metric', {
  'meta.user': userId,
  'meta.platform': 'tiktok'
});

Why Time Series over regular collection:

MongoDB Time Series collections use columnar compression — ~10x smaller on disk for numeric time series data
Optimized for range scans on the timeField
Built-in bucketing for weekly/monthly aggregation via $densify and $setWindowFields
Mongoose 7.1.1 (current version) supports timeseries schema option natively
Requires MongoDB 5.0+ (check Atlas cluster version — likely already 6.0+ or 7.0+)

Fallback: If the MongoDB instance is below 5.0, the same document shape works as a regular collection with a compound index { 'meta.user': 1, 'meta.platform': 1, 'meta.metric': 1, date: 1 }. The repository abstracts this — consumers don’t know or care.

Demographics as Namespaced MetricPoints

Demographics data (gender, country, city, age breakdowns) is stored as individual MetricPoint documents with namespaced metric keys:

javascript

// Each demographic dimension becomes a MetricPoint
{ meta: { platform: 'instagram', metric: 'Demographics:Country:US' }, value: 34.2, date: ... }
{ meta: { platform: 'instagram', metric: 'Demographics:Country:UK' }, value: 12.1, date: ... }
{ meta: { platform: 'instagram', metric: 'Demographics:Gender:Female' }, value: 58.0, date: ... }
{ meta: { platform: 'instagram', metric: 'Demographics:Age:25-34' }, value: 42.3, date: ... }
{ meta: { platform: 'spotify', metric: 'Demographics:City:Los Angeles' }, value: 1250, date: ... }

Why not a separate collection? A ContentSnapshot collection was originally proposed for demographics + account-level content summaries. An audit revealed this was overbuilt:

The posts/tracks/videos arrays the collection was designed to store are dead or deprecated in the current schemas (IG posts is only read by one legacy path in smartRollupService; Spotify tracks/releases are deprecated and always empty; TikTok videos is redundant with ScheduledPost)
Demographics are only ever consumed as “give me the latest snapshot” — no code queries demographic history across dates
Storing demographics as MetricPoints keeps the data model uniform and enables future demographic trend charting with zero additional work

Repository interface for demographics:

javascript

// Write: the repository receives the demographics object and flattens it
repository.saveDemographics(userId, 'instagram', {
  gender: [{ label: 'Female', value: 58.0 }, ...],
  country: [{ label: 'US', value: 34.2 }, ...],
  // ...
});
// Internally writes one MetricPoint per dimension per label

// Read: the repository reassembles the demographics object
const demographics = await repository.getDemographics(userId, 'instagram');
// Returns: { gender: [...], country: [...], city: [...], age: [...] }
// Consumers never see the flattened storage format

Why `ContentSnapshot` Was Removed (v4)

The v3 plan included a ContentSnapshot collection to store demographics and the posts/tracks/videos arrays from platform metrics documents. A codebase audit revealed:

**InstagramMetrics.posts** — still read by smartRollupService.js (lines 305-317) for AI top posts context, but gamification code explicitly uses ScheduledPost instead. The smartRollupService read should be migrated to ScheduledPost as pre-work (Phase 0).
**SpotifyMetrics.tracks** — deprecated. New writes use tracks: []. Data migrated to TrackModel. smartRollupService has a vestigial fallback read that should be removed.
**SpotifyMetrics.releases** — deprecated. Same pattern as tracks. Data migrated to ReleaseModel.
**TikTokMetrics.videos** — still written by tiktokService.writeUserMetrics() and read by overviewHelper.aggregateVideoMetrics(), but redundant with ScheduledPost.platformStatus.tiktok.metrics. Will be migrated in Phase 2.
**TikTokMetrics.history** — still written by tiktokCronService.js and read by overviewHelper. Will be retired when tiktokCronService is removed in Phase 2.

Building a new collection around these fields would have enshrined dead data in a new schema. The right approach is to clean up the dead fields (Phase 0) and store only demographics — which fits naturally in MetricPoint.

What Happens to the Old Collections

The existing InstagramMetricsModel, SpotifyMetricsModel, TikTokMetricsModel stay in place during migration. The repository writes to both old and new collections during a transition period. Once all consumers are migrated and data is verified, the old collections become read-only archives.

SocialModel stays as-is — it stores connection state (isLinked, authorized, tokens) that is separate from metrics. The audienceCount field continues to be updated by the repository from the latest Total Followers MetricPoint.

ScheduledPost stays as-is — it stores content management data and per-post engagement metrics. See the Scope Boundary section above for details on future unification.

Architecture

Why REST (Not GraphQL) — For Now

The team agreed to start with a flexible REST API rather than introducing GraphQL. The rationale:

No existing GraphQL in the system — introducing it for a single service adds operational overhead (new tooling, learning curve, debugging patterns) without proportional benefit
REST can be equally flexible — parameterized endpoints with rich query parameters achieve the same filtering/aggregation capabilities. The Task API already demonstrates this pattern in the codebase.
Lower barrier for the team — everyone on the team already knows REST; the focus should be on cleaning up the data layer, not learning a new API paradigm
Arti can build REST queries dynamically too — the AI agent doesn’t need GraphQL introspection to construct flexible queries; well-documented REST endpoints with consistent parameter patterns work just as well
GraphQL remains a future option — if the system grows to a point where multiple services would benefit from GraphQL (not just analytics), it can be layered on top of the repository without changing the data layer

Design Decisions the Service Must Own

These are decisions that, if left to individual callers, produce the exact class of silent divergence bugs this plan exists to eliminate. The service handles all three consistently so callers can’t get them wrong.

1. Date Handling

Dates are the most common source of subtle bugs. The API eliminates this by making named periods the primary interface and handling all date/timezone logic server-side:

Named periods (period=week|month|quarter|ytd) — the server computes the actual date range using the authenticated user’s timezone from their profile. “Last week” means the same thing for every consumer, every time.
Custom date ranges (startDate=2026-01-15&endDate=2026-01-31) — accepted as plain YYYY-MM-DD local calendar dates, never timestamps. The server resolves “January 31st” to “end of January 31st in the user’s timezone, converted to UTC.” The caller never constructs Date objects, never calls .setHours(), never decides if endDate is inclusive or exclusive.
Timezone source — always the user’s profile. No timezone query param for normal use. The server already knows the user’s timezone (required for scheduling features). One source of truth.

The repository layer exposes the same interface internally: repository.getTimeSeries(userId, 'instagram', 'Total Followers', { period: 'week' }). Internal consumers (smartRollupService, chart helpers, overview helpers) use the same date resolution path as external API callers.

2. Missing Data Gaps

Crons fail, platform APIs go down, a user links mid-week. When a day has no data point, different callers handle it differently today — some show a gap, some fill with zero, some silently skip the day. If the chart shows a blank day and the AI interprets that as “followers dropped to zero on Tuesday,” the user gets a wrong and alarming insight from something that was just a missing cron run.

The service owns gap handling with a consistent policy:

Time series responses include every day in the requested range. If a day has no data, its value is null — not 0, not omitted. This makes the gap visible and explicit.
**null means “no data was collected”** — callers can distinguish between “the metric was zero” and “we don’t have data for this day.”
An optional fillGaps=lastKnown parameter allows callers to request forward-fill behavior (carry the last known value through gaps). This is useful for cumulative metrics like Total Followers where a missing day almost certainly means the value didn’t change, not that it became unknown. The fill logic lives in the service — callers don’t implement it independently.
The response includes a gapCount field when gaps exist, so callers (especially the AI) can note data quality without needing to scan the array.

3. Growth / Comparison Periods

The AI rollup and the overview dashboard both show things like “followers up 5% this week.” Computing this requires comparing the current period to a previous period. If the service only returns raw values and each caller computes the delta independently, they diverge on: which previous period to compare against, whether to use percentage vs absolute difference, start-of-period vs end-of-period values, and what to do when there’s no data for the comparison period.

The service handles this natively:

**compare=previous** — when used with a named period, the service returns both the current and previous period data in one response. period=week&compare=previous returns this week and last week. The comparison range is always the same length as the current range, immediately preceding it.
Growth is computed server-side — the response includes growth: { absolute: 250, percentage: 1.8 } alongside the raw series data. Every consumer that displays “up 1.8% this week” shows the same number because the same code computed it.
Edge cases handled once — if the comparison period has no data (user just linked), the service returns growth: null rather than forcing callers to decide what to do. No division by zero, no “∞% growth” from different fallback strategies.

REST API Design

Base path: /api/v1/analytics

Read endpoints:

Endpoint	Purpose	Key Parameters
`GET /time-series`	Time series for one metric on one or all platforms	`platform`, `metric`, `period` OR `startDate`+`endDate` (YYYY-MM-DD), `compare` (previous), `fillGaps` (lastKnown), `view` (raw/pct), `granularity` (daily/weekly/monthly)
`GET /compare`	Cross-platform comparison for one metric	`metric`, `period` OR `startDate`+`endDate`, `compare` (previous), `view`
`GET /latest`	Latest values for all metrics on a platform	`platform`
`GET /snapshot`	Single metric latest value	`platform`, `metric`
`GET /demographics`	Audience demographics (latest snapshot)	`platform`
`GET /available-metrics`	Available metrics for a platform (for AI introspection)	`platform`
`GET /connected-platforms`	Available platforms for a user	—

Write endpoints (for Lambdas / service-to-service):

Endpoint	Purpose	Body
`POST /metrics`	Save metrics for a user/platform/date	`{ platform, date, metrics: [{ metric, value }] }`
`POST /metrics/bulk`	Bulk save across platforms/dates	`{ entries: [{ platform, date, metrics }] }`

Validation: All query parameters validated via middleware. Invalid platform values return 400 with the list of valid options (no silent empty results). Metric names validated against canonical constants. period and startDate/endDate are mutually exclusive — passing both returns 400.

Auth: Same JWT validation as existing isLogged middleware for user-facing calls. API key auth (header-based) for service-to-service calls from Lambdas.

Response format (consistent across all endpoints):

javascript

// GET /api/v1/analytics/time-series?platform=instagram&metric=Total Followers&period=week&compare=previous
{
  "data": {
    "platform": "instagram",
    "metric": "Total Followers",
    "granularity": "daily",
    "period": "week",
    "resolvedRange": { "start": "2026-03-09", "end": "2026-03-16" },
    "series": [
      { "date": "2026-03-09", "value": 13950 },
      { "date": "2026-03-10", "value": 14010 },
      { "date": "2026-03-11", "value": null },   // cron didn't run — explicit gap
      { "date": "2026-03-12", "value": 14085 },
      // ...
    ],
    "gapCount": 1,
    "comparison": {
      "resolvedRange": { "start": "2026-03-02", "end": "2026-03-09" },
      "series": [
        { "date": "2026-03-02", "value": 13700 },
        // ...
      ]
    },
    "growth": { "absolute": 250, "percentage": 1.8 }
  }
}

// GET /api/v1/analytics/time-series?platform=instagram&metric=Total Followers&period=week&fillGaps=lastKnown
// Same as above, but null gaps are filled with the last known value:
//   { "date": "2026-03-11", "value": 14010, "filled": true }

// GET /api/v1/analytics/latest?platform=instagram
{
  "data": [
    { "title": "Total Followers", "value": 14200, "date": "2026-03-15" },
    { "title": "Profile Views", "value": 892, "date": "2026-03-15" },
    // ...
  ]
}

// GET /api/v1/analytics/available-metrics?platform=instagram
{
  "data": [
    { "name": "Total Followers", "description": "Total follower count" },
    { "name": "Profile Views", "description": "Daily profile views" },
    // ...
  ]
}

The resolvedRange field lets callers (especially the AI agent) see exactly what date range was used. The growth field ensures every consumer showing “up 1.8% this week” computes it the same way. The gapCount and filled fields make data quality transparent without requiring callers to scan the array.

AI Service Impact

Current Data Flow

The AI service (genesis-train / Arti) does not query MongoDB directly. It receives pre-aggregated data via moduleContext:

genesis-web (smartRollupService) → MongoDB queries → aggregation in Node → moduleContext JSON → POST /chat/stream → genesis-train

What smartRollupService currently queries:

Collection	Model	Data Retrieved
`spotifymetrics`	`SpotifyMetricsModel`	Listeners, followers, streams, demographics, top tracks
`instagrammetrics`	`InstagramMetricsModel`	Followers, likes, comments, shares, saves, reach, impressions
`tiktokmetrics`	`TikTokMetricsModel`	Followers, views, likes, comments, shares
`tracks`	`TrackModel`	Total streams, top tracks by `sp_streams`
`socials`	`SocialModel`	Connected platforms

Additionally, smartRollupContentService.js queries ScheduledPost for top 5 posts by engagement (Content Studio data for AI context).

What Changes

Phase 2 (internal migration): smartRollupService switches from querying legacy models to calling SocialMetricsRepository. The moduleContext shape stays the same — genesis-train sees no change.
Phase 6 (optional): The AI service could be given direct REST API access to construct ad-hoc queries, bypassing the pre-aggregated moduleContext flow. This enables “compare my Instagram engagement to Spotify streams over Q1” without genesis-web having to anticipate every possible query shape.
ScheduledPost queries in smartRollupContentService are unaffected — post-level metrics are out of scope for this plan.

genesis-aws-services Lambda Impact

These Lambdas query metrics collections directly via Mongoose and will need migration in Phase 5:

Lambda	Collections	Files
`instagram-daily-scheduler`	`instagrammetrics`	`saveOverViewHelper.js`, `instagramOverviewHelper.js`, `gamification/ig-engagement/*.js`
`chartmetrics-daily-scheduler`	`spotifymetrics`	`gamification/sp-streams/viral-track.js`, `track-consistency.js`, `multi-track-momentum.js`
`weekly-digest-email`	`spotifymetrics`, `instagrammetrics`	`spotifyReleasesHelper.js`, `instagramPostsHelper.js`

Migration approach: New REST API endpoints are introduced alongside existing routes — nothing breaks. Lambdas can be migrated one by one to call the new API. Each migration is independently testable by comparing the numbers from old vs new path.

Incremental Migration Strategy

Per team discussion, the migration follows an additive, non-breaking approach:

New endpoints run alongside old ones — no existing endpoint is removed or modified until the replacement is verified
Dual-write during transition — the repository writes to both old and new collections so both paths return consistent data
Independent consumer migration — once the repository and API are ready, each consumer (frontend, Lambda, AI service) can be migrated independently by different team members
Manual verification at each step — when updating a consumer, verify that the numbers from the new path match expected values before removing the old code path

Breaking Change Gates

The dual-write period is the safety net. Old collections cannot be archived until ALL consumers are migrated. This is the migration gate checklist:

Gate	What must be true before old collections go read-only	Phase
Internal readers	All chart helpers, overview helpers, and `smartRollupService` read from repository	Phase 2
Internal writers	All cron schedulers, link flows, and overview helpers write through repository	Phase 2
Frontend	`useSocialProgressChartApi` and `useSocialAnalyticsApi` call new REST API endpoints	Phase 4
Lambdas	All 4 Lambda groups call REST API instead of direct Mongoose	Phase 5
AI service	`smartRollupService` uses repository reads (Phase 2); optional direct REST access (Phase 6)	Phase 2/6

No gate can be skipped. The smartRollupService / deprecated-field cleanup was missed during the Spotify tracks/releases migration because only UI-facing consumers were considered. Each gate should have an explicit verification step confirming zero remaining direct Mongoose queries against the old collections.

Lesson learned: When deprecating a field or collection, search for ALL consumers — not just the ones that feed UI. smartRollupService was missed during the Spotify tracks migration because it feeds the AI service, not a visible UI component. The migration gate checklist above is designed to prevent this class of oversight.

Consumer Migration Callsite Inventory

This inventory enables the team to divide migration work. Each item can be a separate subtask:

genesis-web internal readers (Phase 2):

Breaking change boundary: These are internal to genesis-web. The old code paths must be replaced, not just supplemented — once the repository is ready, each reader is migrated individually. Each is a separate subtask.

Callsite	Current Source	New Source	Owner
Chart helpers (3 platform + 1 orchestrator)	Direct Mongoose queries to platform models	`repository.getTimeSeries({ period })` or `repository.getTimeSeries({ startDate, endDate })`	TBD
`overviewHelper` (IG/Spotify/TikTok) — metrics	Direct Mongoose queries	`repository.getLatestMetrics()`	TBD
`overviewHelper` (IG/Spotify/TikTok) — demographics	Direct Mongoose queries	`repository.getDemographics()`	TBD
`smartRollupService.getStreamingAnalyticsData`	Direct Mongoose queries + manual date math	`repository.getTimeSeries({ period: 'week' })` + `repository.getLatestMetrics()`	TBD
`smartRollupService` — Spotify demographics	`SpotifyMetricsModel` `.demographics.country`	`repository.getDemographics(userId, 'spotify')` — only `country` dimension used for AI `top_regions`	TBD

genesis-web internal writers (Phase 2):

Callsite	Platform	Current Target	New Target
`InstagramDailyScheduler`	Instagram	`InstagramMetricsModel`	`repository.saveMetrics()` (dual-write)
`instagramService`	Instagram	`InstagramMetricsModel`	`repository.saveMetrics()` (dual-write)
`overviewHelper` (IG)	Instagram	`InstagramMetricsModel`	`repository.saveMetrics()` (dual-write)
`SongstatsDailyScheduler`	Spotify	`SpotifyMetricsModel`	`repository.saveMetrics()` (dual-write)
`spotifyService`	Spotify	`SpotifyMetricsModel`	`repository.saveMetrics()` (dual-write)
`songstatsArtistMetricsHelper`	Spotify	`SpotifyMetricsModel`	`repository.saveMetrics()` (dual-write)
`ChartMetricsDailyScheduler`	Spotify	`SpotifyMetricsModel`	`repository.saveMetrics()` (dual-write)
`overviewHelper` (Spotify)	Spotify	`SpotifyMetricsModel`	`repository.saveMetrics()` (dual-write)
`diagnostics`	Spotify	`SpotifyMetricsModel`	`repository.saveMetrics()` (dual-write)
`TikTokDailyScheduler`	TikTok	`TikTokMetricsModel`	`repository.saveMetrics()` (dual-write)
`tiktokService` (×2 paths)	TikTok	`TikTokMetricsModel`	`repository.saveMetrics()` (dual-write)
`tiktokCronService`	TikTok	`TikTokMetricsModel`	RETIRE — zombie scheduler, writes `history` format
`overviewHelper` (TikTok)	TikTok	`TikTokMetricsModel`	`repository.saveMetrics()` (dual-write)

Frontend (Phase 4):

Breaking change boundary: Once the old overview/chart endpoints are retired (after dual-running period), all frontend hooks below must be migrated or they will break. Each hook is a separate subtask. useUserSocials reads audienceCount from SocialModel via the profile API — this is NOT affected since audienceCount continues to be updated by the repository.

Hook/Component	Current Endpoint	New Endpoint	Consumers
`useSocialProgressChartApi`	`GET /charts/progress-chart`	`GET /api/v1/analytics/time-series?period=week` (or user-selected range)	`FollowersGrowthChart`, `AnalyticsSection`
`useSocialAnalyticsApi`	`GET /social/overview/:slug`	`GET /api/v1/analytics/latest` + `GET /api/v1/analytics/demographics`	`MetricsGrid`, `LocationList`, `TopContentTable`
`useSocialAnalytics`	Composes both hooks above	Composes new hooks	`AnalyticsSection` (main orchestrator)

Frontend cleanup (can be done in Phase 4):

socialApi.spotify.analytics, socialApi.instagram.analytics, socialApi.tiktok.analytics — defined in socialApi.ts but unused; remove
useSocial → socialApi.getOverview — calls GET /social/overview but response is commented out; remove or migrate

genesis-aws-services Lambdas (Phase 5):

Breaking change boundary: Lambdas currently import Mongoose models directly and query MongoDB. Once the old collections are archived (after dual-write period ends), any Lambda not migrated to the REST API will fail silently or return empty results. Each Lambda migration is a separate subtask.

Lambda	Current Access	New Access
`instagram-daily-scheduler` (overview helpers)	Direct `InstagramMetricsModel`	`POST /api/v1/analytics/metrics` + `GET /api/v1/analytics/latest`
`instagram-daily-scheduler` (gamification)	Direct `InstagramMetricsModel`	`GET /api/v1/analytics/time-series?period=week` (or as needed)
`chartmetrics-daily-scheduler` (gamification)	Direct `SpotifyMetricsModel`	`GET /api/v1/analytics/time-series?period=week` (or as needed)
`weekly-digest-email`	Direct `SpotifyMetricsModel` + `InstagramMetricsModel`	`GET /api/v1/analytics/time-series?period=week`

Phased Implementation

Phase 0: Pre-work — Verify MongoDB + Schema Cleanup (1-2 days)

MongoDB version: Check the Atlas cluster MongoDB version to confirm Time Series collection support (5.0+). If below 5.0, fall back to regular collection with compound index.

Schema cleanup (removes misleading dead fields):

Remove tracks and releases from SpotifyMetrics schema — data already migrated to TrackModel/ReleaseModel, new writes already use []
- Update SongstatsDailyScheduler.js, ChartMetricsDailyScheduler.js, spotifyService.js to stop writing tracks: [] / releases: []
- Update smartRollupService.js to stop reading mostRecent.tracks (use TrackModel directly)
- Update spotifyService.js ChartMetrics path to stop reading todayRecord.tracks/todayRecord.releases
Migrate smartRollupService.js top-posts read (lines 305-317) from InstagramMetrics.posts to ScheduledPost — aligns with what all gamification code already does
Document that TikTokMetrics.history and TikTokMetrics.videos will be retired in Phase 2 when tiktokCronService.js is removed

Why this matters: Dead fields in schemas cause future developers (and AI agents) to assume functionality that doesn’t exist, leading to wasted effort or incorrect designs. The v3 plan’s ContentSnapshot collection was designed around these dead fields — catching them now prevents building infrastructure around abandoned data.

Phase 1: Foundation (4-5 days)

Canonical metric constants (packages/server/constants/metrics.js)
MetricsNormalizer with per-platform title maps (packages/server/utils/metricsNormalizer.js)
MetricPoint time series model (packages/server/models/MetricPoint.js) — stores both metrics and demographics as namespaced keys
SocialMetricsRepository read + write methods (packages/server/repositories/socialMetricsRepository.js)
- saveDemographics() — flattens demographics object into namespaced MetricPoints
- getDemographics() — reassembles flattened MetricPoints into demographics object
Unit tests for normalizer and repository
Covers EVO-3322 (title normalization) and EVO-3323 (schema alignment)

Phase 2: Internal Consumer Migration (6-8 days)

Write path (dual-write to old + new collections):

Instagram: 3 write sites in genesis-web (InstagramDailyScheduler, instagramService, overviewHelper)
Spotify: 8 write sites (SongstatsDailyScheduler, spotifyService, songstatsArtistMetricsHelper, ChartMetricsDailyScheduler, overviewHelper, diagnostics)
TikTok: 5 write sites (TikTokDailyScheduler, tiktokService ×2, tiktokCronService, overviewHelper)
- Retire tiktokCronService.js — this zombie scheduler runs at the same time as TikTokDailyScheduler (both 15 21 * * *) and writes to the history array format via $push. TikTokDailyScheduler writes per-day documents with metrics arrays. Having two schedulers writing two different formats is the root cause of the history vs metrics data model divergence. Remove tiktokCronService from services/cron/index.js and make TikTokDailyScheduler the single TikTok writer through the repository.

Read path:

Replace 3 platform chart helpers (instagramChartsHelper, spotifyChartsHelper, tiktokChartHelper) with repository.getTimeSeries calls — socialChartsHelper is an orchestrator that delegates to these three and will follow automatically
Replace 3 platform overview helpers with repository.getLatestMetrics + repository.getDemographics
Replace smartRollupService DB queries with repository reads

Migration script:

Backfill MetricPoint from existing InstagramMetricsModel, SpotifyMetricsModel, TikTokMetricsModel documents (both metrics arrays and demographics objects)
Verify data integrity: count documents, spot-check values

Phase 3: REST API (2-3 days)

Route definitions (packages/server/routes/analytics/index.js)
Controller functions backed by SocialMetricsRepository (packages/server/controllers/analytics/)
Query parameter validation middleware (platform enum, date format, metric name against constants)
Auth middleware (JWT for user-facing, API key for service-to-service)
Mount at /api/v1/analytics in packages/server/app.js
Consistent error responses with helpful messages (invalid platform → list valid options)

Phase 4: Frontend Migration (3-4 days)

Update useSocialProgressChartApi to call GET /api/v1/analytics/time-series instead of GET /charts/progress-chart
Update useSocialAnalyticsApi to call GET /api/v1/analytics/latest instead of GET /social/overview/:slug
SmartRollupButton can pass the REST API endpoint info to the AI service for direct querying
Keep existing REST endpoints working (no breaking changes for anything not yet migrated)

Phase 5: Lambda Migration + Service Auth (5-7 days)

Add API key auth to analytics endpoints (alongside JWT) for service-to-service calls
Migrate instagram-daily-scheduler Lambda to call REST API for saves and reads
Migrate chartmetrics-daily-scheduler and spotify-sqs-trigger-lambda
Migrate weekly-digest-email to call REST API queries (replaces hardcoded zeros)
Migrate gamification Lambdas to call REST API for reads
Remove dual-write from repository once all Lambdas are migrated
Archive old collections

Phase 6: AI Agent Enhancement (2-3 days)

Give the AI agent the REST API endpoint URLs and parameter documentation
Replace smartRollupService.getStreamingAnalyticsData with the AI agent issuing its own REST API queries based on user context
The AI can now construct ad-hoc queries: “compare my Instagram engagement to Spotify streams over Q1” via parameterized REST calls without any backend changes
Foundation for AI-constructed custom dashboards

Future: Post-Level Metrics Unification

Extract per-post engagement metrics from ScheduledPost.platformStatus.<platform>.metrics into a dedicated collection following MetricPoint patterns
Migrate Content Studio, Top Performing Posts, Campaign Analytics, and smartRollupContentService to use the new collection
Fix the engagement vs engagement_rate bug in smartRollupContentService.js
This iteration can follow the same additive, non-breaking migration pattern established by the account-level work

Testing Strategy

The entire point of this project is that every consumer gets the same numbers for the same question. Testing must verify that property directly, not just that individual functions work in isolation.

Unit Tests (Phase 1)

Test the foundation layer in isolation with known inputs:

MetricsNormalizer — every platform’s title variants resolve to the canonical constant. 'Total followers', 'Total Followers', 'Followers' all resolve to the same canonical key. Unknown titles are rejected, not silently passed through.
Date resolution — { period: 'week' } with a known timezone produces the expected UTC range. YYYY-MM-DD endDate is always inclusive through end-of-day. Timezone edge cases (user near midnight, crossing DST boundaries).
Gap handling — a time series with missing days returns null for those days, not omission. fillGaps=lastKnown forward-fills correctly. gapCount matches.
Growth computation — percentage and absolute growth calculated correctly. null growth when comparison period has no data. No division by zero.

Parity Tests (Phase 2)

The most important test layer. For each consumer migration, verify that the new path produces the same output as the old path for real data:

Dual-read comparison — during migration, the repository can run both the old query and the new query for the same request and compare results. Log discrepancies rather than failing, so they can be investigated without blocking.
Backfill verification — after the migration script runs, compare document counts and spot-check values between old and new collections. Script should produce a verification report (e.g., “Instagram: 45,230 MetricPoints created from 4,523 InstagramMetrics documents, 0 value mismatches”).
Snapshot testing for smartRollupService — capture the current moduleContext output for a set of test users, migrate to repository reads, and assert the output is unchanged. This directly validates that the AI sees the same numbers.

Integration Tests (Phase 3)

Test the REST API endpoints end-to-end:

Parameter validation — invalid platform returns 400 with valid options, not empty results. period and startDate/endDate are mutually exclusive. Metric names validated against constants.
Auth — JWT for user-facing, API key for service-to-service. Unauthenticated requests rejected.
Response shape — resolvedRange present when using named periods. growth present when compare=previous. gapCount present when gaps exist.

Consumer Migration Verification (Phases 4-6)

Each consumer migration is a separate verification point. When migrating a consumer:

Run the old code path and the new code path side by side for the same user/date range
Compare the numbers — they should match (or the new path should be demonstrably more correct, e.g., fixing the 'Total followers' casing bug)
Document any intentional differences (bugs being fixed) vs unexpected differences (regressions)

This is manual for initial migrations but can be scripted for later ones once the comparison pattern is established.

What Tests Prevent

The test suite should make it impossible to reintroduce the bugs that motivated this project:

A test that verifies all known metric title variants resolve to canonical constants (prevents casing bugs)
A test that verifies { period: 'week' } and explicit date range for the same week produce identical query results (prevents date boundary bugs)
A test that verifies null is returned for missing data, never 0 (enforces Data Accuracy Manifesto)
A test that verifies the same repository call from two different callers returns identical results (prevents divergence by construction)

Effort Summary

Phase	Scope	Effort	Dependencies
Phase 0	Verify MongoDB version + schema cleanup	1-2 days	None
Phase 1	Foundation (models, normalizer, repository)	4-5 days	Phase 0
Phase 2	Internal consumer migration + data backfill	6-8 days	Phase 1
Phase 3	REST API	2-3 days	Phase 1
Phase 4	Frontend migration	3-4 days	Phase 3
Phase 5	Lambda migration + service auth	5-7 days	Phase 3
Phase 6	AI agent enhancement	2-3 days	Phase 3

Phases 2 and 3 can run in parallel once Phase 1 is done. Phases 4, 5, and 6 depend on Phase 3. Phase 5 subtasks (per-Lambda) can be divided across team members.

Total: ~24-33 days of work, but parallelizable to ~15-21 calendar days.

Jira Structure

Epic: Unified Metrics Service Layer
- Story: Phase 0 — Verify MongoDB version + schema cleanup (remove dead fields from SpotifyMetrics, migrate smartRollupService.posts read)
- Story: Phase 1 — MetricPoint model (metrics + demographics), MetricsNormalizer, SocialMetricsRepository
  - Folds in EVO-3322 and EVO-3323
- Story: Phase 2 — Internal consumer migration (can split into per-platform subtasks)
  - Subtask: Instagram write path (3 sites)
  - Subtask: Spotify write path (8 sites)
  - Subtask: TikTok write path (5 sites) + retire tiktokCronService.js + history deprecation
  - Subtask: Chart helpers read path (3 platform helpers; socialChartsHelper follows)
  - Subtask: Overview helpers read path (metrics + demographics)
  - Subtask: Smart rollup read path (metrics + Spotify demographics)
  - Subtask: Data backfill migration script
- Story: Phase 3 — REST API (routes, controllers, validation, auth, mount)
- Story: Phase 4 — Frontend migration to new REST API (breaking change gate)
  - Subtask: Migrate useSocialProgressChartApi → GET /api/v1/analytics/time-series
  - Subtask: Migrate useSocialAnalyticsApi → GET /api/v1/analytics/latest + GET /api/v1/analytics/demographics
  - Subtask: Remove dead API code (socialApi.*.analytics, useSocial overview call)
- Story: Phase 5 — Lambda migration + service-to-service auth (breaking change gate)
  - Subtask: instagram-daily-scheduler overview helpers migration
  - Subtask: instagram-daily-scheduler gamification migration
  - Subtask: chartmetrics-daily-scheduler gamification migration
  - Subtask: weekly-digest-email migration
- Story: Phase 6 — AI agent direct REST API access
- Story: Future — ScheduledPost per-post metrics unification (separate iteration)

Unified Metrics Service Layer (v4)

Design Priorities

Why This Change Is Necessary

The Core Problem: No Shared Query Path

What We Lose by Not Changing

Current Schema Structure (for reference)

Scope Boundary: Account-Level Metrics vs Post-Level Metrics

Why This Distinction Matters

ScheduledPost Consumers (Current State)

Future Iteration: Unified Post Metrics

Target Schema: MetricPoint (Single Unified Collection)

MetricPoint (MongoDB Time Series Collection)

Demographics as Namespaced MetricPoints

Why ContentSnapshot Was Removed (v4)

What Happens to the Old Collections

Architecture

Why REST (Not GraphQL) — For Now

Design Decisions the Service Must Own

1. Date Handling

2. Missing Data Gaps

3. Growth / Comparison Periods

REST API Design

AI Service Impact

Current Data Flow

What Changes

genesis-aws-services Lambda Impact

Incremental Migration Strategy

Breaking Change Gates

Consumer Migration Callsite Inventory

Phased Implementation

Phase 0: Pre-work — Verify MongoDB + Schema Cleanup (1-2 days)

Phase 1: Foundation (4-5 days)

Phase 2: Internal Consumer Migration (6-8 days)

Phase 3: REST API (2-3 days)

Phase 4: Frontend Migration (3-4 days)

Phase 5: Lambda Migration + Service Auth (5-7 days)

Phase 6: AI Agent Enhancement (2-3 days)

Future: Post-Level Metrics Unification

Testing Strategy

Unit Tests (Phase 1)

Parity Tests (Phase 2)

Integration Tests (Phase 3)

Consumer Migration Verification (Phases 4-6)

What Tests Prevent

Effort Summary

Jira Structure

Why `ContentSnapshot` Was Removed (v4)