Inspiration Feed V2 — Implementation Overview

Audience: Stakeholders, product owners, clients, and new engineers joining the project. Status: Live in production. Scope: Describes the system as it is built today, not as proposed. Last verified against code: 2026-08-10

wavedrom_timing

What the Inspiration Feed is
The user journey
System at a glance
The four content tiers
How content gets in — the ingestion layer
How content gets understood — the enrichment layer
How content gets ranked — the scoring layer
How the system learns — the feedback loop
Data model
Scheduling and automation
Cost, quality, and safety controls
External dependencies
Configuration reference
Known limitations
Code map

1. What the Inspiration Feed is

The Inspiration Feed is a personalised short-form video discovery feed for music artists inside the platform. It answers one question for the artist:

“What should I be making right now?”

It does this by continuously collecting TikTok and Instagram Reels that are relevant to the artist’s own taste and genre, ranking them, and presenting them as a scrollable feed inside the product.

What makes it different from just browsing TikTok:

	Browsing TikTok directly	Inspiration Feed
Optimised for	Watch time / entertainment	Creative reference and idea generation
Personalisation basis	Consumption behaviour on the platform	The artist’s own genre, followed creators, and explicit saves
Quality floor	None — anything can surface	Engagement filter removes low-performing posts
Cross-platform	Single platform	TikTok and Instagram in one feed
Persistence	Ephemeral	Posts can be saved into the artist’s own board/workspace
Explainability	Opaque	Every item records why it was shown

Design principles the implementation follows

Shared caches, personal feeds. Scraped posts, caption embeddings, audio features, and thumbnail mirrors are stored once globally and reused across all users. Only the assembled feed (InspirationItem) is per-user. This is what makes the system economically viable as the user base grows.
Nothing expensive on the request path. All scraping, embedding, media mirroring, and audio analysis happen in background workers. A user’s feed request is a database read.
Every item is explainable. Each feed item carries a tier, a category, and a sourceMeta block naming the exact signal that produced it.
Fail soft, never blank. Every tier has a fallback. If a tier produces nothing, the feed still returns content rather than an empty screen.

2. The user journey

Key point for stakeholders: steps ① and ② are the only ones requiring user effort. Step ③ runs unattended, and step ④ makes the feed better over time without the user configuring anything.

3. System at a glance

The feature is built as a self-contained module at packages/server/module/inspiration/. It exposes a single public entry point (index.js) and keeps its internals private, so the rest of the application cannot couple to its implementation details.

The two-speed architecture

This is the single most important structural idea in the system.

Because the feed is pre-computed, a user’s scroll never waits on a scraper, an AI model, or a third-party API. It reads rows that were prepared hours earlier.

4. The four content tiers

Every feed is assembled from four independent sources with a fixed budget split. The budget is expressed as a percentage of INSPIRATION_BATCH_SIZE (default 50 posts).

What each tier is for

Tier	Name	Budget	Answers	Primary ranking
1	`creator_direct`	40%	“What did the artists I follow just post?”	Freshness-dominant (70% freshness / 30% base weight)
2	`caption_similar`	25%	“What else looks like the things I’ve saved?”	AI similarity × freshness
3	`creator_expansion`	25%	“What else have those creators posted?”	Engagement (40%) + freshness (60%)
4	`saved_author`	10%	“Who else do I clearly like, based on behaviour?”	Engagement (40%) + freshness (60%)

Why a tier structure at all

A single ranking function would collapse into whichever signal is strongest — in practice, always the followed creators. The tier budget guarantees that a fixed share of the feed is exploratory (Tier 2) and behavioural (Tier 4), so the feed cannot become a pure “recent posts from followed accounts” list.

Interleaving on read

Tiers are not returned as blocks. On every page request the system:

Allocates each tier a per-page slot proportional to its budget.
Paginates each tier independently at its own rate.
Round-robin interleaves the results so page 1 contains a mix of all four tiers.
Trims to the requested page size.
Tops up from any remaining items if some tiers ran dry.

The result: an artist always sees variety on the first screen, never twenty consecutive posts from one source.

5. How content gets in — the ingestion layer

All external content arrives through Apify, a managed web-scraping platform. The system uses seven distinct scrapers.

Concurrency safety: the in-flight lock

Multiple pipeline runs can overlap — a cron run and a user-triggered refresh, for example. Without protection, both would launch the same paid Apify scrape.

Every scrapeable entity (a hashtag, a creator, a saved post) carries two timestamps:

resolvingAt — set the moment a scrape is claimed. Any other process seeing a resolvingAt newer than 10 minutes skips that entity entirely.
resolvedAt — set when the scrape completes successfully. Used to compute staleness against the TTL.

Failure handling is explicit. If Apify throws, the lock is released immediately so the next run can retry. If a scrape returns zero posts, resolvedAt is deliberately not set — the entity stays stale and will be retried, rather than being cached as a successful empty result.

TTL strategy — different content ages differently

Cache	TTL	Reasoning
`InspirationCreatorPost`	24 hours	Followed artists post frequently; freshness is the core value of Tier 1
`InspirationHashtagPost`	7 days	Hashtag pools change slowly; longer TTL cuts scraping cost significantly
`InspirationItem` (the feed)	7 days	A user’s assembled feed
`InspirationCreator` / `InspirationHashtag` (metadata)	90 days	Bookkeeping records, pruned only when genuinely abandoned
`InspirationCaptionEmbedding` / `InspirationAudioFeature`	permanent	Expensive to compute, never change for a given post
S3 thumbnails and audio clips	permanent	Cheap to store, expensive to re-acquire

All TTLs are enforced by MongoDB TTL indexes — the database expires documents automatically with no cleanup job to maintain.

6. How content gets understood — the enrichment layer

Once a post is cached, four background workers enrich it. All four are in-process asynchronous queues that batch work and run outside the request path.

Worker characteristics

Worker	Batch size	Flush interval	Concurrency	Retries	Purpose
`CaptionEmbedWorker`	50	5s	1	3, with backoff + re-queue	Semantic understanding of the post
`ThumbnailMirrorWorker`	50	7s	3	—	Permanent image that will not expire
`AudioMirrorWorker`	20	10s	2	—	Audio clip for analysis
`AudioAnalysisWorker`	10	15s	3	2	Musical characteristics

All four are idempotent. Each checks whether the work is already done — a database lookup for embeddings and audio features, an S3 HeadObject call for mirrors — before spending anything. A post that flows through the pipeline repeatedly is enriched exactly once.

Why thumbnails are mirrored to S3

TikTok and Instagram CDN thumbnail URLs expire within hours. The feed has a 7-day lifetime. Without mirroring, most of a user’s feed would show broken images after the first day.

The mirror produces a 400×600 JPEG at quality 82 — small enough to be inexpensive at scale, large enough for the feed grid. The feed serves the original CDN URL when available and falls back to the S3 copy, so images remain correct for the full 7 days.

Why audio is analysed

Two music videos can have completely unrelated captions and still be creatively similar because they share a tempo, an energy level, and a groove. Caption text alone misses this. The system extracts a three-dimensional audio fingerprint — normalised BPM, energy, and danceability — and blends it into the similarity calculation at a 20% weight.

7. How content gets ranked — the scoring layer

Freshness — a stepped decay

Rather than a continuous decay curve, the system uses discrete brackets. This is deliberate: it makes ranking behaviour predictable and explainable, and it means posts do not shuffle position every time the score is recomputed.

Post age	Freshness score
Under 1 week	1.0
1 week – 1 month	0.8
1 – 3 months	0.5
3 – 6 months	0.3
6 – 12 months	0.1
Over 12 months	0.0 — excluded entirely

A post with no known date scores 0.5 (neutral). Posts dated in the future — a clock-skew artefact from scrapers — are treated as maximally fresh rather than being discarded.

The engagement quality gate

Before a post can enter the feed, it must clear a platform-specific quality floor. This is the primary defence against the feed filling with low-quality content.

Engagement rate is (likes + comments + shares) / views. The 10,000-view and 5% thresholds together retain roughly the top half of scraped TikTok posts — content the platform’s own algorithm has already validated.

Instagram is handled separately because the Instagram scrapers rarely return reliable view counts; likes are used as the quality proxy instead.

Important exemption: Tier 1 posts — from the artists the user explicitly follows — bypass this filter entirely. A followed artist’s brand-new post has not accumulated views yet, and the user asked to see their content regardless of performance.

Tier 2 in detail — the AI similarity engine

Tier 2 is the most technically involved part of the system.

The taste vector is a single mathematical summary of everything the user has engaged with. Saving a post contributes 1.0; a long watch contributes 0.6; a short watch contributes 0.2. Each seed post’s caption embedding is scaled by its weight, the results are summed, and the total is normalised. The outcome is a vector pointing at the centre of the user’s demonstrated taste.

Two search strategies, one result. In production on MongoDB Atlas, the system uses native $vectorSearch — a single server-side approximate-nearest-neighbour query. In local development and non-Atlas deployments, it falls back to an in-process cosine scan that is deliberately pre-filtered to only posts with a live cache entry, so it never scores orphaned embeddings. If the Atlas query fails at runtime, the system falls back to the in-process path automatically rather than returning an empty tier.

The 80/20 caption-audio blend applies only when both the user’s seeds and the candidate post have audio features available. Otherwise the score is caption-only. This is a graceful degradation, not a hard requirement.

Score composition by tier

Tier	Formula
1 — creator_direct	`0.3 × base_weight + 0.7 × freshness`
2 — caption_similar	`similarity × freshness`, where `similarity = 0.8 × caption + 0.2 × audio`
3 — creator_expansion	`0.4 × engagement + 0.6 × freshness`
3 — audio fallback	`0.3 × engagement + 0.2 × freshness`
4 — saved_author	`0.4 × engagement + 0.6 × freshness`
4 — genre fallback	`0.3 × engagement + 0.2 × freshness`

The fallback formulas produce systematically lower scores than their primary counterparts. This is intentional — fallback content still appears, but always ranks below genuine matches.

Read-time ordering

Within each tier, items are sorted by:

opened (unopened first) → scrolledPast (not-scrolled first)
  → relevanceScore (high first) → postedAt (recent first)

Content the user has already engaged with sinks automatically, so returning to the feed surfaces new material first without any explicit “mark as read” action.

8. How the system learns — the feedback loop

Four user actions feed back into future feeds.

Auto-follow — the highest-leverage behaviour

When a user saves a post, or watches one for 5 seconds or more, the post’s creator is silently added to the user’s followed-creator list.

Capped at 20 auto-followed creators.
Implemented as a sliding window — a single atomic database operation appends the newest creator and drops the oldest when the cap is reached.
Self-cleaning: if the user later unsaves the post and has no remaining saves from that creator, the creator is removed again.

This is what makes the feed improve without configuration. An artist who simply uses the feed naturally accumulates a followed-creator list that drives Tiers 1, 3, and 4.

Hashtag weighting

Tier 3 and Tier 4 hashtag fallbacks are sourced from a weighted blend of four independent signals:

Signal source	Weight	Origin
Explicit preferences	0.4	Hashtags the user typed in settings
Buzz Factory answers	0.2	The onboarding questionnaire elsewhere in the product
Followed-creator hashtags	0.2	Tags used by creators the user follows
Saved-post hashtags	0.2	Tags extracted from captions of saved and visited posts

Weights accumulate — a hashtag appearing in three sources outranks one appearing in a single source. The top 15 hashtags are used for scraping.

Discovery expansion

After Tier 2 runs, the system extracts hashtags from the top 10 most-similar posts and adds them (up to 20) to the user’s discoveredHashtags list at a fixed 0.2 weight. These join the scraping queue on the next run, so the system gradually widens its search based on what actually resonated.

Known weakness. This mechanism drifts toward generic high-volume tags over time, because popular posts carry popular tags. It is one of the primary motivations for the proposed V3 redesign — see §14.

9. Data model

Collections by ownership

The economics of this split. Shared collections are keyed by (platform, externalId) and written at most once regardless of how many users encounter the post. Adding the ten-thousandth user costs one row per feed item — not ten thousand scrapes, embeddings, and thumbnail mirrors.

The feed item — the explainability record

InspirationItem is the central output of the pipeline. Beyond post content, each row records exactly why it was chosen:

Field	Meaning
`tier`	Which tier produced it (1–4)
`category`	The specific source: `creator_direct`, `caption_similar`, `creator_expansion`, `saved_author`, `genre`, `inspiration_post_hashtag`
`relevanceScore`	Final composite ranking score
`freshnessScore`	Age-based component
`captionSimilarity`	Cosine similarity to the taste vector (Tier 2 only)
`audioSimilarity`	Audio-fingerprint similarity (Tier 2 only, when available)
`sourceMeta.hashtagSources`	Which signal buckets contributed
`sourceMeta.sourceCreators`	The specific creator(s) responsible
`sourceMeta.seedPosts`	The user’s own saves/visits that drove this recommendation
`seen` / `opened` / `scrolledPast`	Interaction state driving read ordering

This is what makes a “Why am I seeing this?” feature possible without additional instrumentation.

10. Scheduling and automation

Throttling — and why it is so conservative

The scheduler processes only three users per run by default. This is a direct constraint of the Apify Basic plan, which limits concurrent actor runs and available memory. Processing more users in parallel would cause Apify runs to queue or fail on memory limits.

Three additional guards apply:

Per-user cooldown — a user fetched within the last 10 minutes is skipped.
Inter-user delay — 500ms between users, to spread scraper load.
Missing-token guard — if APIFY_API_TOKEN is absent, the entire job exits immediately without touching the database.

All four values are environment-configurable, so throughput can be raised as the Apify plan is upgraded without a code change.

Failure isolation

A pipeline failure for one user is caught, logged with full context, counted, and does not prevent the remaining users in the batch from being processed.

11. Cost, quality, and safety controls

Cost controls

Control	Effect
Shared caches	Each post is scraped, embedded, mirrored, and analysed once for the entire user base
In-flight locks	Concurrent pipeline runs cannot duplicate a paid scrape
TTL-based staleness	Content is re-scraped only after its TTL expires, not on every run
Cache-fresh short-circuit	If every requested hashtag is fresh, Apify is not called at all
User throttling	Hard cap of 3 users per scheduler run
Hashtag cap	At most 15 hashtags scraped per user
Music scrape cap	At most 5 trending audio IDs per run, once per day each
Enrichment idempotency	Every worker checks for existing output before spending
Cost telemetry	Every Apify run is recorded to `ServiceCost` with actor ID, compute units, USD cost, item count, and duration

The ServiceCost collection makes per-feature spend directly auditable — costs are attributed to labels such as inspiration.tiktok_hashtag and inspiration.tiktok_post.

Quality controls

Control	Effect
Engagement filter	Removes low-view, low-engagement posts (Tier 1 exempt)
Freshness cut-off	Posts older than 12 months are excluded entirely
Per-creator cap	Maximum 5 posts per creator in Tier 1, 3 in Tier 3
Cross-tier deduplication	A post can appear in only one tier per feed
Video-only filter	Instagram carousels and photos are rejected; only Reels are kept
Attribution guard	A scraped post that cannot be confidently attributed to a requested hashtag is discarded rather than mis-labelled
Regional filtering	TikTok posts can be filtered to the user’s country when region signals exist

Resilience

Scenario	Behaviour
Apify scraper fails	Lock released, error logged, pipeline continues with cached content
Scraper returns zero posts	Entity stays stale for retry; not cached as a success
Atlas Vector Search unavailable	Automatic fallback to in-process cosine similarity
genesis-train unreachable	Caption embedding retried 3× with backoff, then re-queued for the next cycle
Thumbnail CDN fetch fails	Logged; item still served with the original CDN URL
Audio extraction fails	Logged; the post simply has no audio features and is ranked on caption alone
Chartmetric rate limit (429)	Retry with a delay parsed from the response body, up to 4 attempts
Chartmetric returns nothing	Songstats used as a fallback provider; results from both are merged
Pipeline produces no Tier 1	Existing Tier 1 items are preserved rather than deleted, avoiding an empty feed

Data safety

All routes require authentication; every query is scoped by userId.
Regular-expression inputs are escaped before use, preventing ReDoS.
Post URLs are parsed against strict patterns; unsupported or malformed URLs are rejected with a clear error.
Feed items are written with $setOnInsert for content fields and $set for scores, so re-running the pipeline updates ranking without resetting a user’s interaction state.

12. External dependencies

Dependency	Role	If unavailable
Apify	All TikTok / Instagram content acquisition	No new content; existing cache continues to serve until TTL expiry
Chartmetric	Genre list, artist discovery, social-handle resolution	Falls back to a curated 12-genre list; Songstats covers social resolution
Songstats	Secondary social-link provider	Chartmetric alone is used
genesis-train	Caption embeddings and audio feature extraction	Tier 2 degrades — the other three tiers are unaffected
AWS S3	Permanent thumbnail and audio storage	Thumbnails fall back to expiring CDN URLs; audio analysis stops
MongoDB Atlas	Primary datastore and vector search	Vector search degrades to in-process cosine; storage is required

Provider redundancy is real, not theoretical. Chartmetric and Songstats have measurably different coverage per artist — Chartmetric resolved 7 of 7 social platforms for one artist where Songstats resolved 4, while both achieve full coverage for major Western artists. The system therefore queries both, merges the results, and keeps the richer set rather than trusting either alone.

13. Configuration reference

Variable	Default	Controls
`INSPIRATION_BATCH_SIZE`	`50`	Total feed size, which determines all tier budgets
`INSPIRATION_TTL_DAYS`	`7`	Feed item lifetime
`INSPIRATION_HASHTAG_CACHE_TTL_DAYS`	`1`	How long a hashtag scrape stays fresh
`INSPIRATION_CRON`	`0 3 * * *`	Scheduler timing (daily 03:00 UTC)
`INSPIRATION_MAX_USERS_PER_RUN`	`3`	Users processed per scheduler run
`INSPIRATION_MIN_INTERVAL_MINUTES`	`10`	Per-user cooldown
`INSPIRATION_USER_BATCH_DELAY_MS`	`500`	Delay between users
`INSPIRATION_FRESHNESS_BRACKETS`	see §7	JSON override of the freshness decay steps
`INSPIRATION_GENRE_HASHTAGS`	12 EDM tags	Genre fallback hashtag pool
`INSPIRATION_CONTENT_REGION`	`US`	Default region when no user signal exists
`APIFY_API_TOKEN`	—	Required. Without it the scheduler exits immediately.
`APIFY_ACTOR_MEMORY_MB`	`1024`	Per-actor memory allocation
`APIFY_ACTOR_TIMEOUT_SECS`	`300`	Per-actor timeout
`CHARTMETRICS_API_BASE_URL`	Chartmetric API	Artist and genre data source
`MEDIA_ASSET_BUCKET_NAME`	—	S3 bucket for thumbnails and audio
`AWS_REGION`	—	S3 region
`SERVICE_AUTH_TOKEN`	—	Authenticates calls to genesis-train
`FASTAPI_URL`	—	genesis-train base URL

Every Apify actor ID is also individually overridable by environment variable, so a scraper can be swapped without a deployment.

14. Known limitations

These are documented candidly because they are the explicit motivation for the proposed V3 redesign.

1. Hashtags are a coarse proxy for taste

A hashtag such as #edm returns gym edits, car edits, and memes alongside genuine music content. Relevance is capped by the coarseness of the tag itself — no amount of ranking sophistication recovers it.

2. Hashtag pools are global, personalisation is retrofitted

InspirationHashtagPost has no user scope. Personalisation for Tiers 3 and 4 is achieved by filtering a shared pool after the fact, rather than being a property of what was fetched in the first place.

3. Discovery expansion drifts toward generic

The discoveredHashtags mechanism extracts tags from the posts a user engaged with most. Because popular posts carry popular tags, the list trends toward high-volume generic tags over time. Engagement can therefore make targeting less specific — the opposite of the intended feedback effect.

4. Cost scales with hashtags, not users

Each hashtag is a separate Apify run per platform — up to 15 tags × 2 platforms per user. A significant share of that yield is then discarded by the engagement filter.

5. Onboarding artists are disconnected from the feed

The onboarding flow resolves each chosen artist’s TikTok and Instagram handles, but those handles are not currently wired into the scraping pipeline. Tier 1 is driven by manually added and auto-followed creators instead. This is the single largest gap between what onboarding promises and what the feed delivers.

6. Throughput is provider-constrained

At 3 users per nightly run, a large user base cannot be fully refreshed daily. This is an Apify plan constraint rather than an architectural one, and is resolved by upgrading the plan and raising INSPIRATION_MAX_USERS_PER_RUN.

7. In-process worker queues are not durable

The enrichment workers hold their queues in application memory. A server restart drops pending enrichment work. In practice this self-heals — the next pipeline run re-enqueues anything still missing — but individual items may be delayed by a cycle.

The proposed direction

V3_PLAN.md proposes replacing hashtag-based discovery with artist-based discovery, using Chartmetric’s verified artist-similarity graph. Limitations 1–5 are addressed directly: content pools become artist-scoped, similarity comes from a music-domain graph rather than string matching, and engagement feeds back into which artists to expand — a signal that sharpens rather than blurs with use.

Notably, roughly 70% of the V3 plan reuses infrastructure this document describes. The ingestion layer, all four enrichment workers, the taste vector, vector search, the scoring functions, and the scrape orchestration all carry over unchanged. Only the selection of what to scrape changes.

15. Code map

For engineers picking up the module.

packages/server/module/inspiration/
├── index.js                          Public API — the only sanctioned import surface
├── V3_PLAN.md                        Proposed successor design
│
├── feed/
│   ├── feedService.js                Pipeline + 4-tier assembler + feed read  (~2,280 lines)
│   └── scoring.js                    Pure ranking math — freshness, hashtag weights
│
├── ingestion/
│   ├── scheduler.js                  Nightly cron job
│   ├── hashtagScrapeService.js       Hashtag scraping + Apify → cache mapping
│   ├── savedPostScrapeService.js     URL-sourced saved-post resolution
│   └── enrichment/
│       ├── captionEmbedWorker.js     Caption → vector via genesis-train
│       ├── thumbnailMirrorWorker.js  Thumbnail → resize → S3
│       ├── audioMirrorWorker.js      Video → 30s audio clip → S3
│       └── audioAnalysisWorker.js    Audio clip → BPM / energy / key
│
├── http/
│   ├── inspirationRoutes.js          Route table
│   ├── inspirationController.js      Request handlers + Swagger annotations
│   └── preferences.schema.js         Request validation
│
├── preferences/
│   ├── preferencesService.js         Saved posts, creators, auto-follow, hashtag signals
│   └── urlParser.js                  Pure URL / handle parsing
│
├── shared/
│   └── hashtagText.js                Hashtag extraction + config helpers
│
└── models/                           10 Mongoose schemas (see §9)

This document describes the system as implemented and verified against the source on 2026-08-10.