Lead Qualification - Scope

Lead Classification System - Project Plan

Project Status: 📋 Planning & Validation Phase
Last Updated: 2026-04-20
Owner: Jacob Stallings
Timeline: 10-15 business days (1 sprint, single developer)

Executive Summary

Build a lead classification system that evaluates artist leads and assigns them a “level” based on activity metrics. The system uses ChartMetric API exclusively for Instagram and Spotify data to classify leads, then pushes enriched lead data to HubSpot for sales outreach.

Key Goal: Proactively assess and qualify leads as early as possible using external data sources.

Tech Stack: Python (extend existing leadgen repo)
Deployment: CLI script → AWS Lambda → Step Functions (at scale)

Quick Reference

Question	Answer
Timeline?	10-15 days (1 sprint, single developer)
Cost?	~$0-5/month (Apify already available, no new services)
Level thresholds?	5 posts (Level 2), 15 posts (Level 3), 30 posts (Level 4)
Tech stack?	Python (extend `leadgen` repo)
Biggest blocker?	ChartMetric label discovery workaround (validate first 3-5 days)
Deployment?	MVP: CLI script → Production: AWS Lambda → Scale: Step Functions

🚨 Critical Technical Requirements

1. Level Classification System

The main app uses simple post count thresholds:

Level 2: 5 posts
Level 3: 15 posts
Level 4: 30 posts

Requirement: We need Instagram post counts (not just URLs/followers) for accurate classification.

2. ChartMetric Instagram Linking

ChartMetric provides Instagram URLs via /api/artist/{id}/urls:

⚠️ Reliability unknown - must test with 50-100 artists (Days 1-2)
❌ No post counts - ChartMetric provides URLs but NOT post counts
✅ Apify available - Already used elsewhere, likely no additional cost

3. SongStats Constraint

Blocker: SongStats charges per artist. Discovering 100-1,000 artists = $$$

Solution (ChartMetric only):

Search label → Get artists → Get releases → Verify label from metadata
Reverse index: label → artists
Risk: If release metadata lacks label info, discovery may be incomplete

Validation: Days 1-3 critical test

4. Tech Stack

Python-based extension of existing leadgen repo:

✅ Single unified pipeline (discovery → HubSpot)
✅ Ideal for future ML/RAG enhancements
✅ Easy Lambda/Step Function deployment

System Components & Architecture

leadgen (Python)
- Purpose: Discovers artist leads from label catalogs
- Current State: Uses SongStats API
- Plan: Migrate to ChartMetric API
ig-hubspot-tool (Nest.js + AWS Lambda)
- Purpose: Tracks Instagram DM activity and syncs to HubSpot
- Integration: Picks up leads after classification (no changes needed)
genesis-web (Nest.js + React)
- Contains: Level threshold constants and classification logic
- Use: Reference thresholds for lead classification

System Components

Existing Repositories

┌─────────────────────────────────────────────────────────────────┐
│             Lead Pipeline (Python - ChartMetric Only)            │
│                    Single Unified Service                        │
└─────────────────────────────────────────────────────────────────┘
         │
         ▼
┌─────────────────┐
│  Step 1:        │  Lead Discovery (NEW APPROACH)
│  Label → Artist │  ├─ ChartMetric: Search label
│  Discovery      │  ├─ Find artists via search
│  (ChartMetric)  │  ├─ Get releases: /artist/{id}/tracks
│                 │  └─ Verify label from release metadata
│                 │  ⚠️ CRITICAL: Validate this workaround Week 1
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Step 2:        │  Metrics Enrichment
│  ChartMetric    │  ├─ Monthly listeners
│  Enrichment     │  ├─ Follower counts
│                 │  └─ Instagram URL (via /artist/{id}/urls)
│                 │  ⚠️ CRITICAL: Test URL reliability Week 1
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Step 3:        │  Instagram Data (Conditional)
│  Apify/Bypass   │  ├─ IF Instagram URL from ChartMetric: Parse handle
│  (Optional)     │  ├─ ELSE IF needed: Use Apify to discover
│                 │  └─ Get follower count, post count
│                 │  💰 Decision: Use Apify ($) or proxy metrics (free)?
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Step 4:        │  Classification
│  Level & Type   │  ├─ Apply post thresholds (5/15/30 posts)
│  Classification  │  ├─ Assign level (1-5)
│                 │  └─ Detect problem types (V2)
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  Step 5:        │  HubSpot Ingestion
│  HubSpot CRM    │  ├─ Create/update contact
│                 │  ├─ Set custom properties
│                 │  └─ Add to lead pipeline
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│  (Separate)     │  DM Tracking
│  ig-hubspot-tool│  └─ Existing service (no changes needed)
└─────────────────┘

CSV Output (Optional):
├─ leads_discovered.csv     (after Step 1 - for records)
├─ leads_enriched.csv       (after Step 3 - for records)
└─ leads_classified.csv     (after Step 4 - for records)

Note: CSV outputs are for record-keeping only, NOT required for next steps.
      Pipeline flows automatically from Step 1 → Step 5.

❌ SongStats: REMOVED - Per-artist pricing too expensive at scale
✅ ChartMetric: Primary data source (subscription-based pricing)

Deployment Options

Option 1: Python CLI (MVP)

Single python -m leadgen --labels "Label1" "Label2" command
Runs all 5 steps sequentially
Output: Logs + optional CSV + HubSpot contacts created
Trigger: Cron job, manual execution

Option 2: AWS Lambda (Production)

Single Lambda function, Python runtime
Trigger: EventBridge schedule (weekly/monthly batch)
Timeout: 15 minutes (adjust based on volume)
Memory: 1024 MB

Option 3: AWS Step Functions (Scale)

Orchestrate steps as separate Lambda functions
Parallel processing for multiple labels
Error handling and retries per step
CloudWatch monitoring per step

yaml

# Step Function Definition
StartAt: DiscoverLeads
States:
  DiscoverLeads:
    Type: Task
    Resource: arn:aws:lambda:LeadDiscovery
    Next: EnrichMetrics
  
  EnrichMetrics:
    Type: Task
    Resource: arn:aws:lambda:ChartMetricEnrichment
    Next: ClassifyLeads
  
  ClassifyLeads:
    Type: Task
    Resource: arn:aws:lambda:Classification
    Next: IngestHubSpot
  
  IngestHubSpot:
    Type: Task
    Resource: arn:aws:lambda:HubSpotIngestion
    End: true

Detailed Implementation Plan

Phase 1: ChartMetric Integration & Validation

Goal: Replace SongStats with ChartMetric and validate label discovery approach

Duration: 5-7 days

Requirements

Input: CSV from leadgen with:
- Artist name
- Spotify artist ID
- Discovered via label
Data Collection:
- Instagram: Use Apify to scrape:
  - Username (if discoverable)
  - Follower count
  - Post count (last 6 months)
  - Engagement rate (if available)
- Spotify: Use ChartMetrics API to fetch:
  - Monthly listeners
  - Follower count
  - Release count (last 6 months)
Classification Logic:
- Apply thresholds from genesis-web/packages/server/services/gamification/constants/levelThreshold.js
- Level 2 Threshold (Starting point for leads):
  - 5 Instagram posts (LOW) / 12 posts (MEDIUM) / 15 posts (HIGH)
  - 1 Spotify release
- Level 3 Threshold:
  - 15 IG posts (LOW), 60 posts (MEDIUM), 70 posts (HIGH)
  - 1-5 releases depending on aggression
  - Followers: 60 (MEDIUM), 70 (HIGH)
- Level 4 Threshold:
  - IG Followers: 2,010 (LOW), 3,099 (MEDIUM), 4,149 (HIGH)
  - Spotify Monthly Listeners: 10,058 (LOW), 15,115 (MEDIUM), 20,173 (HIGH)
  - Spotify Followers: 219 (LOW), 304 (MEDIUM/HIGH)

Output:

Enriched lead object:

json

{
  "artistName": "Artist Name",
  "spotifyId": "abc123",
  "instagramUsername": "artist_handle",
  "level": 3,
  "levelName": "Consistent Creator",
  "metrics": {
    "instagramFollowers": 5200,
    "instagramPostCount": 45,
    "spotifyMonthlyListeners": 12000,
    "spotifyFollowers": 350,
    "spotifyReleaseCount": 3
  },
  "classifiedAt": "2026-04-20T12:00:00Z"
}

HubSpot Integration:
- Create HubSpot contact with custom properties:
  - instagram_username
  - instagram_followers
  - spotify_artist_id
  - spotify_monthly_listeners
  - lead_level (1-5)
  - lead_level_name (e.g., “Consistent Creator”)
  - classified_at
- Use existing HubSpot Private App from ig-hubspot-tool

Technical Decisions

Apify Scraper: Use Apify’s Instagram Profile Scraper actor (already in use elsewhere)
- Cost Estimate: Confirm with existing subscription
- Rate Limits: Based on company’s Apify plan
ChartMetric API: Primary data source for all metrics
- Reason: Better rate limits for bulk operations
- Cost: Existing subscription (confirm current plan includes required endpoints)
Level Assignment: Fixed post count thresholds (5/15/30)

Open Questions

Can we reliably link Spotify artist → Instagram username via ChartMetric?
- ChartMetric provides URLs via /artist/{id}/urls endpoint
- Need to validate success rate (>70% for viability)
- If fails: Use Apify for discovery (existing subscription)
What’s the cost for Apify at expected scale?
- Company already uses Apify - confirm subscription covers additional usage
- If not covered, estimate: 1,000-10,000 profiles/month

Phase 4: Problem Type Classification

Goal: Beyond just “level,” identify specific growth problems/opportunities.

Classification Categories

User to provide specific threshold combinations and category names. Placeholder structure:

High Engagement, Low Releases
- Threshold: instagramFollowers > X AND spotifyReleaseCount < Y
- Message: “Strong fanbase, needs consistent releases”
Momentum Lost (Fell Off Posting)
- Threshold: historicalPostCount > X AND recentPostCount < Y
- Message: “Was active, posting declined”
High Release Count, Low Followers
- Threshold: spotifyReleaseCount > X AND instagramFollowers < Y
- Message: “Prolific creator, needs audience growth”
Early Career (Small but Active)
- Threshold: instagramFollowers < X AND spotifyReleaseCount > Y AND postConsistency === true
- Message: “Active creator, early stage”
[USER TO DEFINE ADDITIONAL CATEGORIES]
- Threshold: TBD
- Message: TBD

Requirements

Scenario Detection:
- Create a rules engine to evaluate lead metrics against scenario thresholds
- Assign one or more “tags” to each lead based on matches

Enhanced Output:

Add to enriched lead object:

json

{
  "classification": {
    "level": 3,
    "scenarios": ["high_engagement_low_releases", "early_career"],
    "primaryScenario": "high_engagement_low_releases",
    "recommendedAction": "Pitch release planning tools and distribution strategy"
  }
}

HubSpot Integration:
- Add custom properties:
  - lead_scenarios (multi-select or concatenated string)
  - recommended_action (text)
  - primary_scenario

Open Questions

How many scenario categories are needed? (User to define)
Should scenarios be mutually exclusive or can leads have multiple tags?
Do we need historical data (6-month trends) or just current snapshot?

Phase 5: RAG-Based Classification (Future Work)

Goal: Use RAG (Retrieval-Augmented Generation) to dynamically classify leads using LLM + context.

High-Level Plan (Future Enhancement)

Vector Database: Store artist profiles and classification examples
RAG Pipeline: Query similar artist profiles → LLM classifies new lead
Benefits:
- More nuanced classification beyond static thresholds
- Learns from successful/unsuccessful outreach patterns
- Can incorporate qualitative data (genre, bio, aesthetic)
Integration Point: Replace rules engine in V2 with RAG pipeline

Status: 🚧 Future enhancement - not part of current scope.

Implementation Phases

Phase 1: ChartMetric Integration & Validation

Goal: Replace SongStats with ChartMetric and validate label discovery approach

Tasks

[ ] Task 1.1: Validate ChartMetric label discovery workaround (CRITICAL)
- Test the release → label metadata approach:
  - Get sample artist: /api/artist/{id}
  - Get their releases: /api/artist/{id}/tracks or /api/artist/{id}/albums
  - Check if label info is in release metadata
- Test reverse search: Can we find artists by label?
  - Try /api/search?q={label_name}&type=artists
  - Check if results can be filtered by label
- Success criteria: Can discover 50+ artists from a single label query
- Decision point: If workaround fails, need alternative (manual artist lists?)
[ ] Task 1.2: Test ChartMetric Instagram URL endpoint
- Test /api/artist/{id}/urls with 50-100 sample artists
- Measure: % with Instagram URLs populated, accuracy of links
- Document success rate and data quality
- Decision point: >70% success = proceed, <70% = need Apify
[ ] Task 1.3: Create ChartMetric Python client
- Model after existing songstats_client.py structure
- Implement endpoints:
  - GET /api/artist/{chartmetric_id} (basic info)
  - GET /api/artist/{chartmetric_id}/urls (social links)
  - GET /api/artist/{chartmetric_id}/spotify-monthly-listeners (metrics)
  - GET /api/artist/{chartmetric_id}/spotify-audience-stats (followers)
  - GET /api/artist/{chartmetric_id}/tracks (releases + label metadata)
  - GET /api/search (label/artist search)
- Rate limiting and retry logic
- Unit tests
[ ] Task 1.4: Implement label discovery logic
- Port existing leadgen/pipeline.py to use ChartMetric
- Replace resolve_labels() and fetch_label_artists() with ChartMetric workaround
- New function: discover_artists_by_label(label_name) -> List[ArtistLead]
- Test with 3-5 sample labels, compare results to old SongStats output
[ ] Task 1.5: Update ArtistLead model
- Add chartmetric_id field
- Add post_count field (for level classification)
- Add level and level_name fields
- Keep existing Instagram/Spotify fields

Estimated Duration: 5-7 days

Critical Decision Points:

Task 1.1: Label discovery feasibility - BLOCKS entire migration
Task 1.2: Instagram URL success rate - Determines if Apify needed (+$$$)

Phase 2: Instagram Post Count (Critical for Classification)

Goal: Get Instagram post count for level assignment

Challenge

ChartMetric’s /urls endpoint provides Instagram URLs but NOT post counts. We need post counts to classify leads into levels (5/15/30 posts).

Options

Option A: Apify Instagram Scraper (Paid)

Cost: $49-500/month depending on volume
Reliability: High
Data: Follower count, post count, engagement rate
Setup: Simple API integration

Option B: Instagram Basic Display API (Free but Limited)

Cost: Free
Limitation: Requires OAuth (not viable for leads)
Verdict: Not feasible

Option C: Skip Post Count, Use Follower/Listener Proxy

Use follower counts as proxy for activity level
Classification based on follower/listener thresholds
Pros: No additional cost
Cons: Less accurate, doesn’t match app’s system

Option D: Hybrid - Post Count for New Leads Only

For leads < 6 months old: Use Apify to get post count
For older leads: Skip or use proxy metrics
Reduces Apify usage (cost optimization)

Tasks

[ ] Task 2.1: Decide on Instagram data strategy (based on Phase 1 results)
- If ChartMetric URL success rate >70% AND budget allows: Go with Apify
- Otherwise: Use Option C (follower/listener proxy)
[ ] Task 2.2 (Conditional - if Apify chosen): Set up Apify integration
- Sign up for Apify account
- Test Instagram Profile Scraper with 10-20 artists
- Implement apify_client.py with rate limiting
- Add post count to ArtistLead model
[ ] Task 2.3 (Alternative - if proxy chosen): Update classification logic
- Use follower/listener based thresholds
- Document difference from app’s post-based system
- Add note that post-based classification requires Apify

Estimated Duration: 2-3 days (setup) or skip if using proxy

Cost Decision: Get approval for Apify before proceeding with Option A

Phase 3: Classification & HubSpot Integration

Goal: Implement level classification and push to HubSpot

Tasks

[ ] Task 3.1: Implement classification logic
- Create classifier.py module
- Implement classification thresholds:
  python
```
LEVEL_2_THRESHOLD = 5  # posts
LEVEL_3_THRESHOLD = 15  # posts
LEVEL_4_THRESHOLD = 30  # posts
```
- Classification function: classify_lead(post_count: int) -> (level: int, level_name: str)
- Handle edge cases (no Instagram data, etc.)
- Unit tests
[ ] Task 3.2: Create HubSpot Python client
- Use HubSpot Private App token (from ig-hubspot-tool)
- Implement:
  - Contact search by Spotify artist ID
  - Contact creation with custom properties
  - Contact update (if exists)
- Rate limiting (100 req/10 sec)
- Retry logic with exponential backoff
- Unit tests

[ ] Task 3.3: Define HubSpot custom properties

Create custom properties in HubSpot account:

json

{
  "spotify_artist_id": "string",
  "chartmetric_artist_id": "string",
  "instagram_username": "string",
  "instagram_followers": "number",
  "instagram_post_count": "number",
  "spotify_monthly_listeners": "number",
  "spotify_followers": "number",
  "lead_level": "number",
  "lead_level_name": "string",
  "classified_at": "datetime",
  "data_sources": "string"  // e.g., "chartmetric,apify"
}

Document property IDs for API usage

[ ] Task 3.4: Update pipeline to include classification + HubSpot

Modify pipeline.py:

python

async def run_full_pipeline():
    # Step 1: Discover leads (existing)
    leads = await discover_leads_from_labels(labels)
    
    # Step 2: Enrich with ChartMetric (new)
    leads = await enrich_with_chartmetric(leads)
    
    # Step 3: Enrich with Apify (conditional, new)
    if use_apify:
        leads = await enrich_with_apify(leads)
    
    # Step 4: Classify leads (new)
    leads = await classify_leads(leads)
    
    # Step 5: Push to HubSpot (new)
    results = await ingest_to_hubspot(leads)
    
    # Step 6: Output CSV for records (optional)
    write_csv(leads, output_path)
    
    return results

Add CLI flags: --skip-hubspot, --use-apify, --dry-run

[ ] Task 3.5: Error handling & logging
- Handle ChartMetric rate limits
- Handle HubSpot rate limits
- Log failed classifications to separate file
- Retry failed HubSpot ingestions
- CloudWatch logging (if deployed to Lambda)
[ ] Task 3.6: Integration testing
- End-to-end test with 10 sample labels
- Verify HubSpot contacts created correctly
- Verify custom properties populated
- Test error recovery (API failures, rate limits)

Estimated Duration: 5-7 days

Phase 4: Problem Type Classification

Goal: Add scenario detection beyond static levels

Prerequisites

User provides:
- List of scenario categories (names + descriptions)
- Threshold combinations for each scenario
- Recommended actions for each scenario

Tasks

Task 4.1: Define scenario rules engine
- Create JSON config for scenario thresholds
- Build rules evaluator (if/then logic)
- Assign scenarios to leads based on metrics
Task 4.2: Update HubSpot integration
- Add custom properties for scenarios
- Update contact creation logic
Task 4.3: Testing
- Validate scenario detection with sample data
- Ensure multiple scenarios can be assigned

Estimated Duration: 3-5 days

Phase 5: Integration with IG-HubSpot Tool

Goal: Connect classified leads to DM tracking pipeline

Tasks

Task 5.1: Document handoff process
- Lead created in HubSpot → tracked by ig-hubspot-tool
- Ensure custom properties are preserved
Task 5.2: (Optional) Automate lead discovery → classification → HubSpot
- Cron job or Lambda scheduled function
- Weekly/monthly lead discovery + classification batch

Estimated Duration: 1-2 days

Repository Structure Decision

Option A: Extend `leadgen` Repo (RECOMMENDED)

Pros:

Lead discovery and classification are conceptually related
Same tech stack (Python) - easy integration
Reuse existing SongStats client (if using hybrid approach)
Single CLI for full pipeline (discovery → classification → HubSpot)
Python is better for ML/RAG future work (V3)

Cons:

None significant - this is the clean choice

Architecture:

leadgen/
├── leadgen/
│   ├── songstats_client.py      # Existing: SongStats for label discovery
│   ├── chartmetric_client.py    # NEW: ChartMetric for metrics + IG links
│   ├── classifier.py            # NEW: Level classification logic
│   ├── hubspot_client.py        # NEW: HubSpot contact creation
│   ├── apify_client.py          # NEW (Optional): Apify for Instagram
│   ├── pipeline.py              # UPDATE: Extend with classification step
│   └── models.py                # UPDATE: Add ClassifiedLead model

Option B: Extend `ig-hubspot-tool` Repo

Pros:

Already has HubSpot integration

Cons:

Wrong tech stack (TypeScript/Nest.js) - leadgen is Python
Purpose misalignment (DM tracking vs lead classification)
Would require TypeScript rewrite or Python microservice in a Node repo

Option C: New Repo `lead-classifier`

Pros:

Clean separation of concerns

Cons:

Additional repo to manage
Doesn’t solve tech stack issue
Artificial separation - classification is part of lead enrichment pipeline

Recommendation: Option A - Extend `leadgen` Repo with Python

Keep Python: Makes V3 RAG integration trivial (langchain, llamaindex, etc.)
Single unified pipeline: leadgen becomes full lead-to-HubSpot pipeline
Reuse existing code: SongStats client, rate limiting, models
Easy local testing: Python CLI with all steps in one place

Deployment Strategy:

V1 (MVP): Run as Python CLI script (cron job or manual)
V2 (Production): Deploy as AWS Lambda (Python runtime) triggered by:
- EventBridge schedule (weekly/monthly batch)
- S3 upload trigger (new label list uploaded)
- API Gateway endpoint (on-demand classification)

V3 (Scale): AWS Step Functions orchestration:

Step 1: Lead Discovery (leadgen)
     ↓
Step 2: ChartMetric Enrichment
     ↓
Step 3: Apify Instagram (if needed)
     ↓
Step 4: Classification
     ↓
Step 5: HubSpot Ingestion

Why Not TypeScript?

Python has better ML/AI ecosystem for V3
Leadgen is already Python - don’t split tech stacks
HubSpot integration is simple HTTP API (works fine in Python)
Can still containerize and orchestrate with Step Functions

Data Sources & APIs

1. ChartMetrics API (Spotify + Instagram Data)

Purpose: Replace SongStats for lead discovery and classification

Critical Discovery: Instagram Linking via ChartMetrics

ChartMetric provides an Artist URLs endpoint that may include social media links:

Endpoint: GET /api/artist/{chartmetric_id}/urls (API Doc)
Returns: Array of URLs including Instagram, TikTok, Facebook, YouTube, Twitter, etc.
Reliability: TBD - NEEDS VALIDATION
- This is the BIGGEST feasibility question for the project
- We need to test with 50-100 sample artists to determine:
  - What % have Instagram links populated?
  - How accurate are the links?
  - Are handles parseable from URLs?
- If this fails, fallback options:
  1. Use Apify for Instagram discovery (costs money)
  2. Skip Instagram data entirely for leads (Spotify-only classification)
  3. Manual review/enrichment process

SongStats → ChartMetrics Endpoint Mapping

SongStats Endpoint (DEPRECATED)	ChartMetrics Equivalent	Status	Notes
`/labels/search`	`/api/search?type=labels`	⚠️ Needs testing	May work, need to validate
`/labels/catalog`	Workaround via releases	⚠️ Complex	Get artist → releases → label, then reverse search. See workaround strategy.
`/artists/info`	`/api/artist/{id}`	✅ Confirmed	Basic artist metadata
`/artists/info` (links)	`/api/artist/{id}/urls`	⚠️ Needs testing	Social media links including Instagram (CRITICAL to test)
`/artists/stats`	`/api/artist/{id}/spotify-monthly-listeners`	✅ Confirmed	Spotify monthly listeners over time
`/artists/stats` (followers)	`/api/artist/{id}/spotify-audience-stats`	✅ Confirmed	Spotify follower count
N/A	`/api/artist/{id}/tracks`	⚠️ Needs testing	CRITICAL: Check if includes label metadata
N/A	`/api/artist/{id}/albums`	⚠️ Needs testing	Alternative to tracks for label discovery
N/A	`/api/search?q={query}&type=artists`	✅ Available	General artist search (can filter by label in results?)

Why Not SongStats: Per-artist pricing makes it prohibitively expensive for discovering 100-1,000 artists at scale. Must migrate fully to ChartMetric.

Label Discovery via ChartMetric (Required - SongStats Too Expensive)

Constraint: SongStats charges per artist. Discovering 100-1,000 artists would be $$$. MUST use ChartMetric only.

Proposed Workaround (Artist → Releases → Label → Reverse Search):

python

# Step 1: Search for label
label_results = search_chartmetric(query=label_name, type="labels")
label_id = label_results[0]['id']  # Get ChartMetric label ID

# Step 2: Search for artists on that label
artist_results = search_chartmetric(query=label_name, type="artists")

# Step 3: For each artist, get releases and verify label
verified_artists = []
for artist in artist_results:
    releases = get_artist_releases(artist['id'])
    for release in releases:
        if release['label_id'] == label_id or release['label_name'] == label_name:
            verified_artists.append(artist)
            break  # Artist confirmed on this label

# Step 4: Alternative - If releases have label metadata, crawl outward
seed_artists = get_known_artists_on_label()
for seed_artist in seed_artists:
    releases = get_artist_releases(seed_artist['id'])
    for release in releases:
        # Find other artists on the same release (collaborations)
        # Or search for other releases on this label
        related_artists = find_artists_by_label(release['label_id'])
        verified_artists.extend(related_artists)

Key Questions to Validate (Week 1):

Does /api/artist/{id}/tracks or /api/artist/{id}/albums include label metadata?
How complete is the label info in release metadata?
Can we search by label ID: /api/search?label_id={id}&type=artists?
What’s the success rate compared to SongStats’ label catalog?

Fallback: If release metadata lacks label info, we may need to:

Use /api/search?q={label_name}&type=artists and trust search relevance
Accept lower discovery rate vs SongStats
Supplement with manual artist lists

ChartMetric Rate Limits

Tier dependent (need to check existing account)
Generally more generous than SongStats for bulk operations
Typical enterprise tier: 100-200 requests/minute

ChartMetric Cost

Likely covered by existing subscription
Need to confirm current plan includes required endpoints

2. Apify Instagram Profile Scraper

Purpose: Get Instagram data without OAuth

Data Available:

Username
Follower count
Following count
Post count
Recent posts (with engagement metrics)
Bio, profile picture, etc.

Rate Limits:

Based on Apify plan (e.g., 1,000 profiles = $49/month)
Check documentation for exact limits

Cost Estimate (To Validate):

1,000 leads/month: ~$49
5,000 leads/month: ~$200-250
10,000 leads/month: ~$400-500

Alternative: Apify Instagram Hashtag Scraper (if username not known)

3. HubSpot API (CRM)

Purpose: Create/update contacts with enriched lead data

Endpoints:

POST /crm/v3/objects/contacts (create contact)
PATCH /crm/v3/objects/contacts/{id} (update contact)
Custom property creation (one-time setup)

Authentication: Private App token (existing from ig-hubspot-tool)

Rate Limits: 100 requests/10 seconds (standard HubSpot limit)

Custom HubSpot Properties (To Create)

typescript

// Lead Classification Properties
{
  "lead_level": "number", // 1-5
  "lead_level_name": "string", // e.g., "Consistent Creator"
  "classified_at": "datetime",
  
  // Instagram Metrics
  "instagram_username": "string",
  "instagram_followers": "number",
  "instagram_post_count": "number",
  "instagram_engagement_rate": "number", // percentage
  
  // Spotify Metrics
  "spotify_artist_id": "string",
  "spotify_monthly_listeners": "number",
  "spotify_followers": "number",
  "spotify_release_count": "number",
  
  // V2: Problem Type Classification
  "lead_scenarios": "string", // comma-separated tags
  "primary_scenario": "string",
  "recommended_action": "string"
}

Note: Some properties may already exist from ig-hubspot-tool (e.g., instagram_username). Reuse where possible.

Cost Estimation

Monthly Operating Costs

Base Scenario: 1,000 Leads/Month

Service	Monthly Cost	Notes
ChartMetric API	$0	Existing subscription
Apify (Instagram data)	$0-50	Already used elsewhere - confirm additional usage covered
AWS Lambda	$0-5	Within free tier for expected volume
HubSpot API	$0	Existing Private App
Total	$0-55/month	Minimal to no additional cost

High Volume: 10,000 Leads/Month

Service	Monthly Cost	Notes
ChartMetric API	$0-50	May need higher tier (confirm current plan)
Apify	$0-100	Confirm coverage at higher volume
AWS Lambda	$10-20	More invocations
Total	$10-170/month	Still minimal

Alternative: Proxy Metrics (No Apify)

If Apify coverage is insufficient or you prefer to avoid it:

Service	Monthly Cost	Notes
ChartMetric API	$0	Use follower/listener data instead of post counts
AWS Lambda	$0-5	Within free tier
Total	$0-5/month	Free option

Trade-off: Using follower/listener counts as proxy for activity level is less accurate than post counts but requires no additional services.

Development Cost

10-15 business days of developer time (1 sprint)
Uses existing infrastructure (no new services to provision)
No upfront infrastructure costs

Technical Decision Reference

Question	Answer	Status
What are the level thresholds?	5 posts (Level 2), 15 posts (Level 3), 30 posts (Level 4)	✅ Confirmed
Can we use ChartMetric for Instagram?	Provides URLs via `/artist/{id}/urls`, but reliability TBD	⚠️ Needs testing
Do we need Apify?	YES for post counts (ChartMetric doesn’t provide), MAYBE for URL discovery	⚠️ Needs testing
Can we replace SongStats entirely?	YES (required) - SongStats too expensive. Use release → label workaround.	⚠️ Needs validation
What tech stack?	Python (extend `leadgen` repo)	✅ Decided
Automatic pipeline (no CSV handoff)?	YES - single unified Python pipeline	✅ Decided
Deployment strategy?	MVP: CLI, Production: Lambda, Scale: Step Functions	✅ Decided
Biggest blocker?	ChartMetric label discovery viability (validate Days 1-3)	🚨 Critical
Second biggest blocker?	ChartMetric Instagram URL reliability (validate Days 1-3)	🚨 Critical
Third blocker?	Post count strategy decision (Apify or proxy metrics)	⚠️ Budget decision

Open Questions & Blockers

Critical Path Blockers

ChartMetric Label Discovery Viability (HIGHEST PRIORITY) 🚨
- ACTION NEEDED (Days 1-3): Test release → label metadata approach
- Method:
  1. Get known artist on specific label
  2. Call /api/artist/{id}/tracks or /api/artist/{id}/albums
  3. Check if release metadata includes label name/ID
  4. Try reverse search: /api/search?q={label_name}&type=artists
- Success Criteria: Can discover 50+ artists from a label query
- If Success: ChartMetric-only approach is viable
- If Failure: Need fallback (manual artist lists? alternative discovery method?)
- Timeline: Days 1-3 - BLOCKS entire project
ChartMetric Instagram URL Reliability (CRITICAL) ⚠️
- ACTION NEEDED (Days 1-2): Test /api/artist/{id}/urls with 50-100 artists
- Success Criteria: >70% of artists have valid Instagram URLs
- If Success: Proceed with ChartMetric-only approach (no Apify needed for URL discovery)
- If Failure: Must use Apify for Instagram discovery (adds $50-500/month cost)
- Timeline: Week 1 - BLOCKS entire project feasibility
Instagram Post Count Strategy (HIGH PRIORITY) 💰
- ChartMetric provides Instagram URLS but NOT post counts
- Post counts are REQUIRED for V2 level classification (5/15/30 thresholds)
- Options:
  - A: Use Apify ($49-500/month) - accurate but costly
  - B: Use follower/listener proxy (OLD V1 thresholds) - free but less accurate
  - C: Hybrid (Apify for new leads only) - cost optimization
- Decision needed: Which option fits budget and accuracy requirements?
- Timeline: Week 2 - BLOCKS classification implementation
Spotify ID → ChartMetric ID Mapping (HIGH PRIORITY) 🔍
- How do we convert Spotify artist IDs to ChartMetric IDs?
- Potential solutions:
  - ChartMetric search by Spotify ID (if endpoint exists)
  - Artist name fuzzy matching
  - ISRC code lookup
- Impact: If mapping fails, can’t use ChartMetric data
- Timeline: Week 1-2

Non-Blocking Questions

ChartMetric Rate Limits (LOW PRIORITY)
- Confirm current ChartMetric plan and rate limits
- Check if required endpoints are included in plan
- Impact: May need rate limiting strategy
HubSpot Ingestion Volume (LOW PRIORITY)
- Expected volume: 100-1,000 leads per batch
- HubSpot rate limit: 100 req/10 sec (sufficient for batch processing)
Deployment Strategy (LOW PRIORITY)
- Start with CLI script (manual/cron execution)
- Migrate to Lambda when automation needed

Success Metrics

MVP Success Criteria

90%+ of leads successfully classified (level assigned)
Instagram data retrieved for 70%+ of leads (some won’t have IG)
All leads pushed to HubSpot with custom properties populated
Processing time: <5 seconds per lead (reasonable for batch job)
Cost per lead: <$0.50 (including any additional services + compute)

Enhanced Success Criteria (Phase 4)

All leads have at least one scenario tag assigned
Scenario accuracy validated against manual review (90%+)
Sales team finds scenarios actionable for outreach

Timeline Estimate

Total Duration: 10-15 Business Days (1 Sprint, Single Developer)

Phase	Duration (Days)	Dependencies	Key Deliverables
Validation	3-5	ChartMetric API access	Label discovery viability, Instagram URL success rate
Phase 1: ChartMetric Integration	5-7	Validation passes	ChartMetric client, label→artist discovery logic
Phase 2: Instagram Strategy	2-3	Phase 1 complete	Instagram data collection (Apify or proxy)
Phase 3: Classification & HubSpot	5-7	Phases 1-2 complete	Level classification, HubSpot contact creation
Phase 4: Problem Type Detection	3-5	Phase 3 complete, scenario definitions provided	Scenario-based lead tagging
Phase 5: Production Deploy	1-2	Phase 3+ complete	Lambda deployment, monitoring setup
Total	10-15 days		Phases 4-5 optional for MVP

Critical Path

Days 1-3: Validation
    ├─ ChartMetric label discovery works? → YES → Proceed
    │                                      → NO → Fallback strategy (+2-3 days)
    │
    └─ Instagram URL success >70%? → YES → Use ChartMetric URLs
                                    → NO → Use Apify for discovery

Days 4-10: Core Development (Phases 1-3)
    └─ Minimum viable product (MVP) complete

Days 11-15: Enhancements (Optional)
    └─ Scenario detection + production deployment

Note: MVP requires only Phases 1-3 (validation + classification + HubSpot). Phases 4-5 can be added later.

Next Steps (Immediate Actions)

Week 1: Critical Validation (HIGHEST PRIORITY)

Validate ChartMetric Label Discovery Workaround (BLOCKS ENTIRE PROJECT) 🚨
- Get ChartMetric API access (confirm existing account + endpoints)
- Test the release → label metadata approach:
  - Pick 3 sample artists you know are on specific labels
  - Call /api/artist/{id}/tracks or /api/artist/{id}/albums
  - Verify label info is in the response
- Test reverse discovery:
  - Try /api/search?q={label_name}&type=artists
  - Can you find 10+ artists from one label?
  - How accurate are the results?
- Decision point:
  - If workaround works: Proceed with ChartMetric-only
  - If workaround fails: Need fallback strategy (manual artist lists? hybrid with limited SongStats?)
- Document success rate and limitations
Validate ChartMetric Instagram URLs (BLOCKS FEASIBILITY) ⚠️
- Test /api/artist/{id}/urls with 50-100 sample artists
- Measure:
  - % with Instagram URLs populated
  - Accuracy of URLs (do they work?)
  - Handle extraction success rate
- Decision point:
  - If >70% success: Proceed without Apify for URL discovery
  - If <70% success: Budget for Apify (~$50-500/month)
- Document findings and update plan
Decide Instagram Post Count Strategy 💰
- ChartMetric provides URLs but NOT post counts
- Post counts REQUIRED for accurate classification (5/15/30 thresholds)
- Option A: Budget $49-500/month for Apify (accurate)
- Option B: Use follower/listener proxy (free, less accurate)
- Option C: Hybrid (Apify for new leads only, older leads use proxy)
- Get budget approval if choosing Option A

Week 2: HubSpot Setup

Create HubSpot Custom Properties
- Access HubSpot account (from ig-hubspot-tool setup)
- Create custom properties (see list in Phase 3, Task 3.3)
- Test contact creation with properties via API
- Document property IDs

Week 3: Kick Off Development

Start Phase 1 Implementation
- Create chartmetric_client.py in leadgen repo
- Implement Instagram URL fetching
- Update ArtistLead model with new fields
- Test end-to-end enrichment with ChartMetric

Risks & Mitigations

Risk	Impact	Likelihood	Mitigation
Apify costs prohibitive for scale	Medium	Low	Company already has Apify subscription; confirm coverage
Instagram username matching insufficient	High	Medium	Fallback: Spotify-only classification (skip Instagram data)
ChartMetric label discovery workaround fails	Critical	Medium	Alternative: Manual artist lists, or limited SongStats use with budget cap
ChartMetric API lacks needed data	Medium	Low	Hybrid approach: supplement with Apify where needed
HubSpot rate limits slow ingestion	Low	Low	Batch requests, implement retry logic
Post count thresholds don’t match lead behavior	Medium	Medium	Adjust thresholds or use follower/listener proxy

Appendix

A. Level Threshold Reference (From Main App)

The main app’s gamification system uses post count thresholds:

Level 1 (Onboarding)

Complete Buzz Factory questionnaire (3 sections)
Link Instagram + TikTok (both required)
For Leads: Skip Level 1 entirely (they don’t use the app yet)

Level 2 (Starting Out)

Threshold: 5 Instagram/TikTok posts (cumulative, no time limit)
For Leads: Check if they have 5+ posts in last 6 months

Level 3 (Consistent Creator)

Threshold: 15 Instagram/TikTok posts (cumulative)
Period: 56 days for pace calculation (but total is cumulative)
For Leads: Check if they have 15+ posts in last 6 months

Level 4 (Growing Artist)

Threshold: 30 Instagram/TikTok posts (cumulative)
Period: 56 days for pace calculation (but total is cumulative)
For Leads: Check if they have 30+ posts in last 6 months

Level 5 (Established Artist)

Not yet implemented

Note: Follower/listener counts are tracked but only used for “problem type” classification in Phase 4, not for level assignment.

B. Example Enriched Lead Object

json

{
  "lead_id": "uuid-12345",
  "artistName": "DJ Example",
  "spotifyId": "spotify:artist:abc123",
  "chartmetricId": "cm123456",
  "instagramUsername": "djexample",
  "discoveredVia": "label_catalog",
  "seedLabel": "Toolroom Records",
  
  "classification": {
    "level": 3,
    "levelName": "Consistent Creator",
    "meetsThreshold": {
      "instagramPosts": true,
      "hasInstagramData": true,
      "hasSpotifyData": true
    }
  },
  
  "metrics": {
    "instagram": {
      "username": "djexample",
      "followers": 5200,
      "postCount": 18,
      "lastPostDate": "2026-04-15"
    },
    "spotify": {
      "artistId": "abc123",
      "monthlyListeners": 12000,
      "followers": 350,
      "lastReleaseDate": "2026-03-01"
    }
  },
  
  "classifiedAt": "2026-04-20T12:00:00Z",
  "dataSource": {
    "instagram": "chartmetric+apify",
    "spotify": "chartmetric"
  }
}

C. Example Lead with Problem Type Classification

json

{
  // ... (all base fields) ...
  
  "scenarioAnalysis": {
    "scenarios": [
      {
        "id": "high_engagement_low_releases",
        "name": "High Engagement, Low Releases",
        "matched": true,
        "confidence": 0.85,
        "recommendedAction": "Pitch release planning tools and distribution strategy"
      },
      {
        "id": "early_career",
        "name": "Early Career (Small but Active)",
        "matched": true,
        "confidence": 0.72,
        "recommendedAction": "Offer onboarding and growth strategy consultation"
      }
    ],
    "primaryScenario": "high_engagement_low_releases"
  }
}

Glossary

Lead: An artist discovered via leadgen who is a potential Genesis user
Classification: Assigning a lead to a level (1-5) and problem type categories
Level: Same as gamification levels in main app (Starting Out → Growing Artist)
Scenario: A specific growth problem/opportunity pattern (e.g., “high engagement, low releases”)
Apify: Web scraping platform for Instagram data (already in use)
ChartMetric: Music analytics API (primary data source)
HubSpot: CRM where enriched leads are stored

Document History

Date	Version	Changes	Author
2026-04-20	1.0	Initial plan	Jacob Stallings + Claude

Lead Classification System - Project Plan

Executive Summary

Quick Reference

🚨 Critical Technical Requirements

1. Level Classification System

2. ChartMetric Instagram Linking

3. SongStats Constraint

4. Tech Stack

System Components & Architecture

System Components

Existing Repositories

Deployment Options

Option 1: Python CLI (MVP)

Option 2: AWS Lambda (Production)

Option 3: AWS Step Functions (Scale)

Detailed Implementation Plan

Phase 1: ChartMetric Integration & Validation

Requirements

Technical Decisions

Open Questions

Phase 4: Problem Type Classification

Classification Categories

Requirements

Open Questions

Phase 5: RAG-Based Classification (Future Work)

High-Level Plan (Future Enhancement)

Implementation Phases

Phase 1: ChartMetric Integration & Validation

Tasks

Phase 2: Instagram Post Count (Critical for Classification)

Challenge

Options

Tasks

Phase 3: Classification & HubSpot Integration

Tasks

Phase 4: Problem Type Classification

Prerequisites

Tasks

Phase 5: Integration with IG-HubSpot Tool

Tasks

Repository Structure Decision

Option A: Extend leadgen Repo (RECOMMENDED)

Option B: Extend ig-hubspot-tool Repo

Option C: New Repo lead-classifier

Recommendation: Option A - Extend leadgen Repo with Python

Data Sources & APIs

1. ChartMetrics API (Spotify + Instagram Data)

Critical Discovery: Instagram Linking via ChartMetrics

SongStats → ChartMetrics Endpoint Mapping

Label Discovery via ChartMetric (Required - SongStats Too Expensive)

ChartMetric Rate Limits

ChartMetric Cost

2. Apify Instagram Profile Scraper

3. HubSpot API (CRM)

Custom HubSpot Properties (To Create)

Cost Estimation

Monthly Operating Costs

Alternative: Proxy Metrics (No Apify)

Development Cost

Technical Decision Reference

Open Questions & Blockers

Critical Path Blockers

Non-Blocking Questions

Success Metrics

MVP Success Criteria

Enhanced Success Criteria (Phase 4)

Timeline Estimate

Critical Path

Next Steps (Immediate Actions)

Week 1: Critical Validation (HIGHEST PRIORITY)

Week 2: HubSpot Setup

Week 3: Kick Off Development

Risks & Mitigations

Appendix

A. Level Threshold Reference (From Main App)

Level 1 (Onboarding)

Level 2 (Starting Out)

Level 3 (Consistent Creator)

Level 4 (Growing Artist)

Level 5 (Established Artist)

Option A: Extend `leadgen` Repo (RECOMMENDED)

Option B: Extend `ig-hubspot-tool` Repo

Option C: New Repo `lead-classifier`

Recommendation: Option A - Extend `leadgen` Repo with Python