Menu

Individual skill

hubspot-warehouse-sync

0upvotes

Sync HubSpot CRM data to a data warehouse (BigQuery, Snowflake, or Postgres) for analytics and reporting. Covers initial backfill of millions of records under the 500K/day rate limit, incremental CDC polling via hs_lastmodifieddate, schema-drift detection with ALTER TABLE generation, association sync for contacts/deals/companies, and idempotent upsert patterns that prevent duplicate rows on retry. Use when building a HubSpot → warehouse pipeline, resyncing after a schema change, debugging duplicate rows or missing CDC updates, or recovering from a rate-limit burnout mid-backfill. Trigger with "hubspot warehouse sync", "hubspot bigquery", "hubspot snowflake", "hubspot postgres etl", "hubspot backfill", "hubspot cdc", "hubspot schema drift", "hubspot duplicate rows", "hubspot to data warehouse".

skills/.curated/hubspot-warehouse-sync/SKILL.md

Read pinned source

Source commit 0500b275e963c027f6b410e8c49c0446945c574e

What was checked

Matches the observed repository HEAD
Yes
Scanner rules are current
No
Repository scan complete
No
Skill scan complete
No
Observed behavior
Not checked yet
Last checked
Nov 12, 2026

These are observations at the recorded commit, not a guarantee of safety. Review the source before running it.

Install this skill

Choose an agent in your Stack or copy a command below. Commands fetch the pinned commit and copy only this skill directory. Existing destinations are refused. Copying a command does not install anything.

An install command is unavailable: current rules, complete repository and skill scans, an eligible behavior level and a permitted license are required.

Repository context

Repository stars (not skill ratings)
2,916
Repository last updated
Nov 12, 2026
License
MIT

Observed agent integration: No specific integration evidence. Standard SKILL.md installation does not imply verified compatibility.

Files and excerpt

File metadata and permitted, redacted excerpts come from the recorded scan. Source links are pinned to that commit.

Browse files
Read excerpt

# HubSpot Warehouse Sync

## Overview

Move HubSpot CRM data to BigQuery, Snowflake, or Postgres in a way that survives production — not just the demo. This is not a connector walkthrough. It is the extraction and load code your pipeline runs when a 2M-contact backfill burns through the 500K daily call quota at noon, when CDC misses three days of deal updates because association changes do not update `hs_lastmodifieddate`, when a portal admin adds a custom property and your warehouse table schema silently drifts, and when a network timeout at record 45,000 causes your retry to insert 100 duplicate rows.

The six production failures this skill prevents:

1. **Backfill exhausting the daily rate limit before completing** — 2M contacts at 100 records/call is 20,000 calls. At 100 calls/10s the math says 33 minutes, but that burns your entire 500K daily quota before noon and takes every other integration down with it. Token bucket rate limiting with a configurable daily ceiling is non-optional.
2. **CDC missing association changes** — HubSpot's `hs_lastmodifieddate` is updated when any property on the contact record changes, but not when an association is created or deleted. A contact-to-deal link added by a sales rep is invisible to a lastmodifieddate-based incremental poll. Association CDC requires a separate poll strategy.
3. **Schema drift causing silent extraction failures** — when a portal admin adds or removes a custom property, the warehouse table schema and the extraction property list become misaligned. New properties are dropped on the floor. Removed properties cause KeyError on row construction. Neither failure raises an alarm without explicit schema validation.
4. **Duplicate rows on batch retry** — a network failure mid-batch causes the batch to be re-sent. Without a proper upsert key the warehouse gets duplicate rows that are invisible until an analyst notices double-counted revenue. The correct upsert key for contacts is `id` (HubSpot object ID), not a composite of name/email.
5. **Large payload failures on associated object inline pulls** — contacts with thousands of engagement records cause payload sizes that exceed HTTP response limits when associations are pulled inline. Associations must be fetched in a separate batch read pass.
6. **Timezone inconsistency in aggregations** — HubSpot stores all timestamps as Unix milliseconds in UTC. If the warehouse session timezone is set to a local timezone, `DATE(created_at)` aggregations produce different daily totals depending on where the analyst runs the query.
Back to top