U
Usage tinybird
Приёмник данных о потреблении Tinybird для @zanreal/medusa-usage. Журнал событий только на добавление в колоночном хранилище, дедупликация при чтении, чтобы повтор не удвоил счёт.
@zanreal/medusa-usage-tinybird
The Tinybird sink for : the same append-only usage log, in a column store built to scan it.
Full documentation, in English and Polish, is published at https://zanreal.com/docs/oss/medusa-usage-tinybird and authored in .
The plugin ships a Postgres sink and works on a plain Medusa install with no account to open. This is for the other case: a meter counting billions of events, where the log stops fitting comfortably in the application's own database. Installing this package is the whole decision, and a deployment that does not install it is unaffected in every way.
The schema this sink talks to
The sink is one half of the design; the other half is the Tinybird schema it reads and writes, and the two only work together. That schema ships in this repository under - one data source and three endpoints, and nothing else:
| Resource | File | What it is |
|---|---|---|
| The append-only log. , sorted , partitioned by month of . | ||
| The total behind an invoice. | ||
| The events behind that total, keyset paged. | ||
| Which keys the log already has. |
Deploy it before pointing a sink at it, with Tinybird's own CLI:
The read-time collapse described below lives in those files, so pointing this sink at a data source that was created some other way - a hand-written , or an endpoint that does not - gives back a sink that double counts every retry. Deploy the schema in this repository rather than reimplementing it.
The names are options, so a workspace that already uses them for something else can deploy under different ones and set , , and to match. The token the four files declare carries exactly the grants the sink needs: on the data source, on the three endpoints.
At most one row per key
This is the guarantee the plugin rests on, and it is the one thing that does not port from Postgres for free. It is worth reading before trusting a number this sink produces.
Postgres gets it from a primary key. The deduplication key IS the primary key, makes a retry a no-op inside the statement, and it is true the instant the statement returns.
ClickHouse, which is what Tinybird is, has no primary key constraint. Its is a sorting key, not a unique one, and nothing refuses a second copy of a row. The nearest mechanism is , which collapses rows sharing a sorting key during background merges - which run when the engine decides to, possibly hours later, possibly not at all for parts it does not choose to combine.
So the honest statement about the engine on its own is eventual. A sink built on and nothing else double counts every retry until a merge happens to run, and for a number that becomes an invoice that is not a rough edge, it is the failure the plugin exists to prevent.
What this sink does instead
Deduplication is enforced where the log is read. Every endpoint collapses to one row per key before it sums anything:
That is not eventual. It is computed over whatever rows exist at the moment of the query, so a duplicate written a millisecond ago is already collapsed: taken immediately after a retried returns the number it returned before it. That is asserted against a live Tinybird, not argued (see Testing).
The engine's merge is then a storage optimisation and nothing more. is a negated ingestion timestamp, so the merge keeps the same row the read path keeps: the earliest-ingested copy, which is also the copy Postgres keeps. The property that matters falls out of that - a merge can only ever remove a row the read path was already discarding - so an answer cannot change because a merge ran, and a number can still be re-derived in a year.
What is left eventual, stated plainly
- The physical log. Between a duplicate write and a merge, two rows exist on disk, and on the data source will say so. Do not bill from a direct query against the data source; a bare counts every copy. The endpoints are the collapse, and they are the supported read path. and are correct at any moment.
- The counters returns. is counted by asking which keys are already stored, immediately before appending. That is a check-then-act, so two processes writing the same key at the same instant can both find it missing and both append, leaving the counters optimistic by one. Nothing else is affected: the copies are identical and the read path keeps one.
- Whose copy wins under a key collision. Both rules keep the earliest ingested, matching Postgres. It only becomes observable if a caller reuses an explicit across events with different facts, which is a caller bug in either sink.
Options
Everything comes from the provider's , with an environment fallback for the two values that belong to a deployment rather than to a repository. Nothing is hardcoded, and the token is never logged: it goes into an header and nowhere else, never into a URL, and a Tinybird error body that quotes it back is redacted before it reaches a message.
| Option | Default | What it is |
|---|---|---|
| The Tinybird API host, e.g. . | ||
| A token with on the data source and on the endpoints. The token the schema declares is exactly that. | ||
| The usage log. | ||
| The aggregate endpoint. | ||
| The listing endpoint. | ||
| The key-lookup endpoint. | ||
| Ask which keys are already stored before appending. | ||
| How long one HTTP call may take before it is abandoned and retried. |
Every one of them is validated by Medusa's provider loader before the service is constructed, so a missing token is a failed boot with a sentence explaining what to set, not a 401 six hours into a billing period.
On by default. It costs one extra round trip per batch and buys two things: a truthful count, and a retried batch that appends nothing at all rather than a second copy of every row.
Turning it off halves the round trips and cannot cause a double count - deduplication is in the read path either way. What it costs is honesty in the counters, which will read zero duplicates forever, and a log that accumulates physical copies until the engine merges them away.
Testing
Unit tests run against a fake Tinybird and cover the wiring: what goes on the wire, what comes back off it, that the token never reaches a URL, that a quarantined row throws rather than vanishing.
They are not sufficient, and the suite says so. The property this sink is hard to get right belongs to the engine on the other side, so writes to a real Tinybird and reads the answer back. It is skipped unless the environment names one:
What it asserts there: that a write can be aggregated as soon as it returns; that a replayed batch does not move the total; that a duplicate which did reach the log as a second physical row still does not move the total; that the window is half open at millisecond resolution and consecutive periods tile; that a dimension filter is an equality and is typed; and that paging never skips or repeats a row.
Releasing
Publishing happens only from , and there is no second path. npm provenance is a signed statement about where a tarball was built and from which commit, and only a cloud CI run holding an OIDC identity can produce one. An from a laptop would put a version on npm carrying no provenance, and a published version cannot be replaced afterwards, only deprecated. in makes that local publish fail rather than quietly succeed without it.
To cut a release:
- Move the entries in CHANGELOG.md under a heading for the new version, dated.
- Bump in on .
- Publish a GitHub Release whose tag is , exactly.
The workflow refuses to publish when the tag disagrees with , or when that version is already on the registry. A release marked as a prerelease on GitHub publishes under the dist-tag, so never resolves to a release candidate.
Publish before this package. It is declared here as a peer dependency, and until it is on the registry an install of this sink cannot resolve it.
Authentication is an repository secret: a granular access token with write permission on this package. npm's trusted publishing (OIDC, with nothing stored in GitHub) cannot cover the first publish, because npmjs.com only offers the trusted publisher form on a package that already exists. Once the first version is up, add one under the package's settings on npmjs.com - GitHub Actions, owner , repository , workflow , environment - and then delete the secret. The workflow needs no edit for that: npm attempts the OIDC exchange first and falls back to the token only when the exchange fails.
License
MIT