labs@atheron:~/work/market-data$ cat market-data.md
The data fleet
It knows when it is finished.
Downloading years of market history sounds simple until a venue rate-limits you, pages differently from its documentation, or answers an empty page that is not the end. This fleet handles all three, and only says a venue is complete when every candle is really held.
// what it does
- Holds every candle, from one minute to one week, for every venue we use.
- Backfills history and records live data at the same time, without the two overwriting each other.
- Marks a venue complete only when its whole history is held, and shows it as still filling until then.
- Records venues that block bulk downloads live instead, rather than pretending to backfill them.
- Answers the research and training jobs that read it, on its own servers rather than ours.
// how we built it
- Dedicated servers, so an ingestion run never competes with anything else for a core.
- Every request to a venue goes through one module, so the rate limit is respected in one place and not in nine.
- Storage on volumes attached to each server, bound to the server that holds them.
- A download that fails is a failure: an earlier version wrote a done marker after a transient error, and the history it left behind was short.
- An empty page is not the first candle, so a gap at the front of a venue's history is noticed rather than recorded as the beginning.
// the size of it
- servers
- Dedicated boxes in New York and London, with their own storage volumes
- intervals
- One minute to one week, for every venue we use
- venues
- One module each, with its own quirks written down
- completeness
- Green only when the whole history is held; yellow while it fills
// stack
- Node.js
- TypeScript
- PostgreSQL
- DigitalOcean volumes
- Python
// where it stands
In production on its own servers. It feeds our own research, and nothing is sold or published from it.
// a comparable build
- Build
- ≈ US$68,800 to US$105,000, within 15 weeksCAD 97,900 to 149,700
- Hosting
- ≈ US$853 a monthCAD 1,215 a month
- Support
- ≈ US$2,500 a monthCAD 3,565 a month
Prices in your currency are estimates from today's Bank of Canada rate. All invoicing is in CAD or USD.
the effort in it
- Development
- 394 to 602 h
- Backend developer
- 101 to 154 h
- Senior backend developer
- 90 to 138 h
- QA engineer
- 74 to 114 h
- Junior backend developer
- 50 to 77 h
- In all
- 892 to 1365 h
what this price covers
An ingestion platform of this shape as one build: the venue modules, the rate limiting, the completeness marks, the storage, and the internal API the jobs read it through.
An estimate for a build like this one, worked out from our rate card as you read it. It is not a quote. Yours changes with your scope, and we write a real one after a call.
Open this in the estimator →// the rest
The other projects.
// start
Want one built like this?
The same team, the same tests, the same care. Tell us what you need, or price it yourself first.