The research pipeline
One sentence of input produces a scored report in about four minutes. Six stages run in sequence, each writing its output to Postgres so a later stage can be re-run without repeating the crawl.
- Competitor Discovery. Search queries derived from the idea, results fetched, extracted, enriched and scored for relevance. Results below the relevance floor are discarded rather than counted.
- Feature Extraction. Product, pricing and changelog pages crawled and reduced to a capability matrix. Extraction is keyed by content hash, so an unchanged page is never re-analysed and never produces a phantom diff.
- SEO Research. Seed terms expanded, volume and cost pulled from a keyword provider, difficulty measured against live results.
- Market Gaps. Capabilities nobody ships, weighted by impact against build difficulty.
- Recommendations. Pricing position, business model and a twelve month projection with every assumption labelled as an assumption.
- Report. PDF and PowerPoint generation with source attribution on every figure.
Services
- API. FastAPI on Python 3.11, async throughout, SQLAlchemy over Postgres 16.
- Web. Next.js App Router, React 19, TypeScript, static-exported marketing site and a separate authenticated application.
- Workers. Celery with a Redis broker for the research pipeline and a beat schedule for continuous monitoring.
- Data. Postgres 16 for state, Redis 7 for cache and queue, Alembic for migrations.
- Deployment. Docker Compose across API, worker, beat, web, database and cache, behind a reverse proxy with TLS.
- Identity. Clerk, with JWT verification at the API boundary.
External providers
Every provider sits behind a failover chain with a circuit breaker, so one outage degrades a report rather than failing the run. Where a chain exhausts, the affected field is reported as unmeasured.
- Language models. Multiple providers behind one interface, selected per task rather than globally.
- Web search. A primary search provider with a secondary fallback.
- Crawling. A managed fetcher with a direct-fetch fallback. Target-site errors are classified separately from fetcher faults so a competitor’s 404 cannot trip the breaker.
- SEO data. A keyword and SERP provider for volume, difficulty and cost.
Where the cloud spend goes
SignalDart is being built to run on Google Cloud, and the mapping below is the migration we are working to. The cost centres are container compute for the API and workers, managed Postgres, managed Redis, object storage for generated reports, and egress from the crawling workload.
The workload is bursty by nature: a research run is a few minutes of concurrent crawling and model calls, then nothing. Continuous monitoring adds a steady low-volume schedule across every tracked project, which is where sustained compute and egress accumulate.
How this maps onto Google Cloud
The architecture is deliberately portable — containers, managed Postgres, managed Redis and object storage — so it carries onto Google Cloud without a rewrite. A bursty research run and a steady monitoring schedule are two different scaling problems, and they map onto two different services.
| Component | Runs on today | Google Cloud equivalent |
|---|---|---|
| API (FastAPI) | Container | Cloud Run — request-driven, scales to zero between runs |
| Research workers (Celery) | Container | Cloud Run jobs or GKE Autopilot for the concurrent crawl burst |
| Primary database | PostgreSQL 16 | Cloud SQL for PostgreSQL |
| Queue and cache | Redis 7 | Memorystore for Redis |
| Generated reports | Object storage | Cloud Storage |
| Scheduled monitoring | Celery beat | Cloud Scheduler |
| Language model | Gemini | Vertex AI — already the model in production |
Two things make the migration cheap rather than theoretical. Every stateful dependency is a managed service with a direct Google Cloud counterpart, and the generation step already runs on Gemini, so the model layer needs no change at all — only the endpoint it is called through.
Credits would go to the two costs that scale with usage rather than with headcount: concurrent crawl compute during research runs, and the sustained monitoring schedule that keeps every tracked market current between them.
The parts that exist to stop us inventing things
Most of the engineering that is not obvious from the feature list is here, and it is the part we would point an engineer at first.
- Nullable metrics. Every competitor measurement is nullable. There is no default value anywhere in the model, so an unmeasured field cannot silently become a zero.
- Provenance on every reading. Source URL, provider and observation timestamp are stored alongside the value, not derived later.
- Change detection requires two measurements. A diff is only emitted when both sides were actually measured, which removed a large class of phantom events.
- Content-keyed extraction cache. Analysis is keyed by page content hash, so re-running research on an unchanged market produces no events rather than a fresh set.
- Fuzzy feature matching. Capability names are stemmed and compared on meaningful tokens, so “SSO login” and “single sign-on” are one row rather than two.
Security posture
Projects are scoped to their owning account and cross-tenant access is blocked at the query layer. Secrets are held outside the image; the build context excludes environment files so credentials cannot be baked into a published container.
We hold no third-party certifications yet. SOC 2 and ISO 27001 are on the roadmap and we will not claim either until an auditor signs off. See Security.
Developer access
A REST API and webhooks are available on paid plans. See the API reference.